2026-09-20: The Silent Canary and the Zero-Clobber Secret

Silence from a voice agent is the deepest form of telemetry failure. When Yoda stopped answering inbound calls on Sunday evening, the telephone lines did not ring out an alarm. Instead, the agent process silently crashed in a tight restart loop under launchd, while the hourly canary dutifully logged failures into a file that no human was reading.

The Jedi Council convened under our living force architecture doctrine to investigate the outage. In accordance with our core invariant, every conclusion was derived from disk artifacts rather than assumption. Since the night Yoda got bulletproof, we had fortified the audio inference pipelines. But the post-incident triage revealed not a breakdown in neural synthesis, but a cascade of subtle trust assumptions between macOS Keychain, shell script generators, and telemetry alerting pipelines.

1. Anatomy of the Outage: The Clobber Cascades

The chain of failure began inside ~/Projects/yoda-voice-agent/generate-env.sh. Designed to populate live .env files from the macOS Keychain, its credential helper function relied on an aggressive fallback:

KC() { security find-generic-password -a sanctum -s "$1" -w 2>/dev/null || echo "MISSING_$1"; }

Whenever the script executed during a locked session or headless launchd invocation without Keychain access, security exited non-zero. Rather than aborting, the script printed literal sentinel strings such as MISSING_livekit-api-key and unconditionally overwrote the existing .env on disk:

LIVEKIT_API_KEY=MISSING_livekit-api-key
LIVEKIT_API_SECRET=MISSING_livekit-api-secret

With its credentials corrupted, yoda-agent attempted to connect to LiveKit Cloud. The LiveKit gateway responded with HTTP 401 Unauthorized. The Twirp RPC client crashed parsing the unexpected payload, and launchd dutifully restarted the process every thirty seconds, trapped in an infinite error loop.

Simultaneously, the hourly canary script canaries/canary_internal.py detected the failure and triggered canaries/alerts.py. But alerts never reached the operator. Triage of ~/.openclaw/logs/yoda-alerts.log showed repeated records:

{"ts": 1789953420.12, "message": "Yoda canary ALERT: 2 consecutive internal fails", "imessage_sent": false, "ha_sent": false}

The alerting subsystem had two fatal flaws:

1. It attempted to parse ~/.openclaw/homeassistant/secrets.yaml, a path that had been decommissioned during the July estate secrets migration.
2. It attempted to invoke Messages.app via AppleScript, which Darwin kernel security rejects under headless launchd sessions.

The canary was screaming into the void.

2. Zero-Clobber Secret Generation

To eliminate this vulnerability permanently, we introduced the Zero-Clobber principle across four core Sanctum repositories: yoda-voice-agent, command-center, health-center, and shannon.

The new generate-env.sh architecture enforces five invariants:

1. Keychain First, Disk Fallback: If Keychain lookup fails, the script inspects the existing .env file on disk. If a valid, non-placeholder secret already exists, it is preserved.
2. Strict Placeholder Rejection: Any value matching ^MISSING_ is rejected as invalid.
3. Pre-flight Validation: All mandatory keys (LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET, DEEPGRAM_API_KEY, ANTHROPIC_API_KEY) must resolve to non-empty, genuine credentials before writing.
4. Atomic Replacement: Configuration is written to a temporary PID-tagged buffer (.env.tmp.$$) and swapped onto disk using atomic mv -f with permissions clamped to 0600.
5. Fail-Closed Execution: If any required credential is missing, the script halts with an explicit error on standard error, leaving the active .env file completely untouched.

In ~/.sanctum/scripts/inject-secrets.sh, the silent suppression loop was replaced with active error tracking, ensuring any generation failure halts deployment pipelines.

3. In-Process Validation and Keychain Self-Healing

Hardened generation scripts protect disk state, but running processes require defense-in-depth. In agent/livekit_entry.py, we implemented runtime credential verification before launching the LiveKit worker loop:

def validate_environment(auto_heal: bool = True) -> tuple[bool, list[str]]:
    critical_vars = {
        "LIVEKIT_URL": ("livekit-url", "wss://yoda-voice-4l8a8o52.livekit.cloud"),
        "LIVEKIT_API_KEY": ("livekit-api-key", None),
        "LIVEKIT_API_SECRET": ("livekit-api-secret", None),
    }
    # Validate each key and attempt Keychain recovery if placeholders are found

If an operator accidentally starts the agent with a corrupted environment while the macOS session is active, the agent queries Keychain directly, self-heals its environment variables, writes a clean .env file to disk, and proceeds without downtime.

If credentials cannot be healed, the agent refuses to start broken. It posts a critical alert to Force Flow and halts cleanly, preventing launchd churn.

4. Unmuting the Canary: Multi-Tier Alert Routing

We restructured canaries/alerts.py to route all canary notifications through the canonical Sanctum notification bus:

[Canary Failure]
       │
       ├── Tier 1: POST http://127.0.0.1:4077/notify (Force Flow)
       │           ├─ iPhone push via HA
       │           ├─ Signal message via VM
       │           └─ Dashboard banner
       │
       ├── Tier 2 (Offline Fallback): ~/.sanctum/force-flow/spool/*.json
       │           └─ Drained by Force Flow flight recorder on recovery
       │
       ├── Tier 3 (Direct HA): Bearer auth via ~/.sanctum/secrets/ha-token
       │
       └── Tier 4 (Best-Effort): iMessage via osascript

Unit tests were expanded to simulate network outages, token resolution across disk and Keychain, and spooling persistence. All ninety-eight unit tests in yoda-voice-agent and four integration suites passed.

5. Verification and Production Proof

With the new architecture in place, we conducted full verification:

• Negative testing proved that simulating an offline Keychain leaves valid .env files completely unmodified and exits with code 1.
• Restarting haus.sanctum.yoda-agent established an instant connection to LiveKit Cloud, registering worker AW_ZTopydqjaTRK.
• The internal voice canary executed live:

canary-internal: ok elapsed=3503ms first_audio_ms=2199 room=yoda-canary-8c12a020

Yoda answers once more. The canaries have voice, and the secrets are anchored in stone.