2026-04-17: The Full-Stack Health Sweep

A parallel session fixed one Claude CLI proxy route and closed the ticket. The sweep after was supposed to be a rubber stamp. It was not. Our ten-minute spot-check became nine components degraded or missing across three sessions and two calendar days. The Q2 catalog rename had landed three weeks earlier, and it had quietly broken everything nobody had looked at since. You do not get to rubber-stamp a stack that renamed itself in April and told no one.

Apr 17 — The Sweep Itself

Components Checked

We walked twenty-eight components end to end. Twelve had something wrong.

Component	Status	Notes
Navigator Sidecar (:3344)	FAIL	Process not started; no monitor-status.json for any project
Holocron UI (:3333)	DOWN	Not running
Command Center (:1111)	PASS	Serving HTML
Health Center (:2222)	PASS process / WARN data	/health returns 502 because health-tunnel is down
OBLITERATUS (:7860)	DOWN	remedy_venv.sh does not exist in OBLITERATUS directory
sanctum-watchdog (:2187)	PASS	Reporting overall: degraded with 9 root causes
sanctum-proxy (:4040)	PASS	Health endpoint responds correctly
council-mlx (:1337)	PASS	Running
tommy (:3355)	PASS	Dawn + dusk briefings sent successfully
xtts-server (:8008)	PASS process	Running via homebrew python3.12; LaunchAgent symlink was broken
health-tunnel (:18095)	DOWN	SSH tunnel to VM not established
ha-tunnel (:18092)	DOWN	SSH tunnel to VM not established
graphiti-server (:31416)	DOWN	VM-hosted service, VM SSH unreachable
network-control (:4007)	DOWN	VM-hosted service, VM SSH unreachable
signal-proxy	DOWN	VM unreachable via SSH
anthropic-proxy	DOWN	VM unreachable via SSH
VM (openclaw SSH)	UNREACHABLE	ssh openclaw times out; local qemu-system-aarch64 is running

The rest (tommy, sonos-bridge, voice-agent, lmstudio, memory-vault, home-assistant, kiwix, rewind-dashboard, health-ingester, sanctumctl.py, the sanctum-rs binary, living-force.mdx) were green.

Root Causes Found and Fixed

Eight independent bugs, each its own small story, most tracing to one event: the Q2 catalog rename (285e817) had updated instance.yaml service keys (xtts → xtts_server, gateway → openclaw_gateway, mlx_server → council_mlx), but nothing around those keys had been re-synced since.

1. Runtime manifests stale. render_runtime_services.py had not been re-run since the rename. Three manifests showed DIFF against source — council-mlx.yaml, xtts-server.yaml, openclaw-gateway.yaml. The renderer produced 33 manifests and cleared all diffs.

2. sync_runtime_calibration.py SERVICE_MAP drift. com.sanctum.xtts-server.plist was still mapped to key "xtts" in the SERVICE_MAP constant while instance.yaml now used xtts_server. The enabled() check returned False, so the plist never rendered. Changed "com.sanctum.xtts-server.plist": "xtts" to "xtts_server" in tools/sync_runtime_calibration.py; re-running it created the plist and cleared the launchagent audit.

3. sanctum-xtts-server symlink broken. ~/.sanctum/bin/sanctum-xtts-server pointed to a venv that no longer existed (~/Projects/yoda-voice-agent/.xtts-venv/bin/python), and audit_runtime_launchagents.py flagged it MISSING. The server was in fact running via python3.12 from the LaunchAgent’s PATH — the symlink is the entry point, not the runtime. Repointed to /opt/homebrew/bin/python3.11, the interpreter the pin_deps transformers constraint expects.

4. Legacy living-force plist marker missing. test-sanctum-runtime-audit.sh expects com.sanctum.living-force.plist.disabled as proof the legacy watchdog is retired. Neither the active plist nor the disabled marker existed; created the empty .disabled marker.

5. mlx-finetune/configs/agents.yaml missing. sync_agent_markdown.py defaults to this path, but only the patches/ directory was in the repo, so the script crashed with FileNotFoundError. Created the file with all six canonical agents — windu, quigon, cilghal, jocasta, mundi, yoda — each referencing a workspace subdirectory with workspace_optional: true so missing ones skip gracefully.

6. Test harnesses not updated after the Q2 rename. Three test files still referenced old service slugs and counts. test-sanctum-system-e2e.sh: Services: 30 → Services: 33; xtts --> voice-agent → xtts-server --> voice-agent; proxy mode/server fields retired for routing/providers (the proxy health response never included mode or server — that assertion was aspirational all along). test-sanctum-runtime-audit.sh: SUPPLEMENTAL_COUNT:6 → 9; VOICE_AGENT_DEPS:xtts → xtts_server. test-sanctum-evolution-loop.sh: incident-learn.sh gateway → openclaw-gateway.

7. Agent capabilities stale. ~/.sanctum/config/agent-capabilities.yaml had drifted; sync_agent_capabilities.py brought it back in sync.

8. Four LaunchAgent plists stale. sync_runtime_calibration.py synced gateway.docker, gateway, ha-tunnel, and health-tunnel.

Still Degraded — Infrastructure, Not Code

Six components were still unhealthy at end of day. Every one was a tunnel or VM reachability issue — Qui-Gon’s territory, infrastructure not code:

• health-center /health → 502 (health-tunnel down)
• health export canary → 502 (same tunnel)
• VM → mac MLX bridge → SSH unreachable
• VM → mac LM Studio bridge → SSH unreachable
• Navigator sidecar → not running (no monitor-status.json files, so it starts degraded anyway)
• OBLITERATUS UI → not running (venv setup not done, remedy_venv.sh missing)

The watchdog — Mothma’s ledger, the one voice that never rounds up — reflected all of it honestly: overall: degraded.

A catalog rename has a fanout

instance.yaml service keys touch the runtime manifest renderer, the calibration sync map, the audit test harness, the evolution-loop assertions, and whatever bespoke script remembers the old name. Every rename looks small. Every rename is actually a sweep.

Apr 18 — Infrastructure Recovery

Day one closed nine code issues and left us three infrastructure ones: openclaw VM SSH unreachable, navigator-sidecar down, OBLITERATUS down. This session was meant to finish them. Two of the three fixed themselves overnight — a lesson in how long to wait before believing a failure.

Pre-Session Watchdog State

overall: degraded, 22/33 healthy. Root causes listed: anthropic-proxy, firewalla-bridge, graphiti-server, ha-tunnel, health-center, health-tunnel, network-control, signal-proxy, triage.

The watchdog API answered on :2187, but its last_check_at was stale (14:00 UTC), and launchd kept failing to start with failed to bind port 2187: Address already in use. An orphan watchdog (PID 1494), started by sanctum-bootstrap.sh on Apr 17, was squatting the port and serving stale results.

What Was Wrong

1. Stale watchdog serving cached “VM unreachable” state. PID 1494 had run its last check at 14:00 UTC yesterday, when VM SSH was down. By session start, ssh openclaw echo ok returned immediately, but the watchdog held stale state and launchd couldn’t start because 1494 held the port. Killed 1494; launchd started a fresh instance, and after the 15-second settle delay it ran clean checks. anthropic-proxy, triage, and signal-proxy (partially) all resolved from this one fix — the “VM unreachable” messages were phantom failures. Root cause of the SSH outage: undetermined — qemu-system-aarch64 ran throughout and the bridge was up, so likely a transient network hiccup or brief bridge flap.

2. ha-tunnel plist stale — loaded config used 70707:127.0.0.1:70707. The running launchd ha-tunnel carried a different port spec than the on-disk plist. Disk said 18092:127.0.0.1:18092 (valid SSH -L format); the loaded config still had the old 70707:127.0.0.1:70707 from before the last sync_runtime_calibration.py run, so SSH rejected every attempt with Bad local forwarding specification '70707:127.0.0.1:70707'. launchctl unload + load on /Users/neo/Library/LaunchAgents/com.sanctum.ha-tunnel.plist; port 18092 opened immediately.

3. health-center (:2222) in restart loop. com.sanctum.health-center showed exit code 143 (SIGTERM) across 979 runs. The server started fine but died because a stale test process (PID 92849, started by run_sanctum.sh) held port 2222. Killed it; the launchd-managed health-center took over and stabilized.

4. firewalla-bridge port mismatch. ~/.sanctum/services/firewalla-bridge.yaml declared port: 1984 for the liveness check, but firewalla-bridge.sh binds to 18094 (hardcoded FIREWALLA_BRIDGE_PORT="18094"). The watchdog was checking a port that was never open. Updated the YAML to port: 18094 in the provides, liveness.port, and port fields.

5. navigator-sidecar — already running. It was up (PID 43966) at session start; the previous session’s “not running” finding had self-resolved overnight. Confirmed via curl http://127.0.0.1:3344/status.

6. OBLITERATUS — Python 3.14 + torch startup deadlock. obliteratus ui failed with ModuleNotFoundError: no module named 'obliteratus'. Python 3.14 silently skips .pth files in dot-prefixed directories, and .venv/lib/python3.14/site-packages/ held __editable__.obliteratus-0.1.2.pth and _virtualenv.pth — Python logged Skipping hidden .pth file for each. The package was installed but unreachable. A workaround imported correctly interactively (PYTHONPATH=/path/to/OBLITERATUS ./.venv/bin/obliteratus ui) but not detached: as a background process, torch 2.11.0 stalled loading libtorch_cpu.dylib (216 MB) at low I/O priority (SN state). Interactive: 0.7 seconds. Background: over ten minutes. It stayed down; the proper fix — a Python 3.12 venv at a non-hidden path — carried to the next session.

Post-Session State

overall: degraded, 29/33 healthy (up from 22/33). Newly green: anthropic-proxy, ha-tunnel, health-center, triage, firewalla-bridge. Four services still unhealthy, all pre-existing infrastructure gaps.

Apr 18 — Second Session, The Last Four

Four to go, all on the far side of a tunnel: graphiti-server, health-tunnel, network-control, signal-proxy. We came back for the stragglers.

What Was Wrong

1. health-tunnel port mismatch between plist and VM service. The LaunchAgent plist forwarded 18095→VM:18095, but the health-ingester on the VM was bound to 10.10.10.10:10101. The service YAML checked port: 18095 — never open on the mac side. Updated the plist to forward 127.0.0.1:10101:10.10.10.10:10101 and the YAML to check port: 10101. Killed the stale bootstrap-era tunnel (PID 72802) on the old 10101 forward, then reloaded the LaunchAgent. Port 10101 opened immediately; /health returned {"status":"ok"}.

Port 18095 was vestigial from an earlier health-ingester config bound to loopback:18095; no new port assignments were made.

2. graphiti-server and network-control — missing SSH tunnel plists. Both run inside the VM on 127.0.0.1 (VM loopback), confirmed via lsof -i :31416 -n -P and lsof -i :4007 -n -P on the VM. No mac-side LaunchAgent forwarded these ports, so the watchdog’s checks always found them closed. Created two new SSH tunnel LaunchAgents and matching sanctum-*-tunnel symlinks:

• ~/.sanctum/bin/sanctum-graphiti-tunnel → /usr/bin/ssh ~/Library/LaunchAgents/com.sanctum.graphiti-tunnel.plist — forwards 127.0.0.1:31416:127.0.0.1:31416 via openclaw.
• ~/.sanctum/bin/sanctum-network-control-tunnel → /usr/bin/ssh ~/Library/LaunchAgents/com.sanctum.network-control-tunnel.plist — forwards 127.0.0.1:4007:127.0.0.1:4007 via openclaw.

Both loaded immediately. Verified: graphiti /health returns {"status":"ok","neo4j":"connected"}; network-control /health returns {"status":"ok","dns_connected":true}. Updated both service YAMLs to reference their launchagent fields (previously null). Port 4007 is the canonical network-control port from the original design, and neither port required reassignment.

3. signal-proxy — broken grep pattern. signal-health.sh CHECK 4 (check_forceflow_port) used:

grep -E '127\.0\.0\.1:[0-9]+/api/v1/rpc' "$FORCE_FLOW_PY"

But force_flow.py’s send_signal() uses http://127.0.0.1:8080/v2/send — REST format, not a JSON-RPC path. The pattern never matched, so configured_port was always empty and the check reported cannot parse signal port from force_flow.py. The watchdog read that as overall: 2 (needs_intervention) even though signal was fully healthy. Updated the pattern:

grep -E 'http://127\.0\.0\.1:[0-9]+/v[0-9]+/' "$FORCE_FLOW_PY"

This correctly extracts port 8080. Since configured_port == CANONICAL_PORT (both 8080), CHECK 4 now reports healthy. Full script run: exit 0, all 6 components healthy; the watchdog picks it up on the next check cycle.

4. OBLITERATUS — Python 3.12 venv migration at a non-hidden path. The actual fix for what the previous session had only worked around.

python3.12 -m venv /Users/neo/Documents/Claude_Code/OBLITERATUS/venv
venv/bin/pip install -e ".[spaces]"
venv/bin/python -c "import obliteratus; print('ok')"  # → ok
venv/bin/obliteratus ui --port 7860 --host 127.0.0.1 --no-browser

Torch loaded in under 60 seconds with Python 3.12, within its officially supported range (3.9–3.12). Port 7860 opened; curl http://127.0.0.1:7860/ returned HTTP 200. Created OBLITERATUS/remedy_venv.sh to document the recreation procedure with the correct flags. Why 3.12 fixes the stall: its established import paths the macOS page cache handles efficiently even at SN priority, where 3.14’s newer dispatch and dynamic-linker paths interact poorly with torch’s Metal and OpenMP init.

Post-Session Watchdog State

overall: healthy, 33/33 services healthy (up from 29/33 at session start). Newly green: graphiti-server, health-tunnel, network-control, signal-proxy.

The ten-minute spot-check that takes a day

A health sweep triggered by “a narrow CLI fix” almost always takes longer than the fix. Every instance key, port, and symlink depends on three things nobody updated when it moved. If the sweep finishes quickly, you didn’t look hard enough.

Gotchas for Next Time

• Q2 catalog renames have long tails. After any instance.yaml service key rename, run render_runtime_services.py and re-check the SERVICE_MAP in sync_runtime_calibration.py for stale keys. The two drift independently.
• Symlink audit catches broken venvs. Delete a venv and the .sanctum/bin/ shim symlinks break; audit_runtime_launchagents.py catches it. Fix by recreating the venv or repointing the symlink to the system interpreter.
• Test harness service counts are exact. test-sanctum-system-e2e.sh asserts Services: N; any instance.yaml addition increments it. Update the test the moment you add a service.
• Bootstrap watchdog squats launchd. On boot, sanctum-bootstrap.sh starts a watchdog directly while the launchd com.sanctum.watchdog plist tries to start another. They race for port 2187; bootstrap wins, launchd logs failed to bind port 2187 every ten seconds forever, and the squatter’s cache goes stale. Kill the bootstrap PID; launchd restarts fresh. Long-term: remove the watchdog from sanctum-bootstrap.sh — launchd manages it now.
• launchctl loaded config can diverge from the on-disk plist. launchctl print gui/UID/com.sanctum.ha-tunnel may show different args than the file if the plist was regenerated via sync_runtime_calibration.py but never reloaded. launchctl unload + load is the fix; check with launchctl print first.
• Python 3.14 skips .pth files in hidden dirs. Any editable install in .venv/ (or any dot-prefixed path) breaks silently. Use PYTHONPATH explicitly or recreate the venv at a non-hidden path (venv/). The rule is venv/ not .venv/ until torch officially supports Python 3.13+.
• SSH -L spec depends on where the service binds. VM-loopback services need 127.0.0.1:PORT:127.0.0.1:PORT; bridge-IP services need PORT:10.10.10.10:PORT. Change a bind address without updating the tunnel spec and it forwards to a port nothing listens on. Verify with lsof -i :PORT -n -P on the VM after any bind-config change.
• signal-health.sh grep must track force_flow.py. If send_signal() changes its URL path (/v2/send vs /api/v1/rpc), update CHECK 4’s grep pattern, documented in the script header.

Thirty-three services green, and exactly one asked nothing of us the whole time: Tommy, filing his dawn and dusk briefings straight through the sweep while everything around him drifted. The stack is only ever as healthy as the last thing you renamed — which is why the next morning’s off-catalogue audit went looking for the services no manifest knew about at all.