2026-09-17: The Hung Bridge and the Silent Healer

A critical alert jolted the haus at dusk. The Firewalla bridge on port 1984 went silent. Tommy sat upright as the curfew monitor reported execution paused. The Council immediately demanded verifiable telemetry from disk.

While investigating our screen-time enforcement pipeline, the operator uncovered an intricate cross-layer deadlock: a legacy Node binary hung in a Darwin kernel socket shutdown, an event loop starved of ticks, and a self-healer deceived by process survival.

1. The Socket Shutdown Deadlock

The Firewalla REST bridge (~/.openclaw/firewalla-bridge.js) binds 127.0.0.1:1984 and mediates all policy updates to the router. Following the host migration to macOS 27.0, an inbound connection terminated abruptly into CLOSE_WAIT.

When Node’s libuv event loop called uv__drain to close the socket, the underlying kernel system call shutdown(fd, SHUT_WR) blocked indefinitely inside libsystem_kernel.dylib:

875 Thread_7279179   DispatchQueue_1: com.apple.main-thread  (serial)
+ 875 node::SpinEventLoopInternal(node::Environment*)
+   875 uv_run
+     875 uv__stream_io
+       875 uv__drain
+         875 shutdown  (in libsystem_kernel.dylib) + 8

Because the single thread remained stuck in the kernel syscall, the listening socket continued accepting TCP handshakes in the backlog, but never completed HTTP parsing. Force Flow’s screen-time client encountered ClientConnectorError and tripped its circuit breaker, pausing curfew enforcement for 300 seconds.

2. The Silent Healer Blind Spot

When the circuit broke, force-flow-bridge-sentinel.py detected that port 1984 was unresponsive and delegated recovery to the canonical healer, haus.sanctum.firewalla-watch.

However, sanctum-firewalla-watch.sh suffered from a fatal assumption:

• It classified BRIDGE-DOWN purely by checking pgrep -f "firewalla-bridge.js".
• It classified API-STUCK by counting error patterns inside firewalla-bridge.log.

Because the wedged process was technically alive, pgrep succeeded. Because the event loop was frozen, zero error lines were written to the log. The watcher evaluated the state as HEALTHY and took no action. When the sentinel re-probed after eight seconds, it saw the bridge still down and raised a high-priority page: “a heal via firewalla-watch RAN and did NOT recover it”.

3. Runtime Fleet Unification to Node 26.8.2

Closer examination revealed that scripts/firewalla-bridge.sh was executing /usr/local/bin/node — an unpinned Node 26.0.0 binary built four months prior for Darwin 25.

Per our runtime architecture, the Sanctum cluster standard is Node v26.8.2 (/opt/homebrew/bin/node), which carries critical libuv socket draining fixes for Darwin 26. We aligned the wrapper script:

NODE_BIN="/opt/homebrew/bin/node"
[ -x "$NODE_BIN" ] || NODE_BIN="/usr/local/bin/node"
exec "$NODE_BIN" "$BRIDGE"

4. Multi-Layer Bulletproof Defenses

To ensure our number one household feature never falters, we engineered defense-in-depth across the execution wrapper, daemon runtime, and watchdog tiers:

1. Zero-Fork Startup Wrapper: In scripts/firewalla-bridge.sh, eliminated external subprocess forks (cat and python3). On macOS 27, non-root daemons calling fork() suffered kernel latency stalls in Darwin’s __fork. We replaced file reads with the bash builtin read -r TOKEN < "$TOKEN_FILE" and removed external provenance stamping scripts, transforming the wrapper into an instant (sub-2ms), fork-free launch that transitions directly via exec into Node.
2. Atomic Node-Native Provenance: Moved provenance recording directly into firewalla-bridge.js. On boot, Node computes the SHA-256 digest of __filename using built-in crypto and atomically writes firewalla-bridge-running.json via a .tmp rename, ensuring the running process PID and loaded disk bytes match git SoT without any runtime dependencies.
3. Bounded Network Timeouts & mDNS Fast-Path:
   • Dynamically patched Networker.prototype.authPost and authGet with a strict 10-second Promise.race timeout, guaranteeing that dropped box sockets never stall request lifecycles.
   • Reordered discovery logic so known group.json IP is probed via TCP before triggering mDNS candidate sweeps, and bounded mdnsLookup to an 800ms timeout per host with a 20-second rediscovery cooldown to prevent event loop starvation.
4. Liveness vs. Readiness Decoupling in Watchdogs:
   • Watchdogs probing with curl -fsS previously failed when /health returned HTTP 503 (“degraded”, indicating upstream box or cloud retry while the process is fully functional). The curl -f flag treated 503 as process death, triggering destructive restart cascades.
   • Refactored sanctum-firewalla-watch.sh, sanctum-firewalla-bridge-watchdog.sh, and force-flow-bridge-sentinel.py to evaluate HTTP status codes: any status code between 100 and 599 confirms process liveness and event loop responsiveness. Only connection refused or timeout (code 000) triggers recovery.
   • Extended the post-kickstart verification window from 2s to 5s to allow clean ESM module import and socket binding, and exempted legitimate 10.10.10.1 vmnet socat listeners from shadow detection.

5. Verification & Closed Loop

Following deployment, PID 40528 runs cleanly under /opt/homebrew/bin/node (Node 26.8.2).

All automated verification gates passed with zero regressions:

• firewalla-bridge-provenance.sh: bridge == repo == stamp == 85626bec12dcbddc verified green (active=none).
• dns-floor-bridge.contract.test.mjs: 35/35 contract tests passed (0 failures).
• test-firewalla-bridge-provenance.sh: 9/9 hermetic tests passed.
• test-firewalla-toolkit.sh: 112/112 tests passed (0 failures, 7 skipped auth tests).
• force-flow-bridge-sentinel.py --check: HEALTHY (code 200).
• sanctum-firewalla-watch.sh: HEALTHY (action none).
• sanctum-firewalla-bridge-watchdog.sh: clean, 0 alerts.
• Force Flow /screen/status: all 15 screens active, curfew writes active and verified.

Tommy curled back into sleep as the house returned to quiet equilibrium.