2026-07-11: The gateway learns to fall on its feet

Seven times in three minutes, proxyd told Claude Code that every model on the roster had failed. Not one of them had. The gateway was lying — with a straight face, a clean end_turn, and enough conviction that nothing downstream thought to retry.

The doorman doc gave proxyd a lock on its one door. This note is about what happened when the room behind that door kept catching fire. Over one evening we took the gateway from a service that panicked on a missing config file, to one that fails the way a load-bearing service should. Quietly. With a logged reason. Onto the next model instead of onto the floor.

The incident: one timeout, every model dead

The all models failed bursts had a single root cause, and it was not the one we assumed. The models were not down. proxyd wrapped the entire upstream call — the request-body upload plus the wait for the first byte — in one flat tokio::time::timeout(10s, …). Anthropic’s average time-to-first-byte under load was 12.8 seconds. The timeout sat below the average. So a large-context request from Claude Code was guaranteed to trip it — every time, by arithmetic.

Then it got worse. The timeout fired on every model in the fallback chain in turn — claude-fable-5 → glm-52 → council-local-think, each dying at its own 10s mark. You waited thirty-plus seconds only to be told nothing worked. The fallback chain was not a safety net. It was three identical trapdoors.

The timeout was measuring the wrong thing

A 10s ceiling on time-to-first-byte is reasonable. A 10s ceiling on upload + queue + first-byte for a multi-megabyte prompt is a guillotine. The fix separates the two: a short connect timeout catches genuinely dead hosts in ~3s, and a generous per-model TTFB budget (default 30s, overridable per seat via ttfb_timeout_secs in config.yaml) covers the legitimate wait. Once headers arrive, the stream flows unbounded — a slow generation is not a failure.

The panics: unwrap() is a deployed time bomb

The crash-restart storm — thousands of panics, zero log context — came from unwrap() and expect() on the hot path and at startup. A missing config file expect()ed its way straight into a launchd KeepAlive loop. Worse: the log appender panicked before logging was initialized, so the reason went to a stderr that launchd silently drops. The gateway died screaming into a void, and we could not read a word of it. We replaced every one:

• Startup fails with a logged reason and a real exit code (EX_CONFIG / EX_SOFTWARE) instead of a panic, so KeepAlive throttling paces the restart instead of a stack-trace flood.
• The log appender is now built fallibly and falls back to stdout if the log directory is unwritable — the behavior the comment always claimed but never had.
• A panic hook routes any residual panic into proxyd.log with its file:line before the default handler runs. No more silent deaths.
• The per-request JSON transforms (route.rs, transform.rs) no longer unwrap() as_object_mut() — a malformed request passes through to a clean 400 instead of taking down the handler.

The audit: nine more ways the gateway lied about success

A 38-agent parallel audit — each finding adversarially verified before it counted — surfaced fifteen confirmed reliability defects. The worst of them shared one theme, and it is the one that should worry you most. The gateway kept reporting success it had not achieved.

• An OpenRouter 200 carrying a {"error":…} body with no choices was translated into a fabricated empty end_turn — a fake answer that never triggered fallback. Now it returns an error and the chain advances.
• A mid-stream {"error":…} SSE chunk (which OpenRouter sends after a 200 when a provider dies mid-generation) was silently dropped, so a failed generation ended as a clean end_turn. Now it surfaces an Anthropic error event.
• The streaming translator ran from_utf8_lossy on each raw network chunk, so a multi-byte character — an accented name, an emoji — split across a chunk boundary was corrupted to � and forwarded to you. The buffer is now byte-based and only decodes complete events.
• Prompt-cache injection marked unbounded cache_control breakpoints, manufacturing a 400 past Anthropic’s four-breakpoint limit that the fallback chain could not catch. Now budgeted to four.

Alongside those: the passthrough path was building a fresh, timeout-less HTTP client per request — a full TLS handshake to api.anthropic.com every call, and a hung upstream that pinned the task forever. It now uses the shared pooled client. Keychain lookups are memoized, so securityd is hit once per provider, not once per request. Agent budget buckets are capped, so a header-supplied id can’t grow the map without bound. And the one Windu flagged: a non-loopback caller can no longer bypass the auth gate with a throwaway Authorization header.

Verified, not asserted

All 63 proxy unit tests pass, including a new regression that a non-loopback peer with a garbage Authorization header gets a 401. The end-to-end payload test — a 150MB body through /v1/messages — runs against the live :4040 daemon and must be re-run after deploy: it validates the running binary, not the built one.

Deploy shape

The binary the launchd daemon runs is ~/.sanctum/bin/proxyd, which proxyd-launch execs after loading provider keys from ~/.sanctum/secrets/. The launcher now carries a singleton guard. It defers to a healthy incumbent on :4040 rather than pkill-ing it — which is what caused the old mutual-kill bind-race between the user LaunchAgent and the system LaunchDaemon. Deploys go through deploy_proxyd.sh: back up the live binary, copy the fresh build in, launchctl kickstart -k system/com.sanctum.proxyd, wait for /health, and auto-roll-back if it does not come up in 30s. The fallback-continuity design is where this behavior gets its contract.

Still open

One thing survived the night. claude-3-5-sonnet-or in config.yaml maps to google/gemini-1.5-pro, which is retired from OpenRouter, with a nonsensical max_tokens: 2000000. It is an orphan seat — no fallback chain references it — so it causes no live failures. Delete it, or repoint it to google/gemini-2.5-pro.

That is the whole ambition here. A gateway that, when it goes down, has the decency to tell you why — and lands on the next model, not on the floor.