Backup Dead-Man's-Switch

The manoir backup is healthy. Two restic repos — the T9 drive and an offsite — both fresh daily, both proven restorable by a weekly drill (see Backup & Restore). None of that is the interesting part. The interesting part is that a different backup — the cross-machine snapshot onto the satellite — stopped running and went nearly seven weeks before anyone noticed. It did not fail loudly. It simply went quiet, and the silence looked exactly like success.

That is the real bug. A backup you cannot trust to tell you it broke is not a backup; it is a hope. This is the same lesson Windu learned building the Drift Sentinel: the failures that hurt are the ones no log ever mentions. What follows is that lesson aimed at the one place an on-box guard can never reach.

Why every on-box guard is blind

The backup already alerts on failure — it notifies Force Flow when a run errors or times out. The weekly canary already restores a probe file and checks its hash. Both are good. Both share one fatal property: they run on the same Mac they protect.

When launchd silently drops a timer (which it did, fleet-wide, for weeks), the job never runs, so there is no failure to alert on. When the host sleeps, crashes, or is carried off, every watcher on it sleeps too. When Force Flow itself is down, the alert falls into a hole. The guard and the thing it guards die together. The seven-week silence was not one bug — it was every on-box guard on that Mac failing in the same breath, and no one outside the box was left to hear it.

The absence is the signal

The fix is a watcher that cannot die with the haus. It lives on GitHub:

backup succeeds  ->  sanctum deadman beat backup-fresh
canary passes    ->  sanctum deadman beat restore-drill
                        |
                        v   writes heartbeats.json on GitHub (Ogilthorp3/sanctum-backup-deadman)

hourly GitHub Action reads heartbeats.json:
   any heartbeat older than its budget?
      yes  ->  open a GitHub issue (which emails the owner); the run stays green
      no   ->  close the issue (the backup recovered)

Note the inversion. The backup does not report failure — it reports success, by writing a fresh timestamp. The watcher alarms on the absence of that timestamp. Failure, didn’t-run, host-dead, and network-cut all collapse into the same observable: no fresh heartbeat. There is no failure mode the watcher can sleep through, because it is not watching the Mac at all — it watches a file on GitHub’s infrastructure and asks one question every hour: is it stale? The run’s own colour reflects the watcher’s health, not the backup’s — a stale backup is reported by the issue, and the build goes red only if the watcher itself cannot post.

The beat

A host reports success with one command:

sanctum deadman beat backup-fresh     # the daily restic backup, on success
sanctum deadman beat restore-drill    # the weekly restore canary, on a passing drill

beat writes the heartbeat through the GitHub Contents API using the host’s existing gh token — no clone, no SSH agent, no new secret. That matters: the backup runs as a background LaunchAgent, where an interactive SSH agent may not be reachable, but a token read from a file always is. The beat is best-effort from the backup’s point of view — a failed beat warns but never fails the backup, because the off-box Action catches the resulting missing heartbeat anyway.

Each check carries its own staleness budget, stored next to its timestamp: backup-fresh is overdue after 26 hours (a daily job plus slack), restore-drill after 192 hours (eight days). The watcher needs no configuration of its own — the heartbeat file tells it everything.

Proven, not asserted

The happy path is easy to believe. The alarm path is the one that has to work the day everything else has failed, so it was exercised for real, not stubbed:

• A heartbeat was forced 100 hours stale. The Action opened the issue (the email arrived) and stayed green — a stale backup is an alert, not a broken build.
• A real beat refreshed the heartbeat. The next run closed the issue.

The one rule

A backup must prove three things, continuously: that it is fresh, that it is restorable, and that its proof is visible from somewhere it cannot silence. The first two were already true on manoir. The third is what made the silence possible — and what this closes.

The beat that did not fire

Then the alarm path stopped being a rehearsal: over two days the haus collected its first real receipts. The eye paged manoir:backup-fresh at 38h on 2026-08-13. Ark had saved. gdrive had not. Three stacked holes:

1. rclone timeout looked like “no repo”. ensure_repo ran restic init on the existing cloud repo, init failed, die() skipped the beat. Fixed: snapshots probe is capped (180s) and a timeout or unreachable repo never inits.
2. Calendar-only, no catch-up. 05:00 failed; the unexpected 16:56 reboot did not retry. Next fire was tomorrow. Fixed: sanctum-backup.sh --if-stale 20 on com.sanctum.backup with RunAtLoad. Both repos fresh → re-beat only. Either stale or unread → full backup. The canary, if it still sees stale, waits for the backup lock and kicks --if-stale once.
3. Cold Drive token at 05:00. Recurred 2026-08-14: restic’s rclone helper died on context deadline exceeded before headers. The new path refused to init (correct) and still did not beat (correct) — but gdrive stayed on the 08-13 21:01 snapshot. Fixed: rclone lsf preflight (3 tries) refreshes the token before restic opens its localhost helper; gdrive backup itself retries 3 times.

gdrive is required for the beat. A missing remote or a failed cloud pass is fatal for backup-fresh. That used to be documented as “best-effort.” It was a lie.

sanctum-backup.sh --if-stale 20     # launchd RunAtLoad + 05:00
sanctum-backup.sh                   # force a full pass

A backup that goes quiet now has exactly one place left to hide, and it is a place the haus does not control. The next time a timer slips, you do not find out seven weeks later from a restore that fails — you find out the next morning, from an email an off-box watcher sent while your Mac was still asleep. That was the entire point.