Verify deploys, and add a self-recovering health check

Deploy used to end with `systemctl status`, which a human skims and a script
ignores: it exits zero whether or not anything works, and says nothing about
the sub-services behind the gateway. A deploy could therefore report success
while /scout/ served 502s — which is exactly what happened, undetected, for
three and a half weeks.

scripts/deploy now:
  * syncs scripts/ to ${SITE_DIR}/app-scripts/ so the timer runs the deployed
    version from a stable path
  * restarts sub-services that are not active. They are separate units and a
    doot deploy does not touch them, so one sitting in `failed` after
    exhausting its restart limit stays down through any number of green
    deploys.
  * calls scripts/verify-deploy, which asserts and exits non-zero

scripts/verify-deploy asserts all five binaries are executable, all four units
are active, all four gateway routes answer without a 5xx, and the running build
stamp matches the commit just built.

scripts/health-check runs the same assertions on a 5-minute timer and recovers
what it can: chmod +x a binary that lost its execute bit, reset-failed and
start a unit that gave up. There is no mail/ntfy/apprise on this host, so
alerting is a greppable HEALTH ALERT line in the journal, a status JSON at
data/health-status.json for the dashboard to surface later, and a non-zero exit
so the unit shows in `systemctl --failed`. OnBootSec=2min because a reboot is
when this failure mode first appeared.

Shared assertions live in scripts/lib/site-checks.sh so verify-deploy and
health-check cannot drift apart.

Two bugs were found and fixed in these checks by testing their failure paths,
both of which would have made them report a broken site as healthy:
  * `$(curl ... || echo 000)` concatenates curl's own "000" transport-error
    output with the echoed one, giving "000000", which matched no case arm and
    was reported ok.
  * check_services sets FAILED_SERVICES, but health-check captured it with
    `$(...)`, so the array was populated in a subshell and discarded — silently
    disabling service recovery entirely. Both the calling convention and the
    subshell semantics are now pinned by tests.

scripts/test-health-checks.sh: 9 cases, mostly negative, against a temp tree.
Recovery additionally verified live on this host by stopping hawaii.service and
by clearing scout's execute bit; both were detected, recovered and re-verified.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

2 files changed, 32 insertions, 0 deletions

diff --git a/deployment/doot-health.service b/deployment/doot-health.service
new file mode 100644
index 0000000..33e8e77
--- /dev/null
+++ b/deployment/doot-health.service
@@ -0,0 +1,18 @@
+[Unit]
+Description=doot.terst.org health check (gateway routes + sub-services)
+Documentation=file:///site/doot.terst.org/app-scripts/health-check
+After=network-online.target
+
+[Service]
+Type=oneshot
+# Runs as root: recovery needs systemctl start and chmod on www-data files.
+User=root
+ExecStart=/site/doot.terst.org/app-scripts/health-check
+
+# A failed run should be visible in `systemctl --failed`, so do not swallow the
+# exit code. The unit going failed is a large part of the alerting here, given
+# there is no mail/ntfy on this host.
+SuccessExitStatus=0
+
+[Install]
+WantedBy=multi-user.target
diff --git a/deployment/doot-health.timer b/deployment/doot-health.timer
new file mode 100644
index 0000000..22b687f
--- /dev/null
+++ b/deployment/doot-health.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Run the doot.terst.org health check every 5 minutes
+
+[Timer]
+# 2 minutes after boot: the failure this exists for first appeared at a reboot,
+# when sub-services could not exec and systemd gave up on them.
+OnBootSec=2min
+OnUnitActiveSec=5min
+# Spread the load off exact-minute boundaries.
+RandomizedDelaySec=30s
+Unit=doot-health.service
+
+[Install]
+WantedBy=timers.target