Verify deploys, and add a self-recovering health check

Deploy used to end with `systemctl status`, which a human skims and a script
ignores: it exits zero whether or not anything works, and says nothing about
the sub-services behind the gateway. A deploy could therefore report success
while /scout/ served 502s — which is exactly what happened, undetected, for
three and a half weeks.

scripts/deploy now:
  * syncs scripts/ to ${SITE_DIR}/app-scripts/ so the timer runs the deployed
    version from a stable path
  * restarts sub-services that are not active. They are separate units and a
    doot deploy does not touch them, so one sitting in `failed` after
    exhausting its restart limit stays down through any number of green
    deploys.
  * calls scripts/verify-deploy, which asserts and exits non-zero

scripts/verify-deploy asserts all five binaries are executable, all four units
are active, all four gateway routes answer without a 5xx, and the running build
stamp matches the commit just built.

scripts/health-check runs the same assertions on a 5-minute timer and recovers
what it can: chmod +x a binary that lost its execute bit, reset-failed and
start a unit that gave up. There is no mail/ntfy/apprise on this host, so
alerting is a greppable HEALTH ALERT line in the journal, a status JSON at
data/health-status.json for the dashboard to surface later, and a non-zero exit
so the unit shows in `systemctl --failed`. OnBootSec=2min because a reboot is
when this failure mode first appeared.

Shared assertions live in scripts/lib/site-checks.sh so verify-deploy and
health-check cannot drift apart.

Two bugs were found and fixed in these checks by testing their failure paths,
both of which would have made them report a broken site as healthy:
  * `$(curl ... || echo 000)` concatenates curl's own "000" transport-error
    output with the echoed one, giving "000000", which matched no case arm and
    was reported ok.
  * check_services sets FAILED_SERVICES, but health-check captured it with
    `$(...)`, so the array was populated in a subshell and discarded — silently
    disabling service recovery entirely. Both the calling convention and the
    subshell semantics are now pinned by tests.

scripts/test-health-checks.sh: 9 cases, mostly negative, against a temp tree.
Recovery additionally verified live on this host by stopping hawaii.service and
by clearing scout's execute bit; both were detected, recovered and re-verified.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

1 files changed, 33 insertions, 2 deletions

diff --git a/scripts/deploy b/scripts/deploy
index 6579839..5ec6175 100755
--- a/scripts/deploy
+++ b/scripts/deploy
@@ -32,6 +32,9 @@ rsync -av --delete web/static/ ${SITE_DIR}/public/
 rsync -av --delete web/templates/ ${SITE_DIR}/templates/
 # Sync migrations
 rsync -av --delete migrations/ ${SITE_DIR}/migrations/
+# Sync operational scripts so the health-check timer runs the current version
+# from a stable path rather than whatever happens to be in a source checkout.
+rsync -av --delete scripts/ ${SITE_DIR}/app-scripts/
 
 echo "==> Swapping binary..."
 mv app ${SITE_DIR}/app
@@ -71,9 +74,37 @@ for svc_bin in ${SITE_DIR}/services/*/app ${SITE_DIR}/bin/*; do
     chmod +x "$svc_bin"
 done
 
+# The synced operational scripts need to stay executable too — the health-check
+# timer ExecStarts one of them, and a non-executable ExecStart is the same
+# 203/EXEC failure this whole change exists to prevent.
+chmod +x ${SITE_DIR}/app-scripts/deploy \
+         ${SITE_DIR}/app-scripts/verify-deploy \
+         ${SITE_DIR}/app-scripts/health-check 2>/dev/null || true
+
 echo "==> Starting service..."
 systemctl start ${SERVICE}
-sleep 1
-systemctl status ${SERVICE} --no-pager -l
+sleep 2
+
+# Sub-services are separate units and are NOT restarted by deploying doot. If
+# one is sitting in `failed` after exhausting its restart limit, it stays down
+# through any number of successful deploys — bring them back here so a deploy
+# leaves the whole site healthy, not just the main app.
+echo "==> Ensuring sub-services are running..."
+for unit in scout.service hawaii.service claudomator@${FQDN}.service; do
+    systemctl list-unit-files "${unit}" >/dev/null 2>&1 || continue
+    if [ "$(systemctl is-active "${unit}" 2>/dev/null || true)" != "active" ]; then
+        echo "    ${unit} is not active — restarting"
+        systemctl reset-failed "${unit}" 2>/dev/null || true
+        systemctl start "${unit}" || echo "    WARNING: could not start ${unit}"
+    fi
+done
+sleep 3
+
+# Assert the outcome instead of printing a status block for a human to skim.
+# `systemctl status` exits zero whether or not anything works and says nothing
+# about the gateway's sub-services, so a deploy could report success while
+# /scout/ served 502s.
+echo "==> Verifying deploy..."
+"$(dirname "$0")/verify-deploy" "${BUILD_COMMIT}"
 
 echo "==> Deploy complete!"