Verify deploys, and add a self-recovering health check

Deploy used to end with `systemctl status`, which a human skims and a script
ignores: it exits zero whether or not anything works, and says nothing about
the sub-services behind the gateway. A deploy could therefore report success
while /scout/ served 502s — which is exactly what happened, undetected, for
three and a half weeks.

scripts/deploy now:
  * syncs scripts/ to ${SITE_DIR}/app-scripts/ so the timer runs the deployed
    version from a stable path
  * restarts sub-services that are not active. They are separate units and a
    doot deploy does not touch them, so one sitting in `failed` after
    exhausting its restart limit stays down through any number of green
    deploys.
  * calls scripts/verify-deploy, which asserts and exits non-zero

scripts/verify-deploy asserts all five binaries are executable, all four units
are active, all four gateway routes answer without a 5xx, and the running build
stamp matches the commit just built.

scripts/health-check runs the same assertions on a 5-minute timer and recovers
what it can: chmod +x a binary that lost its execute bit, reset-failed and
start a unit that gave up. There is no mail/ntfy/apprise on this host, so
alerting is a greppable HEALTH ALERT line in the journal, a status JSON at
data/health-status.json for the dashboard to surface later, and a non-zero exit
so the unit shows in `systemctl --failed`. OnBootSec=2min because a reboot is
when this failure mode first appeared.

Shared assertions live in scripts/lib/site-checks.sh so verify-deploy and
health-check cannot drift apart.

Two bugs were found and fixed in these checks by testing their failure paths,
both of which would have made them report a broken site as healthy:
  * `$(curl ... || echo 000)` concatenates curl's own "000" transport-error
    output with the echoed one, giving "000000", which matched no case arm and
    was reported ok.
  * check_services sets FAILED_SERVICES, but health-check captured it with
    `$(...)`, so the array was populated in a subshell and discarded — silently
    disabling service recovery entirely. Both the calling convention and the
    subshell semantics are now pinned by tests.

scripts/test-health-checks.sh: 9 cases, mostly negative, against a temp tree.
Recovery additionally verified live on this host by stopping hawaii.service and
by clearing scout's execute bit; both were detected, recovered and re-verified.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

1 files changed, 134 insertions, 0 deletions

diff --git a/scripts/test-health-checks.sh b/scripts/test-health-checks.sh
new file mode 100755
index 0000000..fbe9e47
--- /dev/null
+++ b/scripts/test-health-checks.sh
@@ -0,0 +1,134 @@
+#!/bin/bash
+#
+# Tests for scripts/lib/site-checks.sh.
+#
+# These drive each check into a failing state and assert it is caught. A checker
+# that only ever reports "ok" would have passed happily through the three-and-a-
+# half-week outage these scripts exist to prevent, so the negative cases are the
+# whole point.
+#
+# Nothing here touches the real site: SITE_DIR and BASE_URL are overridden to a
+# temp tree and a dead port.
+
+set -uo pipefail
+cd "$(dirname "$0")/.."
+
+pass=0; fail=0
+ok()  { printf '  PASS  %s\n' "$1"; pass=$((pass+1)); }
+bad() { printf '  FAIL  %s\n     -> %s\n' "$1" "$2"; fail=$((fail+1)); }
+
+tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
+
+# Build a fake site tree with all five binaries present and executable.
+mkdir -p "$tmp/bin" "$tmp/services/scout" "$tmp/services/hawaii" "$tmp/data"
+for f in "$tmp/app" "$tmp/bin/claudomator" "$tmp/bin/claude" \
+         "$tmp/services/scout/app" "$tmp/services/hawaii/app"; do
+    touch "$f"; chmod +x "$f"
+done
+
+# Load the library against the fake tree. Routes point at a closed port so the
+# route check exercises its transport-error path rather than hitting the network.
+SITE_DIR="$tmp"
+BASE_URL="http://127.0.0.1:9"
+# shellcheck source=scripts/lib/site-checks.sh
+. scripts/lib/site-checks.sh
+
+printf '=== check_binaries ===\n'
+
+out=$(check_binaries); rc=$?
+if [ $rc -eq 0 ]; then ok "passes when every binary is executable"
+else bad "all-executable passes" "rc=$rc"; fi
+
+chmod -x "$tmp/services/scout/app"
+out=$(check_binaries); rc=$?
+if [ $rc -ne 0 ] && printf '%s' "$out" | grep -q "not executable.*scout/app"; then
+    ok "catches a binary that lost its execute bit"
+else
+    bad "lost +x detected" "rc=$rc out=$(printf '%s' "$out" | tr '\n' ' ')"
+fi
+chmod +x "$tmp/services/scout/app"
+
+mv "$tmp/bin/claude" "$tmp/bin/claude.moved"
+out=$(check_binaries); rc=$?
+if [ $rc -ne 0 ] && printf '%s' "$out" | grep -q "missing binary.*claude"; then
+    ok "catches a missing binary"
+else
+    bad "missing binary detected" "rc=$rc out=$(printf '%s' "$out" | tr '\n' ' ')"
+fi
+mv "$tmp/bin/claude.moved" "$tmp/bin/claude"
+
+out=$(check_binaries); rc=$?
+[ $rc -eq 0 ] && ok "recovers to passing once restored" \
+              || bad "restore" "rc=$rc"
+
+printf '=== check_routes ===\n'
+
+out=$(check_routes); rc=$?
+if [ $rc -ne 0 ] && printf '%s' "$out" | grep -q "route / -> 000"; then
+    ok "catches an unreachable gateway (transport error, not a status code)"
+else
+    bad "unreachable gateway" "rc=$rc out=$(printf '%s' "$out" | tr '\n' ' ')"
+fi
+
+# A 502 from a live server is the shape of a dead sub-service. Serve one.
+python3 - "$tmp/port" <<'PY' &
+import http.server, socketserver, sys, threading
+class H(http.server.BaseHTTPRequestHandler):
+    def do_GET(self):
+        self.send_response(502); self.end_headers()
+    def log_message(self, *a): pass
+with socketserver.TCPServer(("127.0.0.1", 0), H) as s:
+    open(sys.argv[1], "w").write(str(s.server_address[1]))
+    s.serve_forever()
+PY
+server_pid=$!
+for _ in $(seq 1 50); do [ -s "$tmp/port" ] && break; sleep 0.1; done
+port=$(cat "$tmp/port" 2>/dev/null || echo "")
+
+if [ -n "$port" ]; then
+    BASE_URL="http://127.0.0.1:${port}"
+    out=$(check_routes); rc=$?
+    if [ $rc -ne 0 ] && printf '%s' "$out" | grep -q "route /scout/ -> 502"; then
+        ok "catches a 502 from a dead sub-service upstream"
+    else
+        bad "502 detection" "rc=$rc out=$(printf '%s' "$out" | tr '\n' ' ')"
+    fi
+else
+    bad "502 detection" "could not start stub server"
+fi
+kill "$server_pid" 2>/dev/null; wait "$server_pid" 2>/dev/null
+
+printf '=== check_services ===\n'
+
+# A unit name that cannot exist must be reported, not silently skipped.
+SITE_SERVICES=("definitely-not-a-real-unit-$$.service")
+# Redirected, not captured in $( ): check_services sets FAILED_SERVICES, and a
+# subshell would discard it. health-check drives recovery off that array, so
+# this test also pins the calling convention.
+check_services > "$tmp/svc.out"; rc=$?
+if [ $rc -ne 0 ] && [ "${#FAILED_SERVICES[@]}" -eq 1 ]; then
+    ok "catches a non-running unit and records it for recovery"
+else
+    bad "dead unit detected" "rc=$rc failed=${#FAILED_SERVICES[@]}"
+fi
+
+# Regression guard for the subshell trap itself.
+FAILED_SERVICES=()
+_discard=$(check_services)
+if [ "${#FAILED_SERVICES[@]}" -eq 0 ]; then
+    ok "documents that command substitution loses FAILED_SERVICES (call directly)"
+else
+    bad "subshell semantics" "array unexpectedly survived a subshell"
+fi
+
+printf '=== check_build_stamp ===\n'
+
+out=$(check_build_stamp "definitely-not-the-running-commit"); rc=$?
+if [ $rc -ne 0 ]; then
+    ok "catches a build stamp that does not match what was deployed"
+else
+    bad "build stamp mismatch" "rc=$rc"
+fi
+
+printf '\n=== summary: %d passed, %d failed ===\n' "$pass" "$fail"
+[ "$fail" -eq 0 ]