From c58f7519936dc547ee2aec5bfbb1e477356656d9 Mon Sep 17 00:00:00 2001 From: Peter Stone Date: Sun, 20 Sep 2026 00:34:23 +0000 Subject: Verify deploys, and add a self-recovering health check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deploy used to end with `systemctl status`, which a human skims and a script ignores: it exits zero whether or not anything works, and says nothing about the sub-services behind the gateway. A deploy could therefore report success while /scout/ served 502s — which is exactly what happened, undetected, for three and a half weeks. scripts/deploy now: * syncs scripts/ to ${SITE_DIR}/app-scripts/ so the timer runs the deployed version from a stable path * restarts sub-services that are not active. They are separate units and a doot deploy does not touch them, so one sitting in `failed` after exhausting its restart limit stays down through any number of green deploys. * calls scripts/verify-deploy, which asserts and exits non-zero scripts/verify-deploy asserts all five binaries are executable, all four units are active, all four gateway routes answer without a 5xx, and the running build stamp matches the commit just built. scripts/health-check runs the same assertions on a 5-minute timer and recovers what it can: chmod +x a binary that lost its execute bit, reset-failed and start a unit that gave up. There is no mail/ntfy/apprise on this host, so alerting is a greppable HEALTH ALERT line in the journal, a status JSON at data/health-status.json for the dashboard to surface later, and a non-zero exit so the unit shows in `systemctl --failed`. OnBootSec=2min because a reboot is when this failure mode first appeared. Shared assertions live in scripts/lib/site-checks.sh so verify-deploy and health-check cannot drift apart. Two bugs were found and fixed in these checks by testing their failure paths, both of which would have made them report a broken site as healthy: * `$(curl ... || echo 000)` concatenates curl's own "000" transport-error output with the echoed one, giving "000000", which matched no case arm and was reported ok. * check_services sets FAILED_SERVICES, but health-check captured it with `$(...)`, so the array was populated in a subshell and discarded — silently disabling service recovery entirely. Both the calling convention and the subshell semantics are now pinned by tests. scripts/test-health-checks.sh: 9 cases, mostly negative, against a temp tree. Recovery additionally verified live on this host by stopping hawaii.service and by clearing scout's execute bit; both were detected, recovered and re-verified. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B --- deployment/doot-health.service | 18 ++++++++++++++++++ deployment/doot-health.timer | 14 ++++++++++++++ 2 files changed, 32 insertions(+) create mode 100644 deployment/doot-health.service create mode 100644 deployment/doot-health.timer (limited to 'deployment') diff --git a/deployment/doot-health.service b/deployment/doot-health.service new file mode 100644 index 0000000..33e8e77 --- /dev/null +++ b/deployment/doot-health.service @@ -0,0 +1,18 @@ +[Unit] +Description=doot.terst.org health check (gateway routes + sub-services) +Documentation=file:///site/doot.terst.org/app-scripts/health-check +After=network-online.target + +[Service] +Type=oneshot +# Runs as root: recovery needs systemctl start and chmod on www-data files. +User=root +ExecStart=/site/doot.terst.org/app-scripts/health-check + +# A failed run should be visible in `systemctl --failed`, so do not swallow the +# exit code. The unit going failed is a large part of the alerting here, given +# there is no mail/ntfy on this host. +SuccessExitStatus=0 + +[Install] +WantedBy=multi-user.target diff --git a/deployment/doot-health.timer b/deployment/doot-health.timer new file mode 100644 index 0000000..22b687f --- /dev/null +++ b/deployment/doot-health.timer @@ -0,0 +1,14 @@ +[Unit] +Description=Run the doot.terst.org health check every 5 minutes + +[Timer] +# 2 minutes after boot: the failure this exists for first appeared at a reboot, +# when sub-services could not exec and systemd gave up on them. +OnBootSec=2min +OnUnitActiveSec=5min +# Spread the load off exact-minute boundaries. +RandomizedDelaySec=30s +Unit=doot-health.service + +[Install] +WantedBy=timers.target -- cgit v1.2.3