|
scripts/health-watchdog.sh was written after the 2026-08-04 incident, where a
liveness stub stayed green for three days while every database-touching request
path was wedged. Its header says "Meant to run from cron every few minutes". It
was never scheduled — not in any unit, not in any crontab — so it has been dead
code since August, offering coverage it did not provide.
Its one real idea is worth keeping: /health pings the database and returns 503
when that fails, which the gateway route checks cannot see, because `/` answers
303 to the login page without going near the database.
check_app_health now does that poll (two attempts, five seconds apart, so a
single blip is not an outage), verify-deploy asserts it, and health-check
recovers from it by restarting the main service — the action the old watchdog
would have taken. The dead script is removed rather than left to imply cover.
Tests now 11, including a stub server returning 503 to prove a listening-but-
wedged app is caught.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Deploy used to end with `systemctl status`, which a human skims and a script
ignores: it exits zero whether or not anything works, and says nothing about
the sub-services behind the gateway. A deploy could therefore report success
while /scout/ served 502s — which is exactly what happened, undetected, for
three and a half weeks.
scripts/deploy now:
* syncs scripts/ to ${SITE_DIR}/app-scripts/ so the timer runs the deployed
version from a stable path
* restarts sub-services that are not active. They are separate units and a
doot deploy does not touch them, so one sitting in `failed` after
exhausting its restart limit stays down through any number of green
deploys.
* calls scripts/verify-deploy, which asserts and exits non-zero
scripts/verify-deploy asserts all five binaries are executable, all four units
are active, all four gateway routes answer without a 5xx, and the running build
stamp matches the commit just built.
scripts/health-check runs the same assertions on a 5-minute timer and recovers
what it can: chmod +x a binary that lost its execute bit, reset-failed and
start a unit that gave up. There is no mail/ntfy/apprise on this host, so
alerting is a greppable HEALTH ALERT line in the journal, a status JSON at
data/health-status.json for the dashboard to surface later, and a non-zero exit
so the unit shows in `systemctl --failed`. OnBootSec=2min because a reboot is
when this failure mode first appeared.
Shared assertions live in scripts/lib/site-checks.sh so verify-deploy and
health-check cannot drift apart.
Two bugs were found and fixed in these checks by testing their failure paths,
both of which would have made them report a broken site as healthy:
* `$(curl ... || echo 000)` concatenates curl's own "000" transport-error
output with the echoed one, giving "000000", which matched no case arm and
was reported ok.
* check_services sets FAILED_SERVICES, but health-check captured it with
`$(...)`, so the array was populated in a subshell and discarded — silently
disabling service recovery entirely. Both the calling convention and the
subshell semantics are now pinned by tests.
scripts/test-health-checks.sh: 9 cases, mostly negative, against a temp tree.
Recovery additionally verified live on this host by stopping hawaii.service and
by clearing scout's execute bit; both were detected, recovered and re-verified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
