From 36a82d934366cc2ddbd92ee3885b1c547eeb8244 Mon Sep 17 00:00:00 2001 From: Peter Stone Date: Sun, 20 Sep 2026 00:36:17 +0000 Subject: Fold the DB-ping health check in; delete the watchdog that never ran MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit scripts/health-watchdog.sh was written after the 2026-08-04 incident, where a liveness stub stayed green for three days while every database-touching request path was wedged. Its header says "Meant to run from cron every few minutes". It was never scheduled — not in any unit, not in any crontab — so it has been dead code since August, offering coverage it did not provide. Its one real idea is worth keeping: /health pings the database and returns 503 when that fails, which the gateway route checks cannot see, because `/` answers 303 to the login page without going near the database. check_app_health now does that poll (two attempts, five seconds apart, so a single blip is not an outage), verify-deploy asserts it, and health-check recovers from it by restarting the main service — the action the old watchdog would have taken. The dead script is removed rather than left to imply cover. Tests now 11, including a stub server returning 503 to prove a listening-but- wedged app is caught. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B --- scripts/health-check | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) (limited to 'scripts/health-check') diff --git a/scripts/health-check b/scripts/health-check index b1c34d0..29481ce 100755 --- a/scripts/health-check +++ b/scripts/health-check @@ -84,15 +84,39 @@ fi check_routes; route_rc=$? [ $route_rc -ne 0 ] && rc=1 +# ── app health (database ping) ──────────────────────────────────────────────── +# Separate from the route checks: the app can be listening and answering +# redirects while every database-touching path is wedged. That is what happened +# on 2026-08-04, undetected for three days. Recovery is a restart of the main +# service, which is what the (never-scheduled) health-watchdog.sh this replaces +# was written to do. +check_app_health; health_rc=$? +if [ $health_rc -ne 0 ]; then + rc=1 + if [ $recover -eq 1 ]; then + log "HEALTH recovering: restarting task-dashboard@doot.terst.org.service (db ping failing)" + systemctl restart task-dashboard@doot.terst.org.service 2>/dev/null || true + sleep 5 + if check_app_health >/dev/null; then + log "HEALTH recovered: app health restored after restart" + recovered+=("restarted task-dashboard (db ping)") + else + log "HEALTH ALERT: app health still failing after restart" + still_broken+=("task-dashboard@doot.terst.org.service (db ping)") + fi + fi +fi + # A recovered site is healthy now even though it was not a moment ago. Re-run # the checks so the final verdict describes the current state, not the state we # walked in on. if [ $rc -ne 0 ] && [ ${#recovered[@]} -gt 0 ]; then echo "==> Re-checking after recovery..." rc=0 - check_binaries >/dev/null || rc=1 - check_services >/dev/null || rc=1 - check_routes >/dev/null || rc=1 + check_binaries >/dev/null || rc=1 + check_services >/dev/null || rc=1 + check_routes >/dev/null || rc=1 + check_app_health >/dev/null || rc=1 fi # ── status file ─────────────────────────────────────────────────────────────── -- cgit v1.2.3