Fold the DB-ping health check in; delete the watchdog that never ran

scripts/health-watchdog.sh was written after the 2026-08-04 incident, where a
liveness stub stayed green for three days while every database-touching request
path was wedged. Its header says "Meant to run from cron every few minutes". It
was never scheduled — not in any unit, not in any crontab — so it has been dead
code since August, offering coverage it did not provide.

Its one real idea is worth keeping: /health pings the database and returns 503
when that fails, which the gateway route checks cannot see, because `/` answers
303 to the login page without going near the database.

check_app_health now does that poll (two attempts, five seconds apart, so a
single blip is not an outage), verify-deploy asserts it, and health-check
recovers from it by restarting the main service — the action the old watchdog
would have taken. The dead script is removed rather than left to imply cover.

Tests now 11, including a stub server returning 503 to prove a listening-but-
wedged app is caught.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

1 files changed, 27 insertions, 3 deletions

diff --git a/scripts/health-check b/scripts/health-check
index b1c34d0..29481ce 100755
--- a/scripts/health-check
+++ b/scripts/health-check
@@ -84,15 +84,39 @@ fi
 check_routes; route_rc=$?
 [ $route_rc -ne 0 ] && rc=1
 
+# ── app health (database ping) ────────────────────────────────────────────────
+# Separate from the route checks: the app can be listening and answering
+# redirects while every database-touching path is wedged. That is what happened
+# on 2026-08-04, undetected for three days. Recovery is a restart of the main
+# service, which is what the (never-scheduled) health-watchdog.sh this replaces
+# was written to do.
+check_app_health; health_rc=$?
+if [ $health_rc -ne 0 ]; then
+    rc=1
+    if [ $recover -eq 1 ]; then
+        log "HEALTH recovering: restarting task-dashboard@doot.terst.org.service (db ping failing)"
+        systemctl restart task-dashboard@doot.terst.org.service 2>/dev/null || true
+        sleep 5
+        if check_app_health >/dev/null; then
+            log "HEALTH recovered: app health restored after restart"
+            recovered+=("restarted task-dashboard (db ping)")
+        else
+            log "HEALTH ALERT: app health still failing after restart"
+            still_broken+=("task-dashboard@doot.terst.org.service (db ping)")
+        fi
+    fi
+fi
+
 # A recovered site is healthy now even though it was not a moment ago. Re-run
 # the checks so the final verdict describes the current state, not the state we
 # walked in on.
 if [ $rc -ne 0 ] && [ ${#recovered[@]} -gt 0 ]; then
     echo "==> Re-checking after recovery..."
     rc=0
-    check_binaries >/dev/null || rc=1
-    check_services >/dev/null || rc=1
-    check_routes   >/dev/null || rc=1
+    check_binaries   >/dev/null || rc=1
+    check_services   >/dev/null || rc=1
+    check_routes     >/dev/null || rc=1
+    check_app_health >/dev/null || rc=1
 fi
 
 # ── status file ───────────────────────────────────────────────────────────────