Fold the DB-ping health check in; delete the watchdog that never ran

scripts/health-watchdog.sh was written after the 2026-08-04 incident, where a
liveness stub stayed green for three days while every database-touching request
path was wedged. Its header says "Meant to run from cron every few minutes". It
was never scheduled — not in any unit, not in any crontab — so it has been dead
code since August, offering coverage it did not provide.

Its one real idea is worth keeping: /health pings the database and returns 503
when that fails, which the gateway route checks cannot see, because `/` answers
303 to the login page without going near the database.

check_app_health now does that poll (two attempts, five seconds apart, so a
single blip is not an outage), verify-deploy asserts it, and health-check
recovers from it by restarting the main service — the action the old watchdog
would have taken. The dead script is removed rather than left to imply cover.

Tests now 11, including a stub server returning 503 to prove a listening-but-
wedged app is caught.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B

1 files changed, 0 insertions, 33 deletions

diff --git a/scripts/health-watchdog.sh b/scripts/health-watchdog.sh
deleted file mode 100755
index 9fde6a5..0000000
--- a/scripts/health-watchdog.sh
+++ /dev/null
@@ -1,33 +0,0 @@
-#!/bin/bash
-# Health watchdog for the doot server.
-#
-# /health pings the database (see cmd/dashboard/main.go) rather than
-# unconditionally returning 200 -- it used to be a liveness stub that
-# stayed green for three days during the 2026-08-04 incident while every
-# DB-touching request path was wedged. This polls it and restarts the
-# service if it's stuck. Meant to run from cron every few minutes.
-#
-# Usage: ./scripts/health-watchdog.sh
-
-APP_URL="http://127.0.0.1:38080/health"
-SERVICE="task-dashboard@doot.terst.org.service"
-LOG="/var/log/doot-health-watchdog.log"
-TIMEOUT=5
-
-check() {
-    curl -s -o /dev/null -w "%{http_code}" --max-time "$TIMEOUT" "$APP_URL" 2>/dev/null
-}
-
-# Two attempts, 5s apart -- don't restart on a single transient blip.
-code=$(check)
-if [ "$code" != "200" ]; then
-    sleep 5
-    code=$(check)
-fi
-if [ "$code" = "200" ]; then
-    exit 0
-fi
-
-echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') UNHEALTHY (http_code=${code:-timeout}) -- restarting ${SERVICE}" >> "$LOG"
-systemctl restart "$SERVICE"
-echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') restart issued" >> "$LOG"