From 36a82d934366cc2ddbd92ee3885b1c547eeb8244 Mon Sep 17 00:00:00 2001 From: Peter Stone Date: Sun, 20 Sep 2026 00:36:17 +0000 Subject: Fold the DB-ping health check in; delete the watchdog that never ran MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit scripts/health-watchdog.sh was written after the 2026-08-04 incident, where a liveness stub stayed green for three days while every database-touching request path was wedged. Its header says "Meant to run from cron every few minutes". It was never scheduled — not in any unit, not in any crontab — so it has been dead code since August, offering coverage it did not provide. Its one real idea is worth keeping: /health pings the database and returns 503 when that fails, which the gateway route checks cannot see, because `/` answers 303 to the login page without going near the database. check_app_health now does that poll (two attempts, five seconds apart, so a single blip is not an outage), verify-deploy asserts it, and health-check recovers from it by restarting the main service — the action the old watchdog would have taken. The dead script is removed rather than left to imply cover. Tests now 11, including a stub server returning 503 to prove a listening-but- wedged app is caught. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B --- scripts/health-watchdog.sh | 33 --------------------------------- 1 file changed, 33 deletions(-) delete mode 100755 scripts/health-watchdog.sh (limited to 'scripts/health-watchdog.sh') diff --git a/scripts/health-watchdog.sh b/scripts/health-watchdog.sh deleted file mode 100755 index 9fde6a5..0000000 --- a/scripts/health-watchdog.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/bash -# Health watchdog for the doot server. -# -# /health pings the database (see cmd/dashboard/main.go) rather than -# unconditionally returning 200 -- it used to be a liveness stub that -# stayed green for three days during the 2026-08-04 incident while every -# DB-touching request path was wedged. This polls it and restarts the -# service if it's stuck. Meant to run from cron every few minutes. -# -# Usage: ./scripts/health-watchdog.sh - -APP_URL="http://127.0.0.1:38080/health" -SERVICE="task-dashboard@doot.terst.org.service" -LOG="/var/log/doot-health-watchdog.log" -TIMEOUT=5 - -check() { - curl -s -o /dev/null -w "%{http_code}" --max-time "$TIMEOUT" "$APP_URL" 2>/dev/null -} - -# Two attempts, 5s apart -- don't restart on a single transient blip. -code=$(check) -if [ "$code" != "200" ]; then - sleep 5 - code=$(check) -fi -if [ "$code" = "200" ]; then - exit 0 -fi - -echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') UNHEALTHY (http_code=${code:-timeout}) -- restarting ${SERVICE}" >> "$LOG" -systemctl restart "$SERVICE" -echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') restart issued" >> "$LOG" -- cgit v1.2.3