From 0a410243dea33f204764000be81814e541dcae48 Mon Sep 17 00:00:00 2001 From: Peter Stone Date: Tue, 4 Aug 2026 20:10:25 +0000 Subject: Fix production wedge: propagate context to Google Calendar API calls Three .Do() calls in google_calendar.go accepted a ctx parameter but never chained .Context(ctx) into the actual SDK call, so the existing global 60s request timeout never reached the blocking network call. One hung Google Calendar request wedged every DB/session-touching request path in production for three days (2026-08-01 through 2026-08-04), undetected because /health unconditionally returned 200 throughout. - Wire .Context(ctx) into GetUpcomingEvents, GetEventsByDateRange, and GetCalendarList. - Bound aggregateData's four external fetches with a per-fetch sub-context as defense-in-depth (only effective if the callee actually honors ctx -- documented as such, not oversold). - Make GetUpcomingEvents/GetEventsByDateRange fetch calendars concurrently instead of sequentially: a review of this fix caught that a shared per-fetch deadline over a sequential loop would starve calendars past the first under any real latency, silently caching partial results as complete. Concurrent fetches give every calendar an equal shot at the same deadline instead. - /health now does a real PingContext DB check instead of a static "ok" (Handler.PingDB, tested for both healthy and closed-DB cases). - Add internal/api/context_audit_test.go: an AST-based structural guard that fails any future .Do() call in google_*.go missing .Context(...) anywhere in its chain, so this class of bug can't silently recur. Verified by deliberately reintroducing the original bug against a backup and confirming the guard catches it. - Add scripts/health-watchdog.sh: cron job restarts the service if /health fails twice in a row, five minutes apart. go test ./... -race is green. Deployed and live-verified. --- cmd/dashboard/main.go | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) (limited to 'cmd/dashboard') diff --git a/cmd/dashboard/main.go b/cmd/dashboard/main.go index cb2b757..89d12d1 100644 --- a/cmd/dashboard/main.go +++ b/cmd/dashboard/main.go @@ -209,9 +209,25 @@ func main() { // Rate limiter for agent auth (stricter - 10 requests/minute per IP) agentAuthRateLimiter := appmiddleware.NewRateLimiter(10, time.Minute) - // Health check (no auth) + // Health check (no auth). Actually pings the database rather than + // unconditionally returning 200 -- a health check that can't fail isn't + // a health check, it's a liveness stub. That distinction is exactly what + // let the 2026-08-04 incident (DB/session-touching requests wedged for + // three days) go undetected: this endpoint said "ok" the entire time. + // Note the scope: this proves the DB is reachable, not that every request + // path is healthy -- a hung external API call that doesn't touch the DB + // (the actual incident's mechanism) wouldn't be caught by this alone. + // That class of bug is addressed structurally instead, by requiring every + // Google API call to propagate context (see internal/api/context_audit_test.go). r.Get("/health", func(w http.ResponseWriter, r *http.Request) { + ctx, cancel := context.WithTimeout(r.Context(), config.HealthCheckTimeout) + defer cancel() w.Header().Set("Content-Type", "text/plain") + if err := h.PingDB(ctx); err != nil { + w.WriteHeader(http.StatusServiceUnavailable) + _, _ = w.Write([]byte("db unreachable: " + err.Error())) + return + } w.WriteHeader(http.StatusOK) _, _ = w.Write([]byte("ok")) }) -- cgit v1.2.3