diff options
| author | Peter Stone <thepeterstone@gmail.com> | 2026-08-04 20:10:25 +0000 |
|---|---|---|
| committer | Peter Stone <thepeterstone@gmail.com> | 2026-08-04 20:10:25 +0000 |
| commit | 0a410243dea33f204764000be81814e541dcae48 (patch) | |
| tree | f91f8312af0bc70e5da8c254635118efe811dbef /.agent/worklog.md | |
| parent | 2da86b009c76bb7688103da49a9125d35d3a1ed8 (diff) | |
Fix production wedge: propagate context to Google Calendar API calls
Three .Do() calls in google_calendar.go accepted a ctx parameter but
never chained .Context(ctx) into the actual SDK call, so the existing
global 60s request timeout never reached the blocking network call.
One hung Google Calendar request wedged every DB/session-touching
request path in production for three days (2026-08-01 through
2026-08-04), undetected because /health unconditionally returned 200
throughout.
- Wire .Context(ctx) into GetUpcomingEvents, GetEventsByDateRange, and
GetCalendarList.
- Bound aggregateData's four external fetches with a per-fetch
sub-context as defense-in-depth (only effective if the callee
actually honors ctx -- documented as such, not oversold).
- Make GetUpcomingEvents/GetEventsByDateRange fetch calendars
concurrently instead of sequentially: a review of this fix caught
that a shared per-fetch deadline over a sequential loop would starve
calendars past the first under any real latency, silently caching
partial results as complete. Concurrent fetches give every calendar
an equal shot at the same deadline instead.
- /health now does a real PingContext DB check instead of a static
"ok" (Handler.PingDB, tested for both healthy and closed-DB cases).
- Add internal/api/context_audit_test.go: an AST-based structural
guard that fails any future .Do() call in google_*.go missing
.Context(...) anywhere in its chain, so this class of bug can't
silently recur. Verified by deliberately reintroducing the original
bug against a backup and confirming the guard catches it.
- Add scripts/health-watchdog.sh: cron job restarts the service if
/health fails twice in a row, five minutes apart.
go test ./... -race is green. Deployed and live-verified.
Diffstat (limited to '.agent/worklog.md')
| -rw-r--r-- | .agent/worklog.md | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/.agent/worklog.md b/.agent/worklog.md index d41438c..66aeab8 100644 --- a/.agent/worklog.md +++ b/.agent/worklog.md @@ -4,6 +4,7 @@ Cleaned Backlog ## Recently Completed +- **Production incident: server wedged 3 days, root cause + structural fix** — three `.Do()` calls in `google_calendar.go` (`GetUpcomingEvents`, `GetEventsByDateRange`, `GetCalendarList`) accepted a `ctx` param but never chained `.Context(ctx)` into the actual Google API SDK call, so the pre-existing global 60s `middleware.Timeout` never reached the blocking network call — one hung Google Calendar request wedged every DB/session-touching request path for 3 days starting 2026-08-01, undetected because `/health` unconditionally returned 200 the whole time. Fixed: wired `.Context(ctx)` into all three call sites; `aggregateData`'s 4 external fetches (Trello/PlanToEat/Calendar/Tasks) each now get a bounded per-fetch sub-context (`config.HTTPClientTimeout`) as defense-in-depth (documented as exactly that in the comment — it only helps if the callee honors ctx, doesn't independently guarantee anything); `/health` now does a real `PingContext` DB check instead of a static "ok" (`Handler.PingDB`, tested for both healthy and closed-DB cases); a new `scripts/health-watchdog.sh` cron job (every 5 min) restarts the service if `/health` fails twice in a row. Caught by a from-scratch multi-angle code review before commit: the per-fetch timeout, applied to `GetUpcomingEvents`'s sequential per-calendar loop, would have starved calendars past the first under any real latency (and the swallowed per-calendar error meant that got silently cached as "zero events" — a live regression against this account's 3 configured calendars, never shipped). Fixed by making the per-calendar loop concurrent (`fetchAllCalendars`, shared `sync.WaitGroup`) instead of sequential, so every calendar gets an equal shot at the same deadline instead of eating into the next one's share of it — without touching the pre-existing, deliberately-tested "per-calendar errors are logged and skipped, not propagated" contract. Added `internal/api/context_audit_test.go`, an AST-based structural guard that fails any future `.Do()` call in `google_*.go` missing `.Context(...)` anywhere in its chain — verified by deliberately reintroducing the original bug against a real backup and confirming the guard catches it, then restoring. `go test ./... -race` green. Deployed server; Android quick-add fix (was silently no-op-ing on failure, plus its 5 post-`addTask` follow-up calls were still unchecked) deployed separately as `doot-widget.apk`. - **Bucket CRUD + read-only Projects/Labels on the Settings page** — new "Maintenance Buckets" section: create a bucket (name/cycle_days/pick_n), add a pool item by title (creates the task and assigns it in one step via a new form), remove an item, delete a bucket (unbuckets its tasks rather than deleting them). New store methods `GetBucketItems`/`DeleteBucket`. New read-only "Projects" and "Labels" sections (name + color swatch) -- simple enough that read-only was the right call, per user direction, rather than duplicating the Android popup's editing UX on web. Fully covered by store+handler tests (`go test ./...` green). Not yet deployed. - **Tasks tab rework: Chains section, chain checklist modal, chain-completion lock guard, project-name visibility** — the flat Tasks-tab atom list was silently dumping every chain step (locked and unlocked) and dormant bucket-pool items into the grid as ordinary undated cards, with no chain/project context and, worse, a raw Complete checkbox that `CompleteNativeTask` didn't actually guard against for locked steps (would have corrupted the WIP-1 invariant if clicked). Fixed: `CompleteNativeTask` now returns `ErrChainTaskLocked` for a locked chain task (mapped to 400 in both the widget and web complete-atom handlers); chain tasks (locked or unlocked) and dormant bucket items are excluded from the flat atom list entirely; a new "Chains" section on the Tasks tab shows one card per active/paused chain (`GetChains`, `BuildChainSummaries`) with the current step, N/M progress, and a click-through to a new modal (`GET /chains/{id}`, `chain-detail.html`) listing every position in order (locked/unlocked/completed) with pause/resume/abandon buttons -- the web checklist view originally deferred as Android-only. Along the way, found and fixed a real bug this surfaced: resuming a paused chain only flipped the status flag, never actually unlocking the deferred successor, so a chain paused right after a completion would stay stuck forever -- `SetChainStatus` now catches up the deferred advancement on resume, idempotently (no due-date reset if nothing was actually stuck). Regular atom cards also gained a project-name chip for general visibility. Fully covered by store+handler tests (`go test ./...` green). Not yet deployed. - **Linear task chains + recurring maintenance buckets** — implemented the last two items from `[[doot-future-task-scheduling-ideas]]` (items 1 and 2, budgets/availability and labels/projects, turned out to already be shipped -- their spec status headers and the budgets plan's checkboxes had just never been updated to say so; corrected both). Chains: `task_chains` table + `chain_id`/`chain_position`/`chain_unlocked` on `native_tasks` (migration 026), WIP-limit-1 sequencing wired into `CompleteNativeTask`'s existing recurrence-hook pattern, locked tasks excluded from all date-based queries, 5 new `/api/widget/chains*` endpoints, a position badge ("N/M") on web timeline rows and the Android widget row. Buckets: `maintenance_buckets` table + `bucket_id`/`bucket_state`/`bucket_last_active_at` on `native_tasks` (migration 027), staleness-then-priority selection scoring, a new `RunBucketCycleCheck` scheduler loop mirroring `RunRecurrenceCheck`, 5 new endpoints including the distinct Defer action (returns to pool without crediting completion, unlike Complete), a Defer button on both web timeline rows and the Android widget row. Both fully covered by store+handler tests (`go test ./...` green). Deferred (backend/API exists, UI doesn't): a dedicated Android chain-checklist screen, and bucket-management create/add-item screens on web or Android -- both explicitly out of scope in their specs' own interviews. Not yet deployed or built into the Android APK. |
