| Age | Commit message (Collapse) | Author |
|---|
|
scripts/health-watchdog.sh was written after the 2026-08-04 incident, where a
liveness stub stayed green for three days while every database-touching request
path was wedged. Its header says "Meant to run from cron every few minutes". It
was never scheduled — not in any unit, not in any crontab — so it has been dead
code since August, offering coverage it did not provide.
Its one real idea is worth keeping: /health pings the database and returns 503
when that fails, which the gateway route checks cannot see, because `/` answers
303 to the login page without going near the database.
check_app_health now does that poll (two attempts, five seconds apart, so a
single blip is not an outage), verify-deploy asserts it, and health-check
recovers from it by restarting the main service — the action the old watchdog
would have taken. The dead script is removed rather than left to imply cover.
Tests now 11, including a stub server returning 503 to prove a listening-but-
wedged app is caught.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Deploy used to end with `systemctl status`, which a human skims and a script
ignores: it exits zero whether or not anything works, and says nothing about
the sub-services behind the gateway. A deploy could therefore report success
while /scout/ served 502s — which is exactly what happened, undetected, for
three and a half weeks.
scripts/deploy now:
* syncs scripts/ to ${SITE_DIR}/app-scripts/ so the timer runs the deployed
version from a stable path
* restarts sub-services that are not active. They are separate units and a
doot deploy does not touch them, so one sitting in `failed` after
exhausting its restart limit stays down through any number of green
deploys.
* calls scripts/verify-deploy, which asserts and exits non-zero
scripts/verify-deploy asserts all five binaries are executable, all four units
are active, all four gateway routes answer without a 5xx, and the running build
stamp matches the commit just built.
scripts/health-check runs the same assertions on a 5-minute timer and recovers
what it can: chmod +x a binary that lost its execute bit, reset-failed and
start a unit that gave up. There is no mail/ntfy/apprise on this host, so
alerting is a greppable HEALTH ALERT line in the journal, a status JSON at
data/health-status.json for the dashboard to surface later, and a non-zero exit
so the unit shows in `systemctl --failed`. OnBootSec=2min because a reboot is
when this failure mode first appeared.
Shared assertions live in scripts/lib/site-checks.sh so verify-deploy and
health-check cannot drift apart.
Two bugs were found and fixed in these checks by testing their failure paths,
both of which would have made them report a broken site as healthy:
* `$(curl ... || echo 000)` concatenates curl's own "000" transport-error
output with the echoed one, giving "000000", which matched no case arm and
was reported ok.
* check_services sets FAILED_SERVICES, but health-check captured it with
`$(...)`, so the array was populated in a subshell and discarded — silently
disabling service recovery entirely. Both the calling convention and the
subshell semantics are now pinned by tests.
scripts/test-health-checks.sh: 9 cases, mostly negative, against a temp tree.
Recovery additionally verified live on this host by stopping hawaii.service and
by clearing scout's execute bit; both were detected, recovered and re-verified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The permission step ran
find ${SITE_DIR} -type f -exec chmod 644 {} +
chmod +x ${SITE_DIR}/app
over the whole site directory, so it cleared +x on every other executable there
and restored it only for doot's own app. Four binaries were caught:
services/scout/app scout.service
services/hawaii/app hawaii sub-service
bin/claudomator claudomator@doot.terst.org.service
bin/claude exec'd by claudomator via --claude-bin
Nothing appears to break at deploy time: a running service keeps executing from
its already-open file. The damage surfaces at the next reboot, when systemd
cannot exec the binary and the unit dies with status=203/EXEC. scout.service
went down that way at the 2026-08-25 reboot and stayed down three and a half
weeks, serving 502s behind /scout/; claudomator has been crash-looping on the
same error, 426k restarts and counting. Neither is alerted on, because doot
itself is healthy and only those paths fail.
Permission fixing is now scoped to the directories this script actually manages
(public, templates, migrations) plus doot's own binary and .env. Every
executable under services/*/app and bin/* is then re-asserted +x, since that is
what an outage looks like when it is wrong and asserting it is free.
Verified against a synthetic site tree: all five binaries keep 755, assets 644.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Three .Do() calls in google_calendar.go accepted a ctx parameter but
never chained .Context(ctx) into the actual SDK call, so the existing
global 60s request timeout never reached the blocking network call.
One hung Google Calendar request wedged every DB/session-touching
request path in production for three days (2026-08-01 through
2026-08-04), undetected because /health unconditionally returned 200
throughout.
- Wire .Context(ctx) into GetUpcomingEvents, GetEventsByDateRange, and
GetCalendarList.
- Bound aggregateData's four external fetches with a per-fetch
sub-context as defense-in-depth (only effective if the callee
actually honors ctx -- documented as such, not oversold).
- Make GetUpcomingEvents/GetEventsByDateRange fetch calendars
concurrently instead of sequentially: a review of this fix caught
that a shared per-fetch deadline over a sequential loop would starve
calendars past the first under any real latency, silently caching
partial results as complete. Concurrent fetches give every calendar
an equal shot at the same deadline instead.
- /health now does a real PingContext DB check instead of a static
"ok" (Handler.PingDB, tested for both healthy and closed-DB cases).
- Add internal/api/context_audit_test.go: an AST-based structural
guard that fails any future .Do() call in google_*.go missing
.Context(...) anywhere in its chain, so this class of bug can't
silently recur. Verified by deliberately reintroducing the original
bug against a backup and confirming the guard catches it.
- Add scripts/health-watchdog.sh: cron job restarts the service if
/health fails twice in a row, five minutes apart.
go test ./... -race is green. Deployed and live-verified.
|
|
Refuses to run if hostname doesn't match the documented deploy host,
preventing an accidental deploy from a sandbox/dev environment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZ7ikw2ukGJFTHE3bJS7zL
|
|
|
|
Script uses relative paths (npm, go build, rsync) so it must run from
the repo root regardless of the caller's working directory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
- Delete Bug struct, BugToAtom, SourceBug, TypeBug, TypeNote
- Remove bug store methods (SaveBug, GetBugs, ResolveBug, etc.)
- Remove HandleGetBugs, HandleReportBug, bug branches in handlers
- Remove bug routes, bugs.html template, bug UI from index.html
- Remove AddMealToPlanner stub + interface method
- Migration 018: DROP TABLE IF EXISTS bugs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Replace SSH-based deploy.sh with scripts/deploy that runs locally on the
server. Update scripts/bugs, scripts/logs, and scripts/resolve-bug to drop
SSH wrappers. Update CLAUDE.md and DESIGN.md to reflect new local workflow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
- Display build commit hash in unobtrusive footer overlay
- Inject buildCommit/buildTime via ldflags in deploy.sh
- Add assertTemplateContains test helper, refactor existing template tests
- Add scripts/logs for fetching production journalctl via SSH
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
|
Bug fixes:
- #36: Hide recurring tasks until due day (add IsRecurring to Task/Atom)
- Trello cards missing: change filter=visible to filter=open
- Build fix: add missing fmt import in atom.go
Infrastructure:
- Add scripts/bugs and scripts/resolve-bug for DB bug tracking
- Remove issues/ directory (bugs now tracked in DB)
- Add timeline_logic_test.go
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
