| Age | Commit message (Collapse) | Author |
|---|
|
Deploy used to end with `systemctl status`, which a human skims and a script
ignores: it exits zero whether or not anything works, and says nothing about
the sub-services behind the gateway. A deploy could therefore report success
while /scout/ served 502s — which is exactly what happened, undetected, for
three and a half weeks.
scripts/deploy now:
* syncs scripts/ to ${SITE_DIR}/app-scripts/ so the timer runs the deployed
version from a stable path
* restarts sub-services that are not active. They are separate units and a
doot deploy does not touch them, so one sitting in `failed` after
exhausting its restart limit stays down through any number of green
deploys.
* calls scripts/verify-deploy, which asserts and exits non-zero
scripts/verify-deploy asserts all five binaries are executable, all four units
are active, all four gateway routes answer without a 5xx, and the running build
stamp matches the commit just built.
scripts/health-check runs the same assertions on a 5-minute timer and recovers
what it can: chmod +x a binary that lost its execute bit, reset-failed and
start a unit that gave up. There is no mail/ntfy/apprise on this host, so
alerting is a greppable HEALTH ALERT line in the journal, a status JSON at
data/health-status.json for the dashboard to surface later, and a non-zero exit
so the unit shows in `systemctl --failed`. OnBootSec=2min because a reboot is
when this failure mode first appeared.
Shared assertions live in scripts/lib/site-checks.sh so verify-deploy and
health-check cannot drift apart.
Two bugs were found and fixed in these checks by testing their failure paths,
both of which would have made them report a broken site as healthy:
* `$(curl ... || echo 000)` concatenates curl's own "000" transport-error
output with the echoed one, giving "000000", which matched no case arm and
was reported ok.
* check_services sets FAILED_SERVICES, but health-check captured it with
`$(...)`, so the array was populated in a subshell and discarded — silently
disabling service recovery entirely. Both the calling convention and the
subshell semantics are now pinned by tests.
scripts/test-health-checks.sh: 9 cases, mostly negative, against a temp tree.
Recovery additionally verified live on this host by stopping hawaii.service and
by clearing scout's execute bit; both were detected, recovered and re-verified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
The permission step ran
find ${SITE_DIR} -type f -exec chmod 644 {} +
chmod +x ${SITE_DIR}/app
over the whole site directory, so it cleared +x on every other executable there
and restored it only for doot's own app. Four binaries were caught:
services/scout/app scout.service
services/hawaii/app hawaii sub-service
bin/claudomator claudomator@doot.terst.org.service
bin/claude exec'd by claudomator via --claude-bin
Nothing appears to break at deploy time: a running service keeps executing from
its already-open file. The damage surfaces at the next reboot, when systemd
cannot exec the binary and the unit dies with status=203/EXEC. scout.service
went down that way at the 2026-08-25 reboot and stayed down three and a half
weeks, serving 502s behind /scout/; claudomator has been crash-looping on the
same error, 426k restarts and counting. Neither is alerted on, because doot
itself is healthy and only those paths fail.
Permission fixing is now scoped to the directories this script actually manages
(public, templates, migrations) plus doot's own binary and .env. Every
executable under services/*/app and bin/* is then re-asserted +x, since that is
what an outage looks like when it is wrong and asserting it is free.
Verified against a synthetic site tree: all five binaries keep 755, assets 644.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AwnALZL6bD4VBtATDVv3B
|
|
Refuses to run if hostname doesn't match the documented deploy host,
preventing an accidental deploy from a sandbox/dev environment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZ7ikw2ukGJFTHE3bJS7zL
|
|
Script uses relative paths (npm, go build, rsync) so it must run from
the repo root regardless of the caller's working directory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
- Delete Bug struct, BugToAtom, SourceBug, TypeBug, TypeNote
- Remove bug store methods (SaveBug, GetBugs, ResolveBug, etc.)
- Remove HandleGetBugs, HandleReportBug, bug branches in handlers
- Remove bug routes, bugs.html template, bug UI from index.html
- Remove AddMealToPlanner stub + interface method
- Migration 018: DROP TABLE IF EXISTS bugs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Replace SSH-based deploy.sh with scripts/deploy that runs locally on the
server. Update scripts/bugs, scripts/logs, and scripts/resolve-bug to drop
SSH wrappers. Update CLAUDE.md and DESIGN.md to reflect new local workflow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
