diff options
Diffstat (limited to 'project_claudomator_workspace_disk_leak.md')
| -rw-r--r-- | project_claudomator_workspace_disk_leak.md | 16 |
1 files changed, 16 insertions, 0 deletions
diff --git a/project_claudomator_workspace_disk_leak.md b/project_claudomator_workspace_disk_leak.md new file mode 100644 index 0000000..6fb1b28 --- /dev/null +++ b/project_claudomator_workspace_disk_leak.md @@ -0,0 +1,16 @@ +--- +name: claudomator-workspace-disk-leak +description: FIXED 2026-07-10 (commit 06c9730) — ContainerRunner preserved failed-run workspaces forever with no expiry; Pool.RunWorkspaceCleanup now sweeps hourly +metadata: + node_type: memory + type: project + originSessionId: 560b9055-7a3f-483e-b1a0-1a4c70419390 +--- + +**Status: fixed**, commit `06c9730` on `main` (2026-07-10) — `internal/executor.Pool.RunWorkspaceCleanup(ctx, interval, maxAge)` sweeps `claudomator-workspace-*` dirs older than `DefaultWorkspaceCleanupMaxAge` (24h) every `DefaultWorkspaceCleanupInterval` (1h), started via `go pool.RunWorkspaceCleanup(...)` in `internal/cli/serve.go` right after the story orchestrator. Never removes a directory still referenced as a currently-BLOCKED task's `sandbox_dir`, regardless of age (tested). **Caveat: the live production instance (`/site/doot.terst.org/bin/claudomator serve`) runs a compiled binary built before this fix — it needs a rebuild+redeploy to actually pick this up, which was not done as part of this fix.** Check `ps aux | grep claudomator` / the binary's build time before assuming the live server already has this running. + +Original finding (2026-07-10): `internal/executor.ContainerRunner` bind-mounts a per-run workspace at `/tmp/claudomator-workspace-<id>` for every dispatched task. Its own doc comment says "workspace is only removed on success. On failure, it's preserved for debugging" — a deliberate design choice with no expiry, so failed-run workspaces accumulated forever. Found when the host hit 100% disk full (29MB free on 117GB) — 161 stale workspace dirs, ~17.6GB. Directly caused two real-time dispatch failures (generic "container execution failed: exit status 1") before being diagnosed. Manually cleaned up as an immediate workaround (161 dirs removed, ~15GB freed) before the real fix landed. + +**Why:** Preserving a failed workspace for debugging is reasonable; preserving it *forever* with no retention bound is not — the fix adds a bounded sweep rather than removing the debug-preservation behavior outright. + +**How to apply:** If a claudomator dispatch fails with a vague "exit status 1" on a host running a pre-`06c9730` binary, check `df -h /` and `du -sh /tmp/claudomator-workspace-*` before assuming a credential/infra blip — disk-full was the confirmed cause of at least 2 dispatch failures in this session alone. |