summaryrefslogtreecommitdiff
path: root/project_claudomator_workspace_disk_leak.md
diff options
context:
space:
mode:
authorClaude Code <claude@terst.org>2026-08-15 18:06:00 +0000
committerClaude Code <claude@terst.org>2026-08-15 18:06:00 +0000
commita6d58e009efcc06e7b5221d75ebed36533786dce (patch)
treecfeded4c1bb8e61a4c117aced0b42e7d4aa0e15a /project_claudomator_workspace_disk_leak.md
parenteb0932bddad0daa47c36cd20599944c3dc45482a (diff)
Merge titanium's diverged memory tree into the shared historyHEADmain
Titanium and tungsten had grown two separate, unreconciled memory sets for the same -workspace scope. Ports over 18 titanium-only entries (12 claudomator files, the fork-scope-creep lesson, the doot Google Tasks OAuth limitation, the default-subagent-driven-execution preference, scout's Facebook priority, Hawaii GIS endpoints), merges two true duplicates into their existing tungsten files rather than keeping parallel copies (the migration-rename incident -- same 022->023 rename, same commit, titanium's version added the confirmed-safe-to-patch-prod-directly carve-out tungsten's didn't have; the APK build OOM note, folded into project_doot_widget_build.md as a RAM-constrained-host fallback), and marks doot_future_task_scheduling_ideas.md as historical now that all four of its backlog items (budgets, labels/projects, buckets, chains) are built.
Diffstat (limited to 'project_claudomator_workspace_disk_leak.md')
-rw-r--r--project_claudomator_workspace_disk_leak.md16
1 files changed, 16 insertions, 0 deletions
diff --git a/project_claudomator_workspace_disk_leak.md b/project_claudomator_workspace_disk_leak.md
new file mode 100644
index 0000000..6fb1b28
--- /dev/null
+++ b/project_claudomator_workspace_disk_leak.md
@@ -0,0 +1,16 @@
+---
+name: claudomator-workspace-disk-leak
+description: FIXED 2026-07-10 (commit 06c9730) — ContainerRunner preserved failed-run workspaces forever with no expiry; Pool.RunWorkspaceCleanup now sweeps hourly
+metadata:
+ node_type: memory
+ type: project
+ originSessionId: 560b9055-7a3f-483e-b1a0-1a4c70419390
+---
+
+**Status: fixed**, commit `06c9730` on `main` (2026-07-10) — `internal/executor.Pool.RunWorkspaceCleanup(ctx, interval, maxAge)` sweeps `claudomator-workspace-*` dirs older than `DefaultWorkspaceCleanupMaxAge` (24h) every `DefaultWorkspaceCleanupInterval` (1h), started via `go pool.RunWorkspaceCleanup(...)` in `internal/cli/serve.go` right after the story orchestrator. Never removes a directory still referenced as a currently-BLOCKED task's `sandbox_dir`, regardless of age (tested). **Caveat: the live production instance (`/site/doot.terst.org/bin/claudomator serve`) runs a compiled binary built before this fix — it needs a rebuild+redeploy to actually pick this up, which was not done as part of this fix.** Check `ps aux | grep claudomator` / the binary's build time before assuming the live server already has this running.
+
+Original finding (2026-07-10): `internal/executor.ContainerRunner` bind-mounts a per-run workspace at `/tmp/claudomator-workspace-<id>` for every dispatched task. Its own doc comment says "workspace is only removed on success. On failure, it's preserved for debugging" — a deliberate design choice with no expiry, so failed-run workspaces accumulated forever. Found when the host hit 100% disk full (29MB free on 117GB) — 161 stale workspace dirs, ~17.6GB. Directly caused two real-time dispatch failures (generic "container execution failed: exit status 1") before being diagnosed. Manually cleaned up as an immediate workaround (161 dirs removed, ~15GB freed) before the real fix landed.
+
+**Why:** Preserving a failed workspace for debugging is reasonable; preserving it *forever* with no retention bound is not — the fix adds a bounded sweep rather than removing the debug-preservation behavior outright.
+
+**How to apply:** If a claudomator dispatch fails with a vague "exit status 1" on a host running a pre-`06c9730` binary, check `df -h /` and `du -sh /tmp/claudomator-workspace-*` before assuming a credential/infra blip — disk-full was the confirmed cause of at least 2 dispatch failures in this session alone.