# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. Also check `~/.claude/CLAUDE.md` for user-level development standards (TDD workflow, git practices, session state management, etc.) that apply globally across all projects. ## Canonical Repository **The canonical source of truth is `/workspace/claudomator`.** All development must happen here. Do not work in any other directory unless explicitly instructed. Do not explore `/site/doot.terst.org/` for source files. ## Build & Test Commands ```bash # Build go build ./... # Run all tests go test ./... # Run a single package's tests go test ./internal/executor/... # Run a single test by name go test ./internal/api/ -run TestServer_CreateTask_MissingName # Run with race detector (important for executor/pool tests) go test -race ./... # Build the binary go build -o claudomator ./cmd/claudomator/ ``` > **Note:** `go-sqlite3` uses CGo. A C compiler (`gcc`) must be present for builds and tests. ## Running the Server ```bash # Initialize data directory ./claudomator init # Start API server (default :8484) ./claudomator serve # Run a task file directly (bypasses server) ./claudomator run ./test/fixtures/tasks/simple-task.yaml # List tasks via CLI ./claudomator list ``` Config defaults to `~/.claudomator/config.toml`. Data is stored in `~/.claudomator/` (SQLite DB + execution logs). --- ## Architecture **Pipeline:** CLI/API → `executor.Pool` → `executor.ContainerRunner` → `claude -p` subprocess → SQLite + log files ### Package Overview | Package | Role | |---|---| | `internal/task` | `Task` struct, YAML/JSON parsing, state machine constants, validation | | `internal/executor` | `Pool` (bounded goroutine dispatcher) + `ContainerRunner` (Docker; runs `claude`/`gemini`) + `LocalRunner` (OpenAI-compatible HTTP) + `Classifier` + `AgentChannel` + per-task agent MCP server (`Registry`) + planning preamble | | `internal/storage` | SQLite wrapper; additive migrations; tasks + executions + events tables | | `internal/event` | `Event` type, kind/actor constants — the per-task observability stream | | `internal/budget` | `Accountant` — rolling per-provider spend windows; gates the dispatcher | | `internal/llm` | OpenAI-compatible chat client (used by LocalRunner + Classifier), incl. tool-use | | `internal/api` | HTTP/WebSocket server — REST endpoints, webhook handler, validate, script runner, agent MCP (`/mcp`) + chatbot MCP (`/chatbot/mcp`), budget + events endpoints | | `internal/notify` | `Notifier` interface; webhook, multi, log implementations | | `internal/reporter` | Console/JSON/HTML report generation | | `internal/deployment` | Deployment-status checking (polls URL for expected version) | | `internal/config` | TOML config loading + data-dir layout helpers | | `internal/cli` | Cobra commands: `run`, `serve`, `list`, `status`, `start`, `logs`, `create`, `report`, `init` | | `internal/version` | VCS version detection (`debug.ReadBuildInfo`) | | `internal/provider` | Provider-neutral chat/tool-use interface (`Provider`); adapters: `anthropic`, `google`, `openaicompat` (wraps `internal/llm`, also backs the `groq`/`openrouter`/`openai` runners) | | `internal/agentloop` | Provider-neutral tool-use control-flow loop shared by all `NativeRunner`s | | `internal/sandbox` | `Sandbox` interface (`HostSandbox`/`DockerSandbox`) + pre-tool-use guardrail hooks | | `internal/role` | `RoleConfig`/`Tier`/`Rung` — per-role system prompt + provider/model escalation ladder | | `internal/scheduler` | `Scheduler` — polls role-typed FAILED tasks and retries/escalates them per their role's ladder | | `web` | Embedded static UI (`embed.go`) | ### Key Data Flows **Task execution:** 1. Task created via `POST /api/tasks` or YAML file (`task.ParseFile`) 2. `POST /api/tasks/{id}/run` → `executor.Pool.Submit()` → buffered work queue 3. `dispatch()` goroutine picks from queue, waits for slot, launches `execute()` 4. `execute()` calls `ContainerRunner.Run()` → `claude -p --output-format stream-json` 5. stdout piped through `parseStream()` to `~/.claudomator/executions//stdout.log` 6. Execution result written to SQLite, broadcast via WebSocket to connected clients **Task state machine** (enforced in `storage.UpdateTaskState` via `task.ValidTransition`): ``` PENDING ──→ QUEUED ──→ RUNNING ──→ READY ──→ COMPLETED ↑ │ └──→ PENDING (rejected) │ │ │ ├──→ BLOCKED ──→ READY (all subtasks done) │ │ └──→ QUEUED (question answered) │ │ └──────────────├──→ FAILED ├──→ TIMED_OUT ├──→ CANCELLED └──→ BUDGET_EXCEEDED ``` - **BLOCKED**: Parent task completed but has subtasks that are not yet COMPLETED, OR the agent called `ask_user`. Unblocked by `maybeUnblockParent()` or a user/chatbot answer via `/api/tasks/{id}/answer` (or the chatbot MCP `answer_question` tool). - **READY**: Execution succeeded; awaits manual accept/reject via `/api/tasks/{id}/accept` or `/api/tasks/{id}/reject`. - **COMPLETED**: Terminal — entered only via user accept (top-level) or automatic subtask completion. - `FAILED/TIMED_OUT/CANCELLED/BUDGET_EXCEEDED` all re-enter at `QUEUED` for retry/resume. **WebSocket:** `Hub` fans out task completion events to all connected clients. `Server.StartHub()` must be called before `ListenAndServe`. ### Execution model (ContainerRunner, Docker-based) The production runner for both `claude` and `gemini` agent types is `ContainerRunner`. (The old subprocess `ClaudeRunner`/`GeminiRunner` and the `/tmp` git-sandbox helpers were removed; the standalone Gemini stub is gone.) 1. The repo is cloned into a per-run workspace bind-mounted at `/workspace`; a writable `$HOME` is staged at `/home/agent`. The agent runs as the host uid (`--user`), with `IS_SANDBOX=1` set so `claude --permission-mode bypassPermissions` is honored even under root. 2. Instructions are written to a file and passed to the CLI; when the MCP back-channel is active and `skip_planning` is false, the planning preamble is prepended. 3. After a successful run, new commits are pushed from the workspace. 4. On BLOCKED (the agent called `ask_user`), the workspace path is stored in `executions.sandbox_dir` so the resume execution reuses it. ### Agent back-channel (MCP) Runners expose a normalized `AgentChannel` (`AskUser`, `ReportSummary`, `SpawnSubtask`, `RecordProgress`). Transport is per-runner: - **ContainerRunner** mints a per-task bearer token (`executor.Registry`) and points the agent at the host MCP server: claude via `--mcp-config`, gemini via `~/.gemini/settings.json`. The server (`/mcp`) resolves the task from the token and never trusts an agent-supplied ID. - **LocalRunner** declares the four tools as OpenAI function-calling defs and runs a tool-use loop, re-feeding tool results as message history. `ask_user` records a clarification and returns `ErrAgentBlocked`; the runner turns the buffered question into a `*BlockedError` and the task blocks. ### Task YAML Format ```yaml name: "My Task" description: "Optional longer description" agent: type: "claude" # "claude" (default) or "gemini" (stub, not production-ready) model: "sonnet" # optional; auto-classified by Classifier if omitted instructions: | Do something useful. project_dir: "/path/to/project" # optional; triggers sandbox isolation max_budget_usd: 1.00 permission_mode: "bypassPermissions" # default; or "default", "acceptEdits" allowed_tools: ["Bash", "Read", "Edit"] disallowed_tools: [] context_files: ["/extra/context/path"] system_prompt_append: "Extra instructions appended to system prompt." skip_planning: false # if false, prepends planning/orchestration preamble additional_args: [] # extra flags forwarded verbatim to claude CLI timeout: "15m" priority: "normal" # "high" | "normal" | "low" (stored but not yet used for scheduling) tags: ["ci"] depends_on: ["other-task-id"] retry: max_attempts: 1 # stored but retry is currently manual via /resume backoff: "exponential" ``` > **Note:** The YAML key is `agent:`, not `claude:`. Earlier docs showed `claude:` which was wrong. Batch files wrap multiple tasks under a `tasks:` key. ### Storage Schema Schema is auto-migrated additively on `storage.Open()` — new columns are `ALTER TABLE ... ADD COLUMN` statements that silently succeed if the column already exists. ``` tasks: id, name, description, config_json, priority, timeout_ns, retry_json, tags_json, depends_on_json, parent_task_id, state, rejection_comment, question_json, summary, elaboration_input, interactions_json, created_at, updated_at executions: id, task_id, start_time, end_time, exit_code, status, stdout_path, stderr_path, artifact_dir, cost_usd, error_msg, session_id, sandbox_dir, changestats_json, commits_json, tokens_in, tokens_out, agent, escalation_rung events: id, task_id, seq, ts, kind, actor, payload_json role_configs: id, role, version, status, config_json, created_at, activated_at, retired_at, proposed_by (UNIQUE(role, version)) ``` The `events` table is the per-task observability stream (`internal/event`); see the REST `GET /api/tasks/{id}/events` and the chatbot MCP `get_events` tool. `executions.agent` records which provider ran each execution, for budget accounting (`SpendByProviderSince`). `executions.escalation_rung` records the 0-based index into the active role's `EscalationLadder` a role-typed task's execution ran at (0 for non-role tasks); see "Role-based dispatch & escalation" below. `role_configs` holds versioned `internal/role.RoleConfig` blobs (`config_json`) per role, `status` one of `draft`/`active`/`retired`. `ActivateRoleConfigVersion` enforces "at most one active row per role" transactionally (retire-then-activate in one transaction), the same way `UpdateTaskState` enforces the task state machine in a transaction rather than in schema — see `internal/storage/roleconfig.go`. JSON blobs: `config_json` (AgentConfig on `tasks`; RoleConfig on `role_configs`), `retry_json`, `tags_json`, `depends_on_json`, `interactions_json`, `changestats_json`, `commits_json`. --- ## Features ### Planning Preamble & Orchestration When `agent.skip_planning` is false (the default), `withPlanningPreamble()` prepends a system-level prompt that points the agent at its MCP/tool-use back-channel rather than file/REST conventions: - Break work taking more than ~3 minutes into pieces via the `spawn_subtask` tool, then stop - Call `ask_user` (and end the turn) when it needs a decision - Commit all changes before finishing - Call `report_summary` before finishing The four tools — `ask_user`, `report_summary`, `spawn_subtask`, `record_progress` — are served over MCP (claude/gemini) or OpenAI tool-use (local); the old `CLAUDOMATOR_QUESTION_FILE`/`CLAUDOMATOR_SUMMARY_FILE` conventions were removed. ### Changestats After each execution, changestats (files changed, lines added/removed) are parsed from git `diff --stat` output in `stdout.log` and stored in `executions.changestats_json`. > **Duplication debt:** Changestats are extracted in two places: `executor.Pool.handleRunResult()` and `api.Server.processResult()`. Both write the same value to the same row (idempotent), but the double-extraction is confusing and should be consolidated. See [Design Debt](#design-debt). **Parser:** `internal/task/changestats.go` — `ParseChangestatFromOutput`, `ParseChangestatFromFile`. **Frontend:** `web/app.js` renders a `.changestats-badge` on COMPLETED/READY task cards. ### GitHub Webhook Integration `POST /api/webhooks/github` accepts `check_run` and `workflow_run` events. Returns `{"task_id": "..."}` (200) on task creation or 204 if ignored. #### Config (`~/.claudomator/config.toml`) ```toml webhook_secret = "your-github-webhook-secret" # HMAC-SHA256; skip validation if omitted [[projects]] name = "myrepo" dir = "/workspace/myrepo" ``` #### Matching logic Repository name matched case-insensitively against each project's `name` and the basename of its `dir`. Falls back to the only configured project if no match found. #### Task creation Tasks created for: - `check_run` with `action: completed` and `conclusion: failure` - `workflow_run` with `action: completed` and `conclusion: failure` or `timed_out` Tagged `["ci", "auto"]`, capped at $3 USD, allowed tools: Read, Edit, Bash, Glob, Grep. ### Task creation (chatbot-driven) The natural-language `POST /api/tasks/elaborate` endpoint and the web create form were removed. Chatbots now create and drive tasks through the chatbot MCP server (`/chatbot/mcp`, shared bearer token): `submit_task`, `list_tasks`, `get_task`, `get_events`, `answer_question`, `accept_task`, `reject_task`, `cancel_task`. The web UI is observability-only (task tree, live logs, event timeline, accept/reject, budget headroom). ### Budget gating `internal/budget.Accountant` tracks per-provider spend in a rolling window (default 5h, `[budget]` config). Before running, the dispatcher reroutes paid work to the free local runner when a cap would be breached, else blocks the task (`BUDGET_EXCEEDED`). `GET /api/budget` surfaces headroom. Local runners are free. ### Auth / binding The server binds `127.0.0.1` by default and refuses a non-loopback bind unless `external_bind_allowed = true` (front external access with a reverse proxy). The shared `api_token` guards the web UI + chatbot MCP; agent MCP uses per-task tokens; GitHub webhooks use HMAC. ### Model Classifier `executor.Classifier` calls the Gemini CLI (`gemini-2.5-flash-lite`) to pick the best Claude or Gemini model for a task. Falls back to the default model (`sonnet`) if Gemini fails. Agent type is selected first by load balancer; classifier only picks the model within that agent. > **Implementation gap:** Output parsing is brittle — strips `"Loaded cached credentials."` lines and markdown fences by string matching. No fallback if Gemini CLI isn't installed. Classification results are not cached or logged for learning. ### Role-based dispatch & escalation A task opts into role-based dispatch by setting `agent.role` (`task.AgentConfig.Role`) to a role name with an **active** `role_configs` version (see the REST endpoints below). Tasks with `agent.role` unset (every existing YAML/chatbot task shape) are completely unaffected — this is purely additive. - **Initial dispatch** (`internal/executor.Pool.execute()`): when `Agent.Role != ""` and `Agent.Type == ""`, the pool resolves tier 0 of the active `role_configs` row's `EscalationLadder`, setting `Agent.Type`/`Agent.Model` on a copy of the task (mirroring the `withFailureHistory` copy-before-mutate pattern) and applying `RoleConfig.SystemPrompt` to `Agent.SystemPromptAppend`. Multi-candidate `round_robin` tiers rotate via `Pool.selectRung`, skipping any provider currently in `Pool.rateLimited` (falling back to whichever clears soonest if all are limited). - **Retry/escalation** (`internal/scheduler.Scheduler`, started by `serve` only — not the one-shot `run` command): polls for role-typed tasks whose latest execution is `FAILED` and, per the ladder tier at that execution's `escalation_rung`: retries at the same rung while under `tier.MaxRetries`, or escalates to the next tier's first candidate if `budget.Accountant.Allow` permits it (recording an `event.KindEscalated` event), or leaves the task `FAILED` for human attention — recording a `final: true` `KindEscalated` event — if the budget denies it or the ladder is exhausted. An in-memory (per-process) "already handled" set keyed by execution ID keeps the poll loop convergent: a task stuck at the end of its ladder gets exactly one `final` event, not one per poll tick. This resets on restart (by design — it's idempotent bookkeeping, not orchestration state, so re-deriving it once is harmless). - **REST**: `POST /api/roles/{role}/versions` (create draft), `GET /api/roles/{role}/versions` (list), `POST /api/roles/{role}/activate?version=N` (activate, atomically retiring whatever was active). > **Stored but not yet enforced:** `RoleConfig.Tools`/`SandboxKind` are round-tripped through storage/API but don't affect tool availability or sandbox selection — those are still applied uniformly by `NativeRunner` regardless of role. `RoleConfig.DefaultBudgetUSD` is read narrowly by the scheduler as the estimated cost passed to `Allow()` when considering an escalation — it is not enforced at initial-dispatch time the way `task.AgentConfig.MaxBudgetUSD` is. See `internal/role/role.go`'s field docs. > **Known simplification:** the scheduler always escalates to `nextTier.Candidates[0]` — it does not round-robin across multiple candidates in a tier the way `Pool.execute()`'s initial-dispatch resolution does. A later phase could extend this if multi-candidate escalation tiers turn out to matter in practice. --- --- ## Design Debt ### RoleConfig.Tools/SandboxKind/DefaultBudgetUSD are stored but not fully enforced Same pattern as `task.Priority`/`RetryConfig` below: `internal/role.RoleConfig.Tools` and `.SandboxKind` round-trip through `role_configs`/the REST endpoints but have no effect on dispatch — tool availability and sandbox selection are still applied uniformly by `NativeRunner` regardless of role. `.DefaultBudgetUSD` is read only narrowly, as the scheduler's estimated escalation cost, not enforced generally at dispatch time. ### Scheduler's escalation candidate selection doesn't round-robin `internal/scheduler.Scheduler` always escalates to `nextTier.Candidates[0]`, unlike `Pool.execute()`'s initial-dispatch tier-0 resolution, which round-robins across multi-candidate tiers. Same-rung retries intentionally reuse the exact same provider/model rather than re-resolving. ### Scheduler's double-processing guard is in-memory only `Scheduler.handled` (keyed by execution ID) prevents re-emitting a `final: true` `KindEscalated` event on every poll tick once a role-typed task's ladder is exhausted, but it's per-process and resets on restart. A restart can produce one extra "reconsideration" (and, if still exhausted/denied, one more `KindEscalated` event) — not an infinite loop, just not persisted. A future phase could persist this via a `tasks` column if that turns out to matter. ### Deprecated task columns not yet dropped (Phase 8 follow-up) `tasks.question_json`, `summary`, `interactions_json`, `elaboration_input` are superseded by the `events` table but still read by the answer flow and task panel. Dropping them requires migrating those reads to events first. ### Gemini tool-use unverified The gemini MCP wiring (`~/.gemini/settings.json`) is in place but was never run against a real `gemini` binary — see `cmd/spike` (`go run ./cmd/spike gemini`). ### Priority field is stored but never used `task.Priority` (`high`, `normal`, `low`) is persisted in SQLite and surfaced in the API. The executor `dispatch()` goroutine uses a simple FIFO channel (`workCh`) with no priority ordering. ### RetryConfig is stored but retry is manual `task.RetryConfig.MaxAttempts` and `Backoff` are parsed and stored. No code reads them during execution. Retries must be triggered manually via `POST /api/tasks/{id}/resume`. ### Changestats extracted in two places `executor.Pool.handleRunResult()` and `api.Server.processResult()` both call `task.ParseChangestatFromFile()` and write to `executions.changestats_json`. The second write is idempotent but wasteful and confusing. One of the two should be removed. ### context.Background() in resume path `api.Server.handleAnswerQuestion()` calls `p.SubmitResume(context.Background(), ...)`. If the HTTP request context is cancelled, the resume still runs. Inversely, if the server shuts down, in-flight resumes using the server's root context would be cancelled while this one would not. Should use a long-lived server-level context, not `Background()`. ### Non-transactional execution creation `pool.execute()` calls `store.CreateExecution(exec)` followed by `store.UpdateTaskState(t.ID, task.StateRunning)` as separate statements. If the server crashes between them, the task stays PENDING while an execution record exists with status RUNNING. Recovery (`RecoverStaleRunning`) partially handles this but the root cause is the missing transaction. ### Elaborate/validate cmd path indirection `Server` has two separate fields `elaborateCmdPath` and `validateCmdPath` that override `claudeBinPath` only for tests. This is a testing-time seam that leaks into the production struct. A cleaner approach would be to inject an `Elaborator` interface. ### `withFailureHistory` mutates a shallow copy In `executor.go`, `withFailureHistory` creates a copy of the task struct (`copy := *t`) but `copy.Agent = t.Agent` copies the struct value — slices inside AgentConfig (`AllowedTools`, `DisallowedTools`, etc.) share the backing array. Appending to `SystemPromptAppend` is safe but any mutation of slices would affect the original. ### Additive migration strategy is fragile `storage.migrate()` lists every `ALTER TABLE ADD COLUMN` statement in code order. The only idempotency guard is catching "column already exists" errors. There is no migration version tracking. Columns dropped in `CREATE TABLE IF NOT EXISTS` and added back via ALTER are indistinguishable from new columns. Concurrent server instances running migrations simultaneously have no protection. --- ## REST API Reference | Method | Endpoint | Description | |--------|----------|-------------| | GET | `/api/tasks` | List tasks; `?state=RUNNING&since=&limit=50` | | POST | `/api/tasks` | Create task (JSON body) | | GET | `/api/tasks/{id}` | Get task | | DELETE | `/api/tasks/{id}` | Delete task + subtasks + executions | | POST | `/api/tasks/{id}/run` | Submit PENDING task to executor | | POST | `/api/tasks/{id}/cancel` | Cancel RUNNING/QUEUED task | | POST | `/api/tasks/{id}/accept` | Accept READY task → COMPLETED | | POST | `/api/tasks/{id}/reject` | Reject READY task → PENDING | | POST | `/api/tasks/{id}/answer` | Answer BLOCKED task question → QUEUED | | POST | `/api/tasks/{id}/resume` | Resume FAILED/TIMED_OUT/CANCELLED task | | GET | `/api/tasks/{id}/subtasks` | List subtasks | | GET | `/api/tasks/{id}/executions` | List execution history | | GET | `/api/executions/{id}` | Get execution | | GET | `/api/executions/{id}/log` | Get execution log (`?tail=100`) | | GET | `/api/executions/{id}/logs/stream` | Stream logs as SSE | | GET | `/api/tasks/{id}/logs/stream` | Stream latest execution logs | | GET | `/api/executions` | List recent executions across all tasks | | GET | `/api/tasks/{id}/deployment-status` | Poll deployment readiness | | GET | `/api/tasks/{id}/events` | Task observability event stream (`?since_seq=N`) | | GET | `/api/budget` | Per-provider rolling-window spend headroom | | POST | `/api/tasks/validate` | Validate task JSON | | POST/GET/DELETE | `/mcp` | Per-task agent MCP back-channel (bearer = minted token) | | POST/GET/DELETE | `/chatbot/mcp` | Chatbot MCP server (bearer = `api_token`) | | POST | `/api/scripts/{name}` | Run named script with task context | | GET | `/api/ws` | WebSocket upgrade (live task updates) | | GET | `/api/workspaces` | List directories under `workspace_root` | | GET | `/api/health` | Server health | | POST | `/api/webhooks/github` | GitHub CI webhook | | POST | `/api/roles/{role}/versions` | Create a new draft `role_configs` version | | GET | `/api/roles/{role}/versions` | List all versions for a role | | POST | `/api/roles/{role}/activate?version=N` | Activate a version (atomically retires the prior active one) | --- ## ADRs See `docs/adr/001-language-and-architecture.md` for the Go + SQLite + WebSocket rationale.