diff options
Diffstat (limited to 'CLAUDE.md')
| -rw-r--r-- | CLAUDE.md | 118 |
1 files changed, 90 insertions, 28 deletions
@@ -62,9 +62,12 @@ Config defaults to `~/.claudomator/config.toml`. Data is stored in `~/.claudomat | Package | Role | |---|---| | `internal/task` | `Task` struct, YAML/JSON parsing, state machine constants, validation | -| `internal/executor` | `Pool` (bounded goroutine dispatcher) + `ClaudeRunner` (subprocess + sandbox) + `GeminiRunner` (stub) + `Classifier` + preamble + question/summary helpers | -| `internal/storage` | SQLite wrapper; additive migrations; tasks + executions tables | -| `internal/api` | HTTP/WebSocket server — REST endpoints, webhook handler, elaborate/validate, script runner | +| `internal/executor` | `Pool` (bounded goroutine dispatcher) + `ContainerRunner` (Docker; runs `claude`/`gemini`) + `LocalRunner` (OpenAI-compatible HTTP) + `Classifier` + `AgentChannel` + per-task agent MCP server (`Registry`) + planning preamble | +| `internal/storage` | SQLite wrapper; additive migrations; tasks + executions + events tables | +| `internal/event` | `Event` type, kind/actor constants — the per-task observability stream | +| `internal/budget` | `Accountant` — rolling per-provider spend windows; gates the dispatcher | +| `internal/llm` | OpenAI-compatible chat client (used by LocalRunner + Classifier), incl. tool-use | +| `internal/api` | HTTP/WebSocket server — REST endpoints, webhook handler, validate, script runner, agent MCP (`/mcp`) + chatbot MCP (`/chatbot/mcp`), budget + events endpoints | | `internal/notify` | `Notifier` interface; webhook, multi, log implementations | | `internal/reporter` | Console/JSON/HTML report generation | | `internal/deployment` | Deployment-status checking (polls URL for expected version) | @@ -98,25 +101,44 @@ PENDING ──→ QUEUED ──→ RUNNING ──→ READY ──→ COMPLETED └──→ BUDGET_EXCEEDED ``` -- **BLOCKED**: Parent task completed but has subtasks that are not yet COMPLETED, OR agent wrote a question file. Unblocked by `maybeUnblockParent()` or user answer via `/api/tasks/{id}/answer`. +- **BLOCKED**: Parent task completed but has subtasks that are not yet COMPLETED, OR the agent called `ask_user`. Unblocked by `maybeUnblockParent()` or a user/chatbot answer via `/api/tasks/{id}/answer` (or the chatbot MCP `answer_question` tool). - **READY**: Execution succeeded; awaits manual accept/reject via `/api/tasks/{id}/accept` or `/api/tasks/{id}/reject`. - **COMPLETED**: Terminal — entered only via user accept (top-level) or automatic subtask completion. - `FAILED/TIMED_OUT/CANCELLED/BUDGET_EXCEEDED` all re-enter at `QUEUED` for retry/resume. **WebSocket:** `Hub` fans out task completion events to all connected clients. `Server.StartHub()` must be called before `ListenAndServe`. -### Sandbox Lifecycle (ContainerRunner (Docker-based)) +### Execution model (ContainerRunner, Docker-based) -When `agent.project_dir` is set: -1. `setupSandbox()` clones the project into `/tmp/claudomator-sandbox-*` via the "local" remote (bare repo), then falls back to "origin", then the working copy path. -2. The claude subprocess runs inside the sandbox. -3. After successful execution, `teardownSandbox()` auto-commits any uncommitted changes (after running a build if `Makefile`/`go.mod`/`gradlew` is present), then pushes new commits to the bare repo (`origin` from the sandbox's perspective). The sandbox is then removed. -4. On failure the sandbox is preserved and its path is returned in the error. -5. On BLOCKED (question written), the sandbox path is stored in `executions.sandbox_dir` so the resume execution can reuse it. +The production runner for both `claude` and `gemini` agent types is +`ContainerRunner`. (The old subprocess `ClaudeRunner`/`GeminiRunner` and the +`/tmp` git-sandbox helpers were removed; the standalone Gemini stub is gone.) -> **Known bug:** Variable shadowing in `claude.go` `Run()` means the outer `sandboxDir` is never assigned (both `setupSandbox` calls use `:=` inside nested blocks). This causes: (a) `teardownSandbox` is never called — work is discarded, sandboxes accumulate in `/tmp`; (b) `BlockedError.SandboxDir` is always `""`, so resume clones a fresh sandbox and loses the agent's partial work. See [Known Bugs](#known-bugs). +1. The repo is cloned into a per-run workspace bind-mounted at `/workspace`; a + writable `$HOME` is staged at `/home/agent`. The agent runs as the host uid + (`--user`), with `IS_SANDBOX=1` set so `claude --permission-mode + bypassPermissions` is honored even under root. +2. Instructions are written to a file and passed to the CLI; when the MCP + back-channel is active and `skip_planning` is false, the planning preamble is + prepended. +3. After a successful run, new commits are pushed from the workspace. +4. On BLOCKED (the agent called `ask_user`), the workspace path is stored in + `executions.sandbox_dir` so the resume execution reuses it. -> **Known bug:** `teardownSandbox` hardcodes `origin/master` when rebasing on conflict. Repos using `main` will fail on concurrent push. See [Known Bugs](#known-bugs). +### Agent back-channel (MCP) + +Runners expose a normalized `AgentChannel` (`AskUser`, `ReportSummary`, +`SpawnSubtask`, `RecordProgress`). Transport is per-runner: + +- **ContainerRunner** mints a per-task bearer token (`executor.Registry`) and + points the agent at the host MCP server: claude via `--mcp-config`, gemini via + `~/.gemini/settings.json`. The server (`/mcp`) resolves the task from the token + and never trusts an agent-supplied ID. +- **LocalRunner** declares the four tools as OpenAI function-calling defs and + runs a tool-use loop, re-feeding tool results as message history. + +`ask_user` records a clarification and returns `ErrAgentBlocked`; the runner +turns the buffered question into a `*BlockedError` and the task blocks. ### Task YAML Format @@ -162,9 +184,17 @@ tasks: id, name, description, config_json, priority, timeout_ns, retry_json executions: id, task_id, start_time, end_time, exit_code, status, stdout_path, stderr_path, artifact_dir, cost_usd, error_msg, session_id, - sandbox_dir, changestats_json, commits_json + sandbox_dir, changestats_json, commits_json, tokens_in, tokens_out, + agent + +events: id, task_id, seq, ts, kind, actor, payload_json ``` +The `events` table is the per-task observability stream (`internal/event`); see +the REST `GET /api/tasks/{id}/events` and the chatbot MCP `get_events` tool. +`executions.agent` records which provider ran each execution, for budget +accounting (`SpendByProviderSince`). + JSON blobs: `config_json` (AgentConfig), `retry_json`, `tags_json`, `depends_on_json`, `interactions_json`, `changestats_json`, `commits_json`. --- @@ -173,13 +203,13 @@ JSON blobs: `config_json` (AgentConfig), `retry_json`, `tags_json`, `depends_on_ ### Planning Preamble & Orchestration -When `agent.skip_planning` is false (the default), `withPlanningPreamble()` prepends a system-level prompt to the agent's instructions that: -- Instructs the agent to POST subtasks to `$CLAUDOMATOR_API_URL/api/tasks` and stop if the task will take more than ~3 minutes -- Instructs the agent to write a JSON question to `$CLAUDOMATOR_QUESTION_FILE` and exit if it needs user input -- Requires all changes to be committed before exit -- Requires a summary written to `$CLAUDOMATOR_SUMMARY_FILE` +When `agent.skip_planning` is false (the default), `withPlanningPreamble()` prepends a system-level prompt that points the agent at its MCP/tool-use back-channel rather than file/REST conventions: +- Break work taking more than ~3 minutes into pieces via the `spawn_subtask` tool, then stop +- Call `ask_user` (and end the turn) when it needs a decision +- Commit all changes before finishing +- Call `report_summary` before finishing -Env vars injected into every execution: `CLAUDOMATOR_API_URL`, `CLAUDOMATOR_TASK_ID`, `CLAUDOMATOR_PROJECT_DIR`, `CLAUDOMATOR_QUESTION_FILE`, `CLAUDOMATOR_SUMMARY_FILE`. +The four tools — `ask_user`, `report_summary`, `spawn_subtask`, `record_progress` — are served over MCP (claude/gemini) or OpenAI tool-use (local); the old `CLAUDOMATOR_QUESTION_FILE`/`CLAUDOMATOR_SUMMARY_FILE` conventions were removed. ### Changestats @@ -217,11 +247,28 @@ Tasks created for: Tagged `["ci", "auto"]`, capped at $3 USD, allowed tools: Read, Edit, Bash, Glob, Grep. -### Elaborate Endpoint +### Task creation (chatbot-driven) -`POST /api/tasks/elaborate` converts natural language → task JSON via a `claude --prompt` invocation. Optionally reads `CLAUDE.md` / `SESSION_STATE.md` from a configured working directory for context. Per-IP rate-limited. +The natural-language `POST /api/tasks/elaborate` endpoint and the web create +form were removed. Chatbots now create and drive tasks through the chatbot MCP +server (`/chatbot/mcp`, shared bearer token): `submit_task`, `list_tasks`, +`get_task`, `get_events`, `answer_question`, `accept_task`, `reject_task`, +`cancel_task`. The web UI is observability-only (task tree, live logs, event +timeline, accept/reject, budget headroom). -> **Implementation gap:** The elaborate endpoint is not tested against real Claude invocations. `sanitizeElaboratedTask()` uses keyword heuristics to infer missing tools (fragile). No caching. +### Budget gating + +`internal/budget.Accountant` tracks per-provider spend in a rolling window +(default 5h, `[budget]` config). Before running, the dispatcher reroutes paid +work to the free local runner when a cap would be breached, else blocks the task +(`BUDGET_EXCEEDED`). `GET /api/budget` surfaces headroom. Local runners are free. + +### Auth / binding + +The server binds `127.0.0.1` by default and refuses a non-loopback bind unless +`external_bind_allowed = true` (front external access with a reverse proxy). The +shared `api_token` guards the web UI + chatbot MCP; agent MCP uses per-task +tokens; GitHub webhooks use HMAC. ### Model Classifier @@ -236,13 +283,25 @@ Tagged `["ci", "auto"]`, capped at $3 USD, allowed tools: Read, Edit, Bash, Glob ## Design Debt -### GeminiRunner is a non-functional stub +### Stories / checker backend not yet removed (Phase 8 follow-up) + +The web stories dashboard was removed, but the backend remains: the `stories` +table + `internal/api/stories.go` handlers + `/api/stories/*` routes, +story-elaborate in `elaborate.go`, and the story/checker logic woven into +`executor.handleRunResult` (`checkStoryCompletion`, `spawnCheckerTask`, +`triggerStoryDeploy`, `ensureStoryBranch`) plus the `tasks` story/checker +columns. This is a large, lifecycle-touching teardown best done as its own pass. + +### Deprecated task columns not yet dropped (Phase 8 follow-up) -`internal/executor/gemini.go` `execOnce()` does not run the `gemini` binary. It starts a goroutine that writes hardcoded fake JSON to a pipe. `parseGeminiStream()` strips markdown fences but does no semantic parsing. There is no session/resume support. +`tasks.question_json`, `summary`, `interactions_json`, `elaboration_input` are +superseded by the `events` table but still read by the answer flow and task +panel. Dropping them requires migrating those reads to events first. -Any task with `agent.type: "gemini"` will silently return canned output. This is dangerous in production. +### Gemini tool-use unverified -**Decision needed:** Either implement GeminiRunner properly (subprocess + stream parsing + sandbox integration mirroring ClaudeRunner) or remove it and the `Classifier` from the codebase until it's ready. +The gemini MCP wiring (`~/.gemini/settings.json`) is in place but was never run +against a real `gemini` binary — see `cmd/spike` (`go run ./cmd/spike gemini`). ### Priority field is stored but never used @@ -300,8 +359,11 @@ In `executor.go`, `withFailureHistory` creates a copy of the task struct (`copy | GET | `/api/tasks/{id}/logs/stream` | Stream latest execution logs | | GET | `/api/executions` | List recent executions across all tasks | | GET | `/api/tasks/{id}/deployment-status` | Poll deployment readiness | -| POST | `/api/tasks/elaborate` | Convert natural language → task JSON | +| GET | `/api/tasks/{id}/events` | Task observability event stream (`?since_seq=N`) | +| GET | `/api/budget` | Per-provider rolling-window spend headroom | | POST | `/api/tasks/validate` | Validate task JSON | +| POST/GET/DELETE | `/mcp` | Per-task agent MCP back-channel (bearer = minted token) | +| POST/GET/DELETE | `/chatbot/mcp` | Chatbot MCP server (bearer = `api_token`) | | POST | `/api/scripts/{name}` | Run named script with task context | | GET | `/api/ws` | WebSocket upgrade (live task updates) | | GET | `/api/workspaces` | List directories under `workspace_root` | |