merge: integrate oss branch — budget gating, MCP back-channel, events, loopback-only bind

Key features from OSS branch:
- Budget gating: rolling per-provider spend caps with BUDGET_EXCEEDED task state
- Agent MCP back-channel: runners mint tokens; /mcp endpoint resolves them
- Chatbot MCP server at /chatbot/mcp (requires api_token in config)
- Event timeline: unified event stream replacing ad-hoc question/summary flows
- Loopback-only default bind (127.0.0.1:8484); external_bind_allowed=true to expose
- repository_url required on task creation (enforces traceability)
- Auto-checker: spawns verification task after each execution

Conflict resolutions:
- serve.go: keep cfg.Runners.XEnabled() conditional registration from main + agentRegistry from oss
- config.go: keep RunnersConfig from main + BudgetConfig/ExternalBindAllowed from oss
- server.go: handleStaticFiles (base-path rewrite) kept; deduplicated duplicate from both sides
- executor/claude.go: accepted oss deletion (ClaudeRunner replaced by ContainerRunner)
- container_test.go: added noopChannel + AgentChannel parameter to Run call

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

1 files changed, 90 insertions, 28 deletions

diff --git a/CLAUDE.md b/CLAUDE.md
index f35c5c5..a65c45a 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -62,9 +62,12 @@ Config defaults to `~/.claudomator/config.toml`. Data is stored in `~/.claudomat
 | Package | Role |
 |---|---|
 | `internal/task` | `Task` struct, YAML/JSON parsing, state machine constants, validation |
-| `internal/executor` | `Pool` (bounded goroutine dispatcher) + `ClaudeRunner` (subprocess + sandbox) + `GeminiRunner` (stub) + `Classifier` + preamble + question/summary helpers |
-| `internal/storage` | SQLite wrapper; additive migrations; tasks + executions tables |
-| `internal/api` | HTTP/WebSocket server — REST endpoints, webhook handler, elaborate/validate, script runner |
+| `internal/executor` | `Pool` (bounded goroutine dispatcher) + `ContainerRunner` (Docker; runs `claude`/`gemini`) + `LocalRunner` (OpenAI-compatible HTTP) + `Classifier` + `AgentChannel` + per-task agent MCP server (`Registry`) + planning preamble |
+| `internal/storage` | SQLite wrapper; additive migrations; tasks + executions + events tables |
+| `internal/event` | `Event` type, kind/actor constants — the per-task observability stream |
+| `internal/budget` | `Accountant` — rolling per-provider spend windows; gates the dispatcher |
+| `internal/llm` | OpenAI-compatible chat client (used by LocalRunner + Classifier), incl. tool-use |
+| `internal/api` | HTTP/WebSocket server — REST endpoints, webhook handler, validate, script runner, agent MCP (`/mcp`) + chatbot MCP (`/chatbot/mcp`), budget + events endpoints |
 | `internal/notify` | `Notifier` interface; webhook, multi, log implementations |
 | `internal/reporter` | Console/JSON/HTML report generation |
 | `internal/deployment` | Deployment-status checking (polls URL for expected version) |
@@ -98,25 +101,44 @@ PENDING ──→ QUEUED ──→ RUNNING ──→ READY ──→ COMPLETED
                               └──→ BUDGET_EXCEEDED
 ```
 
-- **BLOCKED**: Parent task completed but has subtasks that are not yet COMPLETED, OR agent wrote a question file. Unblocked by `maybeUnblockParent()` or user answer via `/api/tasks/{id}/answer`.
+- **BLOCKED**: Parent task completed but has subtasks that are not yet COMPLETED, OR the agent called `ask_user`. Unblocked by `maybeUnblockParent()` or a user/chatbot answer via `/api/tasks/{id}/answer` (or the chatbot MCP `answer_question` tool).
 - **READY**: Execution succeeded; awaits manual accept/reject via `/api/tasks/{id}/accept` or `/api/tasks/{id}/reject`.
 - **COMPLETED**: Terminal — entered only via user accept (top-level) or automatic subtask completion.
 - `FAILED/TIMED_OUT/CANCELLED/BUDGET_EXCEEDED` all re-enter at `QUEUED` for retry/resume.
 
 **WebSocket:** `Hub` fans out task completion events to all connected clients. `Server.StartHub()` must be called before `ListenAndServe`.
 
-### Sandbox Lifecycle (ContainerRunner (Docker-based))
+### Execution model (ContainerRunner, Docker-based)
 
-When `agent.project_dir` is set:
-1. `setupSandbox()` clones the project into `/tmp/claudomator-sandbox-*` via the "local" remote (bare repo), then falls back to "origin", then the working copy path.
-2. The claude subprocess runs inside the sandbox.
-3. After successful execution, `teardownSandbox()` auto-commits any uncommitted changes (after running a build if `Makefile`/`go.mod`/`gradlew` is present), then pushes new commits to the bare repo (`origin` from the sandbox's perspective). The sandbox is then removed.
-4. On failure the sandbox is preserved and its path is returned in the error.
-5. On BLOCKED (question written), the sandbox path is stored in `executions.sandbox_dir` so the resume execution can reuse it.
+The production runner for both `claude` and `gemini` agent types is
+`ContainerRunner`. (The old subprocess `ClaudeRunner`/`GeminiRunner` and the
+`/tmp` git-sandbox helpers were removed; the standalone Gemini stub is gone.)
 
-> ~~**Known bug:** Variable shadowing in `claude.go` `Run()` — fixed in github/main merge; both `setupSandbox` calls correctly use `=` (assign to outer `sandboxDir`).~~
+1. The repo is cloned into a per-run workspace bind-mounted at `/workspace`; a
+   writable `$HOME` is staged at `/home/agent`. The agent runs as the host uid
+   (`--user`), with `IS_SANDBOX=1` set so `claude --permission-mode
+   bypassPermissions` is honored even under root.
+2. Instructions are written to a file and passed to the CLI; when the MCP
+   back-channel is active and `skip_planning` is false, the planning preamble is
+   prepended.
+3. After a successful run, new commits are pushed from the workspace.
+4. On BLOCKED (the agent called `ask_user`), the workspace path is stored in
+   `executions.sandbox_dir` so the resume execution reuses it.
 
-> ~~**Known bug:** `teardownSandbox` hardcodes `origin/master` — fixed; now uses `git pull --rebase` with no branch arg, which follows the configured upstream from `git clone`.~~
+### Agent back-channel (MCP)
+
+Runners expose a normalized `AgentChannel` (`AskUser`, `ReportSummary`,
+`SpawnSubtask`, `RecordProgress`). Transport is per-runner:
+
+- **ContainerRunner** mints a per-task bearer token (`executor.Registry`) and
+  points the agent at the host MCP server: claude via `--mcp-config`, gemini via
+  `~/.gemini/settings.json`. The server (`/mcp`) resolves the task from the token
+  and never trusts an agent-supplied ID.
+- **LocalRunner** declares the four tools as OpenAI function-calling defs and
+  runs a tool-use loop, re-feeding tool results as message history.
+
+`ask_user` records a clarification and returns `ErrAgentBlocked`; the runner
+turns the buffered question into a `*BlockedError` and the task blocks.
 
 ### Task YAML Format
 
@@ -162,9 +184,17 @@ tasks:      id, name, description, config_json, priority, timeout_ns, retry_json
 
 executions: id, task_id, start_time, end_time, exit_code, status, stdout_path,
             stderr_path, artifact_dir, cost_usd, error_msg, session_id,
-            sandbox_dir, changestats_json, commits_json
+            sandbox_dir, changestats_json, commits_json, tokens_in, tokens_out,
+            agent
+
+events:     id, task_id, seq, ts, kind, actor, payload_json
 ```
 
+The `events` table is the per-task observability stream (`internal/event`); see
+the REST `GET /api/tasks/{id}/events` and the chatbot MCP `get_events` tool.
+`executions.agent` records which provider ran each execution, for budget
+accounting (`SpendByProviderSince`).
+
 JSON blobs: `config_json` (AgentConfig), `retry_json`, `tags_json`, `depends_on_json`, `interactions_json`, `changestats_json`, `commits_json`.
 
 ---
@@ -173,13 +203,13 @@ JSON blobs: `config_json` (AgentConfig), `retry_json`, `tags_json`, `depends_on_
 
 ### Planning Preamble & Orchestration
 
-When `agent.skip_planning` is false (the default), `withPlanningPreamble()` prepends a system-level prompt to the agent's instructions that:
-- Instructs the agent to POST subtasks to `$CLAUDOMATOR_API_URL/api/tasks` and stop if the task will take more than ~3 minutes
-- Instructs the agent to write a JSON question to `$CLAUDOMATOR_QUESTION_FILE` and exit if it needs user input
-- Requires all changes to be committed before exit
-- Requires a summary written to `$CLAUDOMATOR_SUMMARY_FILE`
+When `agent.skip_planning` is false (the default), `withPlanningPreamble()` prepends a system-level prompt that points the agent at its MCP/tool-use back-channel rather than file/REST conventions:
+- Break work taking more than ~3 minutes into pieces via the `spawn_subtask` tool, then stop
+- Call `ask_user` (and end the turn) when it needs a decision
+- Commit all changes before finishing
+- Call `report_summary` before finishing
 
-Env vars injected into every execution: `CLAUDOMATOR_API_URL`, `CLAUDOMATOR_TASK_ID`, `CLAUDOMATOR_PROJECT_DIR`, `CLAUDOMATOR_QUESTION_FILE`, `CLAUDOMATOR_SUMMARY_FILE`.
+The four tools — `ask_user`, `report_summary`, `spawn_subtask`, `record_progress` — are served over MCP (claude/gemini) or OpenAI tool-use (local); the old `CLAUDOMATOR_QUESTION_FILE`/`CLAUDOMATOR_SUMMARY_FILE` conventions were removed.
 
 ### Changestats
 
@@ -217,11 +247,28 @@ Tasks created for:
 
 Tagged `["ci", "auto"]`, capped at $3 USD, allowed tools: Read, Edit, Bash, Glob, Grep.
 
-### Elaborate Endpoint
+### Task creation (chatbot-driven)
 
-`POST /api/tasks/elaborate` converts natural language → task JSON via a `claude --prompt` invocation. Optionally reads `CLAUDE.md` / `SESSION_STATE.md` from a configured working directory for context. Per-IP rate-limited.
+The natural-language `POST /api/tasks/elaborate` endpoint and the web create
+form were removed. Chatbots now create and drive tasks through the chatbot MCP
+server (`/chatbot/mcp`, shared bearer token): `submit_task`, `list_tasks`,
+`get_task`, `get_events`, `answer_question`, `accept_task`, `reject_task`,
+`cancel_task`. The web UI is observability-only (task tree, live logs, event
+timeline, accept/reject, budget headroom).
 
-> **Implementation gap:** The elaborate endpoint is not tested against real Claude invocations. `sanitizeElaboratedTask()` uses keyword heuristics to infer missing tools (fragile). No caching.
+### Budget gating
+
+`internal/budget.Accountant` tracks per-provider spend in a rolling window
+(default 5h, `[budget]` config). Before running, the dispatcher reroutes paid
+work to the free local runner when a cap would be breached, else blocks the task
+(`BUDGET_EXCEEDED`). `GET /api/budget` surfaces headroom. Local runners are free.
+
+### Auth / binding
+
+The server binds `127.0.0.1` by default and refuses a non-loopback bind unless
+`external_bind_allowed = true` (front external access with a reverse proxy). The
+shared `api_token` guards the web UI + chatbot MCP; agent MCP uses per-task
+tokens; GitHub webhooks use HMAC.
 
 ### Model Classifier
 
@@ -236,13 +283,25 @@ Tagged `["ci", "auto"]`, capped at $3 USD, allowed tools: Read, Edit, Bash, Glob
 
 ## Design Debt
 
-### GeminiRunner is a non-functional stub
+### Stories / checker backend not yet removed (Phase 8 follow-up)
+
+The web stories dashboard was removed, but the backend remains: the `stories`
+table + `internal/api/stories.go` handlers + `/api/stories/*` routes,
+story-elaborate in `elaborate.go`, and the story/checker logic woven into
+`executor.handleRunResult` (`checkStoryCompletion`, `spawnCheckerTask`,
+`triggerStoryDeploy`, `ensureStoryBranch`) plus the `tasks` story/checker
+columns. This is a large, lifecycle-touching teardown best done as its own pass.
+
+### Deprecated task columns not yet dropped (Phase 8 follow-up)
 
-`internal/executor/gemini.go` `execOnce()` does not run the `gemini` binary. It starts a goroutine that writes hardcoded fake JSON to a pipe. `parseGeminiStream()` strips markdown fences but does no semantic parsing. There is no session/resume support.
+`tasks.question_json`, `summary`, `interactions_json`, `elaboration_input` are
+superseded by the `events` table but still read by the answer flow and task
+panel. Dropping them requires migrating those reads to events first.
 
-Any task with `agent.type: "gemini"` will silently return canned output. This is dangerous in production.
+### Gemini tool-use unverified
 
-**Decision needed:** Either implement GeminiRunner properly (subprocess + stream parsing + sandbox integration mirroring ClaudeRunner) or remove it and the `Classifier` from the codebase until it's ready.
+The gemini MCP wiring (`~/.gemini/settings.json`) is in place but was never run
+against a real `gemini` binary — see `cmd/spike` (`go run ./cmd/spike gemini`).
 
 ### Priority field is stored but never used
 
@@ -300,8 +359,11 @@ In `executor.go`, `withFailureHistory` creates a copy of the task struct (`copy
 | GET | `/api/tasks/{id}/logs/stream` | Stream latest execution logs |
 | GET | `/api/executions` | List recent executions across all tasks |
 | GET | `/api/tasks/{id}/deployment-status` | Poll deployment readiness |
-| POST | `/api/tasks/elaborate` | Convert natural language → task JSON |
+| GET | `/api/tasks/{id}/events` | Task observability event stream (`?since_seq=N`) |
+| GET | `/api/budget` | Per-provider rolling-window spend headroom |
 | POST | `/api/tasks/validate` | Validate task JSON |
+| POST/GET/DELETE | `/mcp` | Per-task agent MCP back-channel (bearer = minted token) |
+| POST/GET/DELETE | `/chatbot/mcp` | Chatbot MCP server (bearer = `api_token`) |
 | POST | `/api/scripts/{name}` | Run named script with task context |
 | GET | `/api/ws` | WebSocket upgrade (live task updates) |
 | GET | `/api/workspaces` | List directories under `workspace_root` |