diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/workflow-and-entry-points.md | 137 |
1 files changed, 78 insertions, 59 deletions
diff --git a/docs/workflow-and-entry-points.md b/docs/workflow-and-entry-points.md index d9312c5..367c5ba 100644 --- a/docs/workflow-and-entry-points.md +++ b/docs/workflow-and-entry-points.md @@ -8,41 +8,41 @@ arbitrated-review mechanism sits relative to that state machine. ## 1. Entry Points Overview -Every arrow below eventually reaches `executor.Pool`, the single choke-point -that bounds concurrency and gates spend. +Every path that ultimately executes work passes through `executor.Pool`, the single +choke-point that bounds concurrency and gates spend. Some entry points submit +directly; others only write storage and let a poll loop or a human /run call submit. ```mermaid flowchart TD subgraph CLI["CLI (claudomator run)"] - CLI_RUN["claudomator run <file.yaml>\n(internal/cli/run.go)"] + CLI_RUN["claudomator run <file.yaml>\n(internal/cli/run.go)\nParses YAML, creates + submits directly"] end subgraph REST["REST API"] - API_CREATE["POST /api/tasks\n(creates PENDING task)"] - API_RUN["POST /api/tasks/{id}/run\n(submits PENDING → QUEUED)"] + API_CREATE["POST /api/tasks\n(creates PENDING task in storage only)"] + API_RUN["POST /api/tasks/{id}/run\n(PENDING → QUEUED + Pool.Submit)"] API_CREATE --> API_RUN end subgraph CHATBOT["Chatbot MCP (/chatbot/mcp)"] - CB_SUBMIT["submit_task tool\n(internal/api/chatbotmcp.go)"] - CB_STORY["create_story tool\n(creates Story row)"] - CB_ACCEPT["accept_story tool\n(REVIEW_READY → DONE)"] - CB_LIST["list_stories / get_story tools\n(read-only)"] + CB_SUBMIT["submit_task tool\n(creates + submits in one step)"] + CB_STORY["create_story tool\n(creates Story row in storage only;\nStoryOrchestrator spawns Builder on next tick)"] + CB_ACCEPT["accept_story tool\n(REVIEW_READY → DONE;\norchestrator triggers retro next tick)"] end subgraph WEBHOOK["GitHub Webhook"] - WH["POST /api/webhooks/github\ncheck_run / workflow_run failure\n(internal/api/webhook.go)"] + WH["POST /api/webhooks/github\ncheck_run / workflow_run failure\n(internal/api/webhook.go)\ncreates PENDING task in storage;\ndoes NOT auto-submit to pool"] end subgraph STORY_API["Story REST API"] - ST_CREATE["POST /api/stories\n(creates Story row)"] + ST_CREATE["POST /api/stories\n(creates Story row in storage only)"] ST_ACCEPT["POST /api/stories/{id}/accept\n(REVIEW_READY → DONE)"] end subgraph AGENT_TOOLS["Agent Back-channel Tools\n(MCP /mcp or native tool-use loop)"] - AG_SPAWN["spawn_subtask\n(no role= → child with parent_task_id)\n(role= → sibling with depends_on)"] - AG_EPIC["propose_epic\n(creates/updates Epic, groups Stories)"] - AG_RC["propose_role_config\n(creates draft role_configs version)"] + AG_SPAWN["spawn_subtask\n(creates PENDING child task;\nparent_task_id always set;\ndoes NOT auto-submit to pool)"] + AG_EPIC["propose_epic\n(creates/updates Epic row;\ngroups Stories; storage only)"] + AG_RC["propose_role_config\n(creates draft role_configs row;\nstorage only)"] end subgraph ORCHESTRATOR["StoryOrchestrator (poll, 15 s)\n(internal/scheduler/story_orchestrator.go)"] @@ -57,32 +57,34 @@ flowchart TD SC_TIMEOUT["tickAskUserTimeouts\n(auto-answer + resume BLOCKED tasks\noutstanding > ask_user_timeout)"] end + STORAGE[("SQLite storage\n(tasks table)")] POOL(["executor.Pool\n(bounded dispatcher)"]) - %% CLI path - CLI_RUN -->|"Pool.Submit() directly"| POOL + %% CLI path — direct submit + CLI_RUN -->|"creates + QUEUED + Pool.Submit()"| POOL - %% REST path + %% REST path — two-step API_RUN -->|"Pool.Submit()"| POOL - %% Chatbot MCP paths - CB_SUBMIT -->|"creates task + auto-submits"| POOL - CB_STORY -->|"story row only;\nStoryOrchestrator spawns Builder"| ORCHESTRATOR - CB_ACCEPT -->|"story REVIEW_READY → DONE;\ntriggers processRetro next tick"| ORCHESTRATOR + %% Chatbot MCP — submit_task is direct; story tools are storage-only + CB_SUBMIT -->|"creates + QUEUED + Pool.Submit()"| POOL + CB_STORY -->|"Story row only"| STORAGE + CB_ACCEPT -->|"Story REVIEW_READY → DONE"| STORAGE - %% Webhook path - WH -->|"creates task + auto-runs"| POOL + %% Webhook — creates PENDING task only; needs /run to execute + WH -->|"creates PENDING task"| STORAGE - %% Story REST paths - ST_CREATE -->|"story row only"| ORCHESTRATOR - ST_ACCEPT -->|"REVIEW_READY → DONE"| ORCHESTRATOR + %% Story REST — storage-only; orchestrator picks up on next tick + ST_CREATE -->|"Story row only"| STORAGE + ST_ACCEPT -->|"Story REVIEW_READY → DONE"| STORAGE - %% Agent tool paths - AG_SPAWN -->|"Pool.Submit() for new child/sibling"| POOL - AG_EPIC -->|"storage only\n(no task spawned)"| POOL - AG_RC -->|"storage only\n(no task spawned)"| POOL + %% Agent tools — spawn_subtask is storage-only; propose_epic/role_config are pure storage + AG_SPAWN -->|"creates PENDING child task\n(scripts/start-next-task,\nStoryOrchestrator, or /run\nsubmit it later)"| STORAGE + AG_EPIC -->|"Epic/Story rows only"| STORAGE + AG_RC -->|"role_configs draft row only"| STORAGE - %% StoryOrchestrator → Pool + %% StoryOrchestrator spawns tasks AND submits them + STORAGE -->|"StoryOrchestrator polls stories;\nspawnRoleTask: CreateTask + QUEUED + Pool.Submit()"| SO_EVAL SO_EVAL -->|"Store.CreateTask + Pool.Submit()"| POOL SO_ARB -->|"Store.CreateTask + Pool.Submit()"| POOL SO_FIX -->|"Store.CreateTask + Pool.Submit()"| POOL @@ -92,31 +94,38 @@ flowchart TD SC_FAIL -->|"Pool.Submit() or Pool.SubmitResume()"| POOL SC_TIMEOUT -->|"Pool.SubmitResume()"| POOL - %% Orchestrator watches Pool results - POOL -->|"task reaches READY/COMPLETED/FAILED;\norchestrator polls"| ORCHESTRATOR + %% Pool outcomes feed the schedulers + POOL -->|"task reaches READY/COMPLETED/FAILED;\norchestrator polls storage"| ORCHESTRATOR POOL -->|"task reaches FAILED;\nscheduler polls"| SCHEDULER - POOL -->|"task stays BLOCKED;\nscheduler polls"| SCHEDULER + POOL -->|"task stays BLOCKED;\nscheduler polls (ask_user timeout)"| SCHEDULER ``` +> **Note on PENDING subtasks:** When an agent calls `spawn_subtask`, the child task +> lands in PENDING state in storage. Nothing auto-submits it. Dispatch happens via one +> of: (a) the `scripts/start-next-task` helper (calls `POST /api/tasks/{id}/run`), +> (b) `StoryOrchestrator.spawnRoleTask` if the subtask is a story-pipeline role task, +> (c) a human or chatbot calling `POST /api/tasks/{id}/run`, or +> (d) `RecoverStaleQueued` on server restart for tasks already in QUEUED state. + --- ## 2. Detailed Entry Point Reference | Entry Point | Code Location | What It Creates | Who Submits to Pool | |---|---|---|---| -| `claudomator run <file>` | `internal/cli/run.go` | Task(s) from YAML | `cli` directly via `Pool.Submit` | -| `POST /api/tasks` + `POST /api/tasks/{id}/run` | `internal/api/taskops.go` | PENDING task, then QUEUED | `api.Server` via `Pool.Submit` | -| Chatbot MCP `submit_task` | `internal/api/chatbotmcp.go` | Task, immediately submitted | `chatbotmcp` via `Pool.Submit` | -| Chatbot MCP `create_story` | `internal/api/chatbotmcp.go` | Story row | `StoryOrchestrator` (next tick) | -| Chatbot MCP `accept_story` | `internal/api/chatbotmcp.go` | Story status DONE | `StoryOrchestrator` (triggers retro) | -| `POST /api/stories` | `internal/api/stories.go` | Story row | `StoryOrchestrator` (next tick) | -| `POST /api/stories/{id}/accept` | `internal/api/stories.go` | Story status DONE | `StoryOrchestrator` (triggers retro) | -| `POST /api/webhooks/github` | `internal/api/webhook.go` | Task auto-tagged `["ci","auto"]` | `api.Server` via `Pool.Submit` | -| Agent `spawn_subtask` (no `role=`) | `internal/executor/agentmcp.go`, `internal/agentloop/tools.go` | Child task with `parent_task_id` | `Pool.Submit` | -| Agent `spawn_subtask` (with `role=`) | same | Sibling task with `depends_on`, `Agent.Role` set, no Type/Model | `Pool.Submit`; role resolved at dispatch | -| Agent `propose_epic` | same | Epic row; groups existing stories | storage only, no task | -| Agent `propose_role_config` | same | Draft `role_configs` version | storage only, no task | -| `StoryOrchestrator.ensureEvaluators` | `internal/scheduler/story_orchestrator.go` | 4 Evaluator tasks per builder node | `Store.CreateTask` + `Pool.Submit` | +| `claudomator run <file>` | `internal/cli/run.go` | Task(s) from YAML | `cli` directly: `CreateTask` → `UpdateTaskState(QUEUED)` → `Pool.Submit` | +| `POST /api/tasks` + `POST /api/tasks/{id}/run` | `internal/api/server.go`, `taskops.go` | PENDING task, then QUEUED | `handleRunTask` via `Pool.Submit` | +| Chatbot MCP `submit_task` | `internal/api/chatbotmcp.go` | Task (PENDING→QUEUED in one step) | `submitTask()` via `Pool.Submit` | +| Chatbot MCP `create_story` | `internal/api/chatbotmcp.go` | Story row in storage | `StoryOrchestrator` (next tick): creates + submits Builder task | +| Chatbot MCP `accept_story` | `internal/api/chatbotmcp.go` | Story status DONE in storage | `StoryOrchestrator` (next tick): spawns retro task | +| `POST /api/stories` | `internal/api/stories.go` | Story row in storage | `StoryOrchestrator` (next tick) | +| `POST /api/stories/{id}/accept` | `internal/api/stories.go` | Story status DONE in storage | `StoryOrchestrator` (next tick): spawns retro task | +| `POST /api/webhooks/github` | `internal/api/webhook.go` | Task in **PENDING** state only | Human/chatbot/script must call `POST /api/tasks/{id}/run` | +| Agent `spawn_subtask` (no `role=`) | `internal/executor/agentmcp.go`, `internal/agentloop/tools.go` | Child task with `parent_task_id`, `agent.type=claude` | **Not auto-submitted**; parent goes BLOCKED; child is dispatched later | +| Agent `spawn_subtask` (with `role=`) | same | Child task with `parent_task_id`, `agent.role` set, no Type/Model | **Not auto-submitted**; role resolved at eventual dispatch via `Pool.execute()` | +| Agent `propose_epic` | `internal/executor/agentmcp.go`, `internal/agentloop/tools.go` | Epic row; links existing stories | Storage only — no task created, no pool call | +| Agent `propose_role_config` | same | Draft `role_configs` version | Storage only — no task created, no pool call | +| `StoryOrchestrator.ensureEvaluators` | `internal/scheduler/story_orchestrator.go` | 4 Evaluator tasks per builder node | `spawnRoleTask`: `CreateTask` → `UpdateTaskState(QUEUED)` → `Pool.Submit` | | `StoryOrchestrator.ensureArbitration` | same | 1 `planner`-role Arbitration task | same | | `StoryOrchestrator.ensureFixAttempt` | same | New root builder task after rejection | same | | `StoryOrchestrator.spawnNestedFixAttempt` | same | New nested builder task after rejection | same | @@ -130,7 +139,7 @@ flowchart TD ```mermaid stateDiagram-v2 - [*] --> PENDING : task created + [*] --> PENDING : task created\n(POST /api/tasks, webhook,\nspawn_subtask, story rows) PENDING --> QUEUED : /run submitted\n(Pool.Submit) @@ -206,13 +215,13 @@ flowchart TD subgraph ReviewCycle["Review Cycle (per builder node, any depth)"] EVAL["4 Evaluator tasks\nevaluator_quality\nevaluator_security\nevaluator_correctness\nevaluator_performance\n\n(depends_on: builder node)"] ARB["1 Arbitration task\nrole: planner\n\n(depends_on: all 4 evaluators)"] - APPROVE{"Approved?"} + APPROVE{"Approved?\n(report_verdict event;\ndefault: approve)"} FIX["Fix-attempt task\n(same role, new builder;\ndepends_on: rejected node)"] end subgraph Terminal["Terminal states"] COMP["builder node → COMPLETED\n(if root: story → REVIEW_READY)"] - HUMAN["POST /api/stories/{id}/accept\n(REVIEW_READY → DONE)"] + HUMAN["POST /api/stories/{id}/accept\n(REVIEW_READY → DONE)\n[only manual step in the chain]"] RETRO["processRetro: retro-role task\nreflects, proposes role_configs drafts"] DONE["story → DONE"] end @@ -241,13 +250,14 @@ Once work reaches the pool, the internal path is: flowchart LR SUBMIT["Pool.Submit(task)\nor Pool.SubmitResume(task, exec)"] QUEUE["workCh buffered channel\n(FIFO, no priority ordering)"] - DISPATCH["dispatch() goroutine\nwaits for semaphore slot\nchecks budget gate\nchecks depends_on satisfied"] - RUNNER["ContainerRunner\n(Docker; git clone; claude -p)\nor LocalRunner (OpenAI-compatible HTTP)"] - RESULT["handleRunResult\nwrite execution row\nupdate task state\nbroadcast WebSocket event\ncascade-fail dependents if terminal-fail"] + DISPATCH["dispatch() goroutine\nwaits for semaphore slot\nchecks budget gate\nchecks depends_on satisfied\nresolves role → Type/Model (role-typed tasks)"] + RUNNER["ContainerRunner\n(Docker; git clone; claude -p)\nor NativeRunner (provider.Provider tool-use loop)"] + RESULT["handleRunResult\nwrite execution row\nupdate task state\nbroadcast WebSocket event\ncascade-fail dependents if terminal-fail\nmaybeUnblockParent if nested task completes"] SUBMIT --> QUEUE --> DISPATCH --> RUNNER --> RESULT RESULT -->|"READY or COMPLETED"| DONE2(["task observable\nvia WS / REST"]) - RESULT -->|"BLOCKED"| BLOCKED2(["workspace preserved\nin sandbox_dir;\nawait answer"]) + RESULT -->|"BLOCKED (ask_user)"| BLOCKED2(["workspace preserved\nin sandbox_dir;\nawait answer"]) + RESULT -->|"BLOCKED (subtasks)"| SUBTASK(["parent BLOCKED;\nPENDING children\nneed separate /run"]) RESULT -->|"FAILED etc."| FAILED2(["Scheduler picks up\non next poll tick"]) ``` @@ -255,11 +265,20 @@ flowchart LR ## Notes -- **`propose_epic` / `propose_role_config`** are agent tools that write to - storage only — they do not create tasks or trigger any further scheduling. - They exist to let a running agent record planning-layer artifacts (epic - groupings; improved role system prompts for human review) as a side-effect of - its execution. +- **`propose_epic` / `propose_role_config`** write to storage only — no tasks + are created and no pool call is made. They exist to let a running agent record + planning-layer artifacts as side-effects of its execution. + +- **`spawn_subtask` does not auto-submit.** The child task is created PENDING. + The parent task's execution then ends, `handleRunResult` detects the pending + children and moves the parent to BLOCKED. Something external must later call + `POST /api/tasks/{id}/run` (or `claudomator start`) for each PENDING child. + The `scripts/start-next-task` helper automates this; `StoryOrchestrator` does + it automatically for story-pipeline role tasks via `spawnRoleTask`. + +- **GitHub webhook does not auto-run.** `createCIFailureTask` stores the task + at PENDING and returns the task ID. A human, chatbot, or polling script must + explicitly call `POST /api/tasks/{id}/run` to queue it. - **`story.RootTaskID` is immutable** after story creation. When a builder node is rejected and replaced, `task.CurrentAttempt(store, anchorID)` walks the |
