summaryrefslogtreecommitdiff
path: root/docs/plans
diff options
context:
space:
mode:
authorClaude <noreply@anthropic.com>2026-05-02 07:54:51 +0000
committerClaude <noreply@anthropic.com>2026-05-02 07:54:51 +0000
commit6c5762848f4f3114a6ece9ce0bc70a84fca040ce (patch)
treec118fe596c66b23dbf23d7aee5d6d6f823d0903a /docs/plans
parentae833b2765c7c8086bf8e1ea8e8ec8ee9b73e656 (diff)
feat(api): enrich CI failure task instructions via local LLM
Phase 3 of "local OSS models as agents" plan. When the webhook handler creates a task for a failed CI run AND a local LLM is configured on the server, the hardcoded 4-step investigation template is replaced with a project-aware investigation plan generated by the LLM. Scope adjustment from the original sketch: the original plan said "summarize fetched workflow logs", but fetching logs requires GitHub API auth that isn't wired. Narrowed to project-context triage — recent git log + CLAUDE.md content + webhook metadata, fed to the LLM with a system prompt asking for 6-12 lines of concrete next steps. Deferred GitHub log fetching to post-epic cleanup. Implementation: - New internal/api/webhook_llm.go holds enrichCIInstructions and its helpers (readRecentCommits via `git log`, readProjectDoc). - enrichCIInstructions is truly additive: any failure mode (no client, HTTP error, empty body, 10s timeout) returns the original fallback template unchanged. Existing webhook tests pass byte-for-byte. - Always preserves a metadata header (repo/branch/SHA/check/URL) ahead of the LLM body so investigators don't lose context if the LLM is terse. - Reuses s.llm (set via Server.SetLLM in Phase 2) — no new config knob, no per-feature gating. Asymmetric opt-out (yes-elaborate, no-CI-triage) deferred until there's actual demand. Tests: - enrichCIInstructions: nil client, LLM 500, empty body all return fallback unchanged. - enrichCIInstructions: success path produces enriched body with metadata header preserved; user prompt contains repo/branch/SHA. - enrichCIInstructions: real git repo (init + 2 commits) → recent commits appear in user prompt. - Webhook handler regression guard: no-LLM path produces the exact legacy template substrings. - Webhook handler with LLM stubbed: task instructions contain LLM body + metadata header. Plan: docs/plans/local-oss-runner.md. https://claude.ai/code/session_017Edeq947TpSm1vQTxMhi1J
Diffstat (limited to 'docs/plans')
-rw-r--r--docs/plans/local-oss-runner.md57
1 files changed, 57 insertions, 0 deletions
diff --git a/docs/plans/local-oss-runner.md b/docs/plans/local-oss-runner.md
index 108495b..c065483 100644
--- a/docs/plans/local-oss-runner.md
+++ b/docs/plans/local-oss-runner.md
@@ -247,3 +247,60 @@ Second-cheapest, second-highest-volume LLM call after classification (one per ta
- `prefer_local_for_elaborate=false` short-circuits to Claude path (preserves current behavior when user opts out)
- Local-failure fallback to Claude verified by test
- Branch pushed
+
+---
+
+# Phase 3 — Focused Plan (CI Failure Triage)
+
+## Scope adjustment from the original sketch
+
+The original Phase 3 sketch was "summarize fetched workflow logs". Fetching GitHub workflow logs requires authenticated GitHub API access (PAT or app token), which is out of scope and would balloon this phase into a GitHub-integration epic. Narrow Phase 3 to **project-context-based triage** — use signals we already have without new dependencies.
+
+What we have at webhook time: `repository.full_name`, `branch`, `SHA`, `check_name`, `html_url`, plus (when matched) a project directory we can read locally.
+
+What the LLM can do with that: produce a tighter, project-aware investigation prompt that names the recent commits, points at suspect files, and gives the agent better starting hypotheses than the current generic 4-step template.
+
+## What ships
+
+- New helper `enrichCIInstructions(ctx, *llm.Client, ciContext, projectDir, fallback string) string`
+- `createCIFailureTask` calls it when `s.llm != nil`; on any error, returns the existing hardcoded template (truly additive — webhook tests for the no-LLM path stay passing unchanged)
+- Helper uses: recent git log (last 5 commits from project_dir if it's a git repo), CLAUDE.md content if present, plus all webhook metadata
+- One configuration knob: reuse `LocalModel.UseForElaborate()` semantics? No — separate flag. Add `LocalModel.PreferForCITriage *bool` defaulting true when endpoint set, opt-out symmetrical with `PreferForElaborate`.
+
+## Explicit non-goals
+
+- No GitHub API integration (no log fetching, no auth)
+- No changes to webhook routing, signature validation, project matching, or task scheduling
+- No changes to the task schema (instructions stays a string)
+- No streaming — one-shot LLM call, sub-2s target
+
+## Task list
+
+1. Add `LocalModel.PreferForCITriage *bool` and `UseForCITriage()` helper, mirroring elaborate
+2. Add `enrichCIInstructions` in `internal/api/webhook.go` (or `webhook_llm.go` if it grows)
+3. Read recent git log from project_dir via `git log --oneline -n 5` (best-effort, swallow errors)
+4. Read CLAUDE.md from project_dir (best-effort)
+5. Build a focused prompt: "CI just failed on this project. Here's metadata + recent commits + project context. Produce a 6-12 line investigation plan that names suspect files/commits when you can, otherwise gives concrete starting steps." Plain text out, not JSON.
+6. Update `createCIFailureTask` to call enrichment when `s.llm != nil && cfg.LocalModel.UseForCITriage()`. Note: the server doesn't currently see the cfg directly — pass the gate as a setter `SetCITriageEnabled(bool)` from serve.go, OR (simpler) just gate on `s.llm != nil` and let users opt out by not calling `SetLLM`. Going with the simpler option since it matches the elaborate split: same `s.llm` for both, server doesn't track per-feature gates.
+7. Wiring in `serve.go`: when `cfg.LocalModel.Endpoint != ""`, `SetLLM(localClient)`. (Already done in Phase 2.) Per-feature opt-out via the `PreferFor*` config flags is read at wire time and could conditionally not call SetLLM, but that gives elaborate/CI an all-or-nothing toggle which is wrong. Better: introduce a separate setter `SetLLMForCITriage` so each feature can be controlled independently.
+
+ Actually, simplest and cleanest: keep one `SetLLM` setter, and gate each call site (`elaborateWithLocal`, `enrichCIInstructions`) by reading a per-feature config flag passed via separate setters. That's getting fiddly. Step back.
+
+ **Final decision:** the per-feature gate doesn't pull its weight in Phase 3. Ship it as: `s.llm != nil` enables both elaborate and CI triage. Users who want elaborate-yes/CI-triage-no can revisit later. The deferred per-feature toggles get added in the post-epic cleanup along with token telemetry — there's no real demand for the asymmetric case yet.
+
+ Revised: drop `PreferForCITriage` entirely; ship a simpler thing.
+8. Tests:
+ - `enrichCIInstructions` with stub LLM returns the LLM body
+ - `enrichCIInstructions` with failing LLM returns `fallback` unchanged
+ - `enrichCIInstructions` includes recent git log when project_dir is a real git repo (use `t.TempDir()` + `git init` + a commit)
+ - Webhook handler test: LLM configured → instructions reflect LLM output
+ - Webhook handler test: LLM not configured → instructions match the existing template byte-for-byte (regression guard)
+9. `go build ./... && go test -race ./...`
+10. Commit as Phase 3 on the same branch
+11. Push
+
+## Stop conditions
+
+- All new tests green under `-race`
+- Existing webhook tests pass byte-for-byte when LLM not configured
+- Build clean; pushed