diff options
| author | Claude <noreply@anthropic.com> | 2026-05-02 07:54:51 +0000 |
|---|---|---|
| committer | Claude <noreply@anthropic.com> | 2026-05-02 07:54:51 +0000 |
| commit | 6c5762848f4f3114a6ece9ce0bc70a84fca040ce (patch) | |
| tree | c118fe596c66b23dbf23d7aee5d6d6f823d0903a /docs | |
| parent | ae833b2765c7c8086bf8e1ea8e8ec8ee9b73e656 (diff) | |
feat(api): enrich CI failure task instructions via local LLM
Phase 3 of "local OSS models as agents" plan. When the webhook handler
creates a task for a failed CI run AND a local LLM is configured on
the server, the hardcoded 4-step investigation template is replaced
with a project-aware investigation plan generated by the LLM.
Scope adjustment from the original sketch: the original plan said
"summarize fetched workflow logs", but fetching logs requires GitHub
API auth that isn't wired. Narrowed to project-context triage —
recent git log + CLAUDE.md content + webhook metadata, fed to the
LLM with a system prompt asking for 6-12 lines of concrete next
steps. Deferred GitHub log fetching to post-epic cleanup.
Implementation:
- New internal/api/webhook_llm.go holds enrichCIInstructions and its
helpers (readRecentCommits via `git log`, readProjectDoc).
- enrichCIInstructions is truly additive: any failure mode (no client,
HTTP error, empty body, 10s timeout) returns the original fallback
template unchanged. Existing webhook tests pass byte-for-byte.
- Always preserves a metadata header (repo/branch/SHA/check/URL)
ahead of the LLM body so investigators don't lose context if the
LLM is terse.
- Reuses s.llm (set via Server.SetLLM in Phase 2) — no new config
knob, no per-feature gating. Asymmetric opt-out (yes-elaborate,
no-CI-triage) deferred until there's actual demand.
Tests:
- enrichCIInstructions: nil client, LLM 500, empty body all return
fallback unchanged.
- enrichCIInstructions: success path produces enriched body with
metadata header preserved; user prompt contains repo/branch/SHA.
- enrichCIInstructions: real git repo (init + 2 commits) → recent
commits appear in user prompt.
- Webhook handler regression guard: no-LLM path produces the exact
legacy template substrings.
- Webhook handler with LLM stubbed: task instructions contain LLM
body + metadata header.
Plan: docs/plans/local-oss-runner.md.
https://claude.ai/code/session_017Edeq947TpSm1vQTxMhi1J
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/plans/local-oss-runner.md | 57 |
1 files changed, 57 insertions, 0 deletions
diff --git a/docs/plans/local-oss-runner.md b/docs/plans/local-oss-runner.md index 108495b..c065483 100644 --- a/docs/plans/local-oss-runner.md +++ b/docs/plans/local-oss-runner.md @@ -247,3 +247,60 @@ Second-cheapest, second-highest-volume LLM call after classification (one per ta - `prefer_local_for_elaborate=false` short-circuits to Claude path (preserves current behavior when user opts out) - Local-failure fallback to Claude verified by test - Branch pushed + +--- + +# Phase 3 — Focused Plan (CI Failure Triage) + +## Scope adjustment from the original sketch + +The original Phase 3 sketch was "summarize fetched workflow logs". Fetching GitHub workflow logs requires authenticated GitHub API access (PAT or app token), which is out of scope and would balloon this phase into a GitHub-integration epic. Narrow Phase 3 to **project-context-based triage** — use signals we already have without new dependencies. + +What we have at webhook time: `repository.full_name`, `branch`, `SHA`, `check_name`, `html_url`, plus (when matched) a project directory we can read locally. + +What the LLM can do with that: produce a tighter, project-aware investigation prompt that names the recent commits, points at suspect files, and gives the agent better starting hypotheses than the current generic 4-step template. + +## What ships + +- New helper `enrichCIInstructions(ctx, *llm.Client, ciContext, projectDir, fallback string) string` +- `createCIFailureTask` calls it when `s.llm != nil`; on any error, returns the existing hardcoded template (truly additive — webhook tests for the no-LLM path stay passing unchanged) +- Helper uses: recent git log (last 5 commits from project_dir if it's a git repo), CLAUDE.md content if present, plus all webhook metadata +- One configuration knob: reuse `LocalModel.UseForElaborate()` semantics? No — separate flag. Add `LocalModel.PreferForCITriage *bool` defaulting true when endpoint set, opt-out symmetrical with `PreferForElaborate`. + +## Explicit non-goals + +- No GitHub API integration (no log fetching, no auth) +- No changes to webhook routing, signature validation, project matching, or task scheduling +- No changes to the task schema (instructions stays a string) +- No streaming — one-shot LLM call, sub-2s target + +## Task list + +1. Add `LocalModel.PreferForCITriage *bool` and `UseForCITriage()` helper, mirroring elaborate +2. Add `enrichCIInstructions` in `internal/api/webhook.go` (or `webhook_llm.go` if it grows) +3. Read recent git log from project_dir via `git log --oneline -n 5` (best-effort, swallow errors) +4. Read CLAUDE.md from project_dir (best-effort) +5. Build a focused prompt: "CI just failed on this project. Here's metadata + recent commits + project context. Produce a 6-12 line investigation plan that names suspect files/commits when you can, otherwise gives concrete starting steps." Plain text out, not JSON. +6. Update `createCIFailureTask` to call enrichment when `s.llm != nil && cfg.LocalModel.UseForCITriage()`. Note: the server doesn't currently see the cfg directly — pass the gate as a setter `SetCITriageEnabled(bool)` from serve.go, OR (simpler) just gate on `s.llm != nil` and let users opt out by not calling `SetLLM`. Going with the simpler option since it matches the elaborate split: same `s.llm` for both, server doesn't track per-feature gates. +7. Wiring in `serve.go`: when `cfg.LocalModel.Endpoint != ""`, `SetLLM(localClient)`. (Already done in Phase 2.) Per-feature opt-out via the `PreferFor*` config flags is read at wire time and could conditionally not call SetLLM, but that gives elaborate/CI an all-or-nothing toggle which is wrong. Better: introduce a separate setter `SetLLMForCITriage` so each feature can be controlled independently. + + Actually, simplest and cleanest: keep one `SetLLM` setter, and gate each call site (`elaborateWithLocal`, `enrichCIInstructions`) by reading a per-feature config flag passed via separate setters. That's getting fiddly. Step back. + + **Final decision:** the per-feature gate doesn't pull its weight in Phase 3. Ship it as: `s.llm != nil` enables both elaborate and CI triage. Users who want elaborate-yes/CI-triage-no can revisit later. The deferred per-feature toggles get added in the post-epic cleanup along with token telemetry — there's no real demand for the asymmetric case yet. + + Revised: drop `PreferForCITriage` entirely; ship a simpler thing. +8. Tests: + - `enrichCIInstructions` with stub LLM returns the LLM body + - `enrichCIInstructions` with failing LLM returns `fallback` unchanged + - `enrichCIInstructions` includes recent git log when project_dir is a real git repo (use `t.TempDir()` + `git init` + a commit) + - Webhook handler test: LLM configured → instructions reflect LLM output + - Webhook handler test: LLM not configured → instructions match the existing template byte-for-byte (regression guard) +9. `go build ./... && go test -race ./...` +10. Commit as Phase 3 on the same branch +11. Push + +## Stop conditions + +- All new tests green under `-race` +- Existing webhook tests pass byte-for-byte when LLM not configured +- Build clean; pushed |
