feat(role): add versioned role configs + escalation ladder + scheduler (Phase 5)

Two parts:

Part A (fixes a gap from Phase 1): Groq/OpenRouter/OpenAI were documented in
docs/api-keys-setup.md as usable once configured, but nothing actually
constructed runners for them. internal/cli/cloudrunners.go consolidates
anthropic/google/groq/openrouter/openai NativeRunner construction into one
table-driven registerCloudRunners() helper, replacing the two hand-written
per-provider blocks in serve.go/run.go. Groq/OpenRouter/OpenAI reuse
openaicompat (no new adapter code) at SandboxKind: "docker".

Part B: the token-husbanding harness's core routing mechanism.

- internal/role: RoleConfig/Tier/Rung -- a role's system prompt and a
  multi-tier (provider, model) escalation ladder, versioned via config_json.
- storage: new role_configs table (draft/active/retired, UNIQUE(role,
  version)) with transactional activate-retires-prior-active semantics;
  new executions.escalation_rung column.
- task.AgentConfig.Role string -- purely additive; every existing task shape
  (Agent.Role == "") is unaffected, proven by TestPool_Execute_NonRoleTask_Unaffected
  plus the full pre-existing suite passing unchanged.
- executor.Pool.execute(): role-typed tasks with no Agent.Type yet resolve
  tier 0 of their active ladder (round-robin across multi-candidate tiers,
  skipping rate-limited providers, falling back to soonest-clearing) before
  the existing pickAgent/Classifier path runs; SystemPrompt applies to
  Agent.SystemPromptAppend. Already-resolved role tasks (scheduler resubmits)
  get their escalation_rung re-derived read-only via findTierIndex.
- internal/scheduler: polls role-typed FAILED tasks, retries at the same
  rung under MaxRetries or escalates to the next tier's first candidate when
  budget.Accountant.Allow() permits (emitting event.KindEscalated), else
  leaves the task FAILED with a final:true KindEscalated event. An
  in-memory per-execution-ID "handled" set keeps the poll loop convergent.
  Started by `serve` only, config knob [scheduler].poll_interval_seconds.
- internal/api: POST/GET /api/roles/{role}/versions, POST
  /api/roles/{role}/activate -- unauthenticated, matching the existing
  projects/tasks REST endpoints' auth posture (only chatbot MCP, agent MCP,
  and WebSocket are api_token-gated in this codebase today).

Documented as stored-but-not-yet-enforced (CLAUDE.md Design Debt, matching
how task.Priority/RetryConfig are already documented): RoleConfig.Tools/
SandboxKind don't affect dispatch yet; DefaultBudgetUSD is read narrowly as
the scheduler's escalation cost estimate, not enforced at initial dispatch;
scheduler escalation always targets Candidates[0] (no round-robin, unlike
initial-dispatch tier-0 resolution); the scheduler's dedupe is per-process
and resets on restart (idempotent, harmless).

go build/vet/test -race -count=1 all pass, 21 packages.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V1moSNCJRcP6kykA4tyUSs

4 files changed, 196 insertions, 76 deletions

diff --git a/internal/cli/cloudrunners.go b/internal/cli/cloudrunners.go
new file mode 100644
index 0000000..55b3efb
--- /dev/null
+++ b/internal/cli/cloudrunners.go
@@ -0,0 +1,91 @@
+package cli
+
+import (
+	"log/slog"
+	"net/http"
+	"time"
+
+	"github.com/thepeterstone/claudomator/internal/config"
+	"github.com/thepeterstone/claudomator/internal/executor"
+	"github.com/thepeterstone/claudomator/internal/llm"
+	"github.com/thepeterstone/claudomator/internal/provider"
+	"github.com/thepeterstone/claudomator/internal/provider/anthropic"
+	"github.com/thepeterstone/claudomator/internal/provider/google"
+	"github.com/thepeterstone/claudomator/internal/provider/openaicompat"
+)
+
+// cloudProviderSpec describes how to construct a provider.Provider for one
+// cloud provider entry in cfg.Providers, and which RunnersConfig field gates
+// it. anthropic/google use their native wire-format adapters; groq/
+// openrouter/openai are OpenAI-wire-compatible and reuse the Phase 1
+// openaicompat adapter — no dedicated adapter package needed for those three
+// (see docs/api-keys-setup.md).
+type cloudProviderSpec struct {
+	name    string
+	enabled func(config.RunnersConfig) bool
+	build   func(pc config.ProviderConfig, timeout time.Duration) provider.Provider
+}
+
+func openaiCompatBuild(pc config.ProviderConfig, timeout time.Duration) provider.Provider {
+	return openaicompat.New(&llm.Client{
+		Endpoint:   pc.Endpoint,
+		Model:      pc.DefaultModel,
+		APIKey:     pc.APIKey,
+		HTTPClient: &http.Client{Timeout: timeout},
+	})
+}
+
+var cloudProviderSpecs = []cloudProviderSpec{
+	{
+		name:    "anthropic",
+		enabled: config.RunnersConfig.AnthropicEnabled,
+		build: func(pc config.ProviderConfig, timeout time.Duration) provider.Provider {
+			return anthropic.New(pc.APIKey, pc.Endpoint, timeout)
+		},
+	},
+	{
+		name:    "google",
+		enabled: config.RunnersConfig.GoogleEnabled,
+		build: func(pc config.ProviderConfig, timeout time.Duration) provider.Provider {
+			return google.New(pc.APIKey, pc.Endpoint, timeout)
+		},
+	},
+	{name: "groq", enabled: config.RunnersConfig.GroqEnabled, build: openaiCompatBuild},
+	{name: "openrouter", enabled: config.RunnersConfig.OpenRouterEnabled, build: openaiCompatBuild},
+	{name: "openai", enabled: config.RunnersConfig.OpenAIEnabled, build: openaiCompatBuild},
+}
+
+// registerCloudRunners constructs and adds a NativeRunner (SandboxKind:
+// "docker", matching the original anthropic/google-only wiring — cloud
+// providers get real container isolation, not the weaker HostSandbox local
+// runners use) into runners for every cloud provider in cloudProviderSpecs
+// that has a non-empty cfg.Providers[name].APIKey and is enabled via its
+// RunnersConfig.<Name>Enabled() gate. Called identically from both `serve`
+// and `run` — the only difference between them (logDir) is a parameter, and
+// neither command does anything provider-specific beyond this.
+func registerCloudRunners(runners map[string]executor.Runner, cfg *config.Config, logger *slog.Logger, logDir string) {
+	for _, spec := range cloudProviderSpecs {
+		pc, ok := cfg.Providers[spec.name]
+		if !ok || pc.APIKey == "" || !spec.enabled(cfg.Runners) {
+			continue
+		}
+		timeout := time.Duration(0)
+		if pc.TimeoutSeconds > 0 {
+			timeout = time.Duration(pc.TimeoutSeconds) * time.Second
+		}
+		runners[spec.name] = &executor.NativeRunner{
+			Provider:     spec.build(pc, timeout),
+			Logger:       logger,
+			LogDir:       logDir,
+			DefaultModel: pc.DefaultModel,
+			// Native cloud providers get real container isolation
+			// (sandbox.DockerSandbox) rather than HostSandbox's host-side
+			// path-prefix confinement — see NativeRunner.SandboxKind doc.
+			SandboxKind:  "docker",
+			SandboxImage: cfg.SandboxImage,
+		}
+		if logger != nil {
+			logger.Info(spec.name+" runner registered", "default_model", pc.DefaultModel, "sandbox", "docker")
+		}
+	}
+}
diff --git a/internal/cli/cloudrunners_test.go b/internal/cli/cloudrunners_test.go
new file mode 100644
index 0000000..7943347
--- /dev/null
+++ b/internal/cli/cloudrunners_test.go
@@ -0,0 +1,78 @@
+package cli
+
+import (
+	"log/slog"
+	"os"
+	"testing"
+
+	"github.com/thepeterstone/claudomator/internal/config"
+	"github.com/thepeterstone/claudomator/internal/executor"
+)
+
+// TestRegisterCloudRunners_RegistersAllFiveConfiguredProviders confirms
+// registerCloudRunners (used identically by both `serve` and `run`) wires up
+// a NativeRunner for every cloud provider that has a non-empty API key —
+// including groq/openrouter/openai, which docs/api-keys-setup.md documented
+// as "works once configured" before this phase actually configured them.
+func TestRegisterCloudRunners_RegistersAllFiveConfiguredProviders(t *testing.T) {
+	cfg := &config.Config{
+		SandboxImage: "test-image:latest",
+		Providers: map[string]config.ProviderConfig{
+			"anthropic":  {APIKey: "ak", DefaultModel: "claude-sonnet-5"},
+			"google":     {APIKey: "gk", DefaultModel: "gemini-2.5-flash"},
+			"groq":       {APIKey: "grk", Endpoint: "https://api.groq.com/openai/v1", DefaultModel: "llama-3.3-70b-versatile"},
+			"openrouter": {APIKey: "ork", Endpoint: "https://openrouter.ai/api/v1", DefaultModel: "meta-llama/llama-3.3-70b-instruct:free"},
+			"openai":     {APIKey: "oak", Endpoint: "https://api.openai.com/v1", DefaultModel: "gpt-4o-mini"},
+		},
+	}
+	runners := map[string]executor.Runner{}
+	logger := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelError}))
+	registerCloudRunners(runners, cfg, logger, t.TempDir())
+
+	for _, name := range []string{"anthropic", "google", "groq", "openrouter", "openai"} {
+		r, ok := runners[name]
+		if !ok {
+			t.Errorf("expected a %q runner to be registered", name)
+			continue
+		}
+		nr, ok := r.(*executor.NativeRunner)
+		if !ok {
+			t.Errorf("%q runner should be a *executor.NativeRunner", name)
+			continue
+		}
+		if nr.Provider == nil {
+			t.Errorf("%q runner has a nil Provider", name)
+		}
+		if nr.SandboxKind != "docker" {
+			t.Errorf("%q runner should use SandboxKind \"docker\", got %q", name, nr.SandboxKind)
+		}
+	}
+}
+
+// TestRegisterCloudRunners_SkipsProvidersWithoutAPIKeyOrDisabled confirms
+// providers are skipped when unconfigured (no api_key) or explicitly
+// disabled via [runners], matching the anthropic/google-only behavior this
+// generalizes.
+func TestRegisterCloudRunners_SkipsProvidersWithoutAPIKeyOrDisabled(t *testing.T) {
+	f := false
+	cfg := &config.Config{
+		Providers: map[string]config.ProviderConfig{
+			"groq":       {APIKey: "", DefaultModel: "x"}, // no key
+			"openrouter": {APIKey: "ork", DefaultModel: "x"},
+		},
+		Runners: config.RunnersConfig{OpenRouter: &f}, // explicitly disabled
+	}
+	runners := map[string]executor.Runner{}
+	logger := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelError}))
+	registerCloudRunners(runners, cfg, logger, t.TempDir())
+
+	if _, ok := runners["groq"]; ok {
+		t.Error("groq should not be registered without an api_key")
+	}
+	if _, ok := runners["openrouter"]; ok {
+		t.Error("openrouter should not be registered when explicitly disabled")
+	}
+	if _, ok := runners["openai"]; ok {
+		t.Error("openai should not be registered when absent from cfg.Providers")
+	}
+}
diff --git a/internal/cli/run.go b/internal/cli/run.go
index 34c3d42..08bfb68 100644
--- a/internal/cli/run.go
+++ b/internal/cli/run.go
@@ -6,12 +6,9 @@ import (
 	"os"
 	"os/signal"
 	"syscall"
-	"time"
 
 	"github.com/spf13/cobra"
 	"github.com/thepeterstone/claudomator/internal/executor"
-	"github.com/thepeterstone/claudomator/internal/provider/anthropic"
-	"github.com/thepeterstone/claudomator/internal/provider/google"
 	"github.com/thepeterstone/claudomator/internal/provider/openaicompat"
 	"github.com/thepeterstone/claudomator/internal/storage"
 	"github.com/thepeterstone/claudomator/internal/task"
@@ -119,39 +116,10 @@ func runTasks(file string, parallel int, dryRun bool) error {
 		}
 	}
 
-	if pc, ok := cfg.Providers["anthropic"]; ok && pc.APIKey != "" && cfg.Runners.AnthropicEnabled() {
-		timeout := time.Duration(0)
-		if pc.TimeoutSeconds > 0 {
-			timeout = time.Duration(pc.TimeoutSeconds) * time.Second
-		}
-		runners["anthropic"] = &executor.NativeRunner{
-			Provider:     anthropic.New(pc.APIKey, pc.Endpoint, timeout),
-			Logger:       logger,
-			LogDir:       cfg.LogDir,
-			DefaultModel: pc.DefaultModel,
-			// See internal/cli/serve.go: native cloud providers get
-			// sandbox.DockerSandbox rather than the weaker HostSandbox.
-			SandboxKind:  "docker",
-			SandboxImage: cfg.SandboxImage,
-		}
-	}
-
-	if pc, ok := cfg.Providers["google"]; ok && pc.APIKey != "" && cfg.Runners.GoogleEnabled() {
-		timeout := time.Duration(0)
-		if pc.TimeoutSeconds > 0 {
-			timeout = time.Duration(pc.TimeoutSeconds) * time.Second
-		}
-		runners["google"] = &executor.NativeRunner{
-			Provider:     google.New(pc.APIKey, pc.Endpoint, timeout),
-			Logger:       logger,
-			LogDir:       cfg.LogDir,
-			DefaultModel: pc.DefaultModel,
-			// See internal/cli/serve.go: native cloud providers get
-			// sandbox.DockerSandbox rather than the weaker HostSandbox.
-			SandboxKind:  "docker",
-			SandboxImage: cfg.SandboxImage,
-		}
-	}
+	// Native cloud providers (anthropic/google's own wire formats, plus
+	// groq/openrouter/openai via the OpenAI-compatible adapter) — see
+	// internal/cli/cloudrunners.go.
+	registerCloudRunners(runners, cfg, logger, cfg.LogDir)
 
 	pool := executor.NewPool(parallel, runners, store, logger)
 	pool.Classifier = &executor.Classifier{
diff --git a/internal/cli/serve.go b/internal/cli/serve.go
index 57c2cbd..656ee66 100644
--- a/internal/cli/serve.go
+++ b/internal/cli/serve.go
@@ -16,9 +16,8 @@ import (
 	"github.com/thepeterstone/claudomator/internal/config"
 	"github.com/thepeterstone/claudomator/internal/executor"
 	"github.com/thepeterstone/claudomator/internal/notify"
-	"github.com/thepeterstone/claudomator/internal/provider/anthropic"
-	"github.com/thepeterstone/claudomator/internal/provider/google"
 	"github.com/thepeterstone/claudomator/internal/provider/openaicompat"
+	"github.com/thepeterstone/claudomator/internal/scheduler"
 	"github.com/thepeterstone/claudomator/internal/storage"
 	"github.com/thepeterstone/claudomator/internal/version"
 )
@@ -156,44 +155,10 @@ func serve(addr, basePath string) error {
 		logger.Info("local runner registered", "endpoint", cfg.LocalModel.Endpoint, "model", cfg.LocalModel.Model)
 	}
 
-	if pc, ok := cfg.Providers["anthropic"]; ok && pc.APIKey != "" && cfg.Runners.AnthropicEnabled() {
-		timeout := time.Duration(0)
-		if pc.TimeoutSeconds > 0 {
-			timeout = time.Duration(pc.TimeoutSeconds) * time.Second
-		}
-		runners["anthropic"] = &executor.NativeRunner{
-			Provider:     anthropic.New(pc.APIKey, pc.Endpoint, timeout),
-			Logger:       logger,
-			LogDir:       cfg.LogDir,
-			DefaultModel: pc.DefaultModel,
-			// Native cloud providers get real container isolation
-			// (sandbox.DockerSandbox) rather than HostSandbox's host-side
-			// path-prefix confinement.
-			SandboxKind:  "docker",
-			SandboxImage: cfg.SandboxImage,
-		}
-		logger.Info("anthropic runner registered", "default_model", pc.DefaultModel, "sandbox", "docker")
-	}
-
-	if pc, ok := cfg.Providers["google"]; ok && pc.APIKey != "" && cfg.Runners.GoogleEnabled() {
-		timeout := time.Duration(0)
-		if pc.TimeoutSeconds > 0 {
-			timeout = time.Duration(pc.TimeoutSeconds) * time.Second
-		}
-		runners["google"] = &executor.NativeRunner{
-			Provider:     google.New(pc.APIKey, pc.Endpoint, timeout),
-			Logger:       logger,
-			LogDir:       cfg.LogDir,
-			DefaultModel: pc.DefaultModel,
-			// Native cloud providers get real container isolation
-			// (sandbox.DockerSandbox) rather than HostSandbox's host-side
-			// path-prefix confinement — same reasoning as the "anthropic"
-			// runner above.
-			SandboxKind:  "docker",
-			SandboxImage: cfg.SandboxImage,
-		}
-		logger.Info("google runner registered", "default_model", pc.DefaultModel, "sandbox", "docker")
-	}
+	// Native cloud providers (anthropic/google's own wire formats, plus
+	// groq/openrouter/openai via the OpenAI-compatible adapter) — see
+	// internal/cli/cloudrunners.go.
+	registerCloudRunners(runners, cfg, logger, cfg.LogDir)
 
 	pool := executor.NewPool(cfg.MaxConcurrent, runners, store, logger)
 	pool.Classifier = &executor.Classifier{
@@ -282,6 +247,24 @@ func serve(addr, basePath string) error {
 	srv.SetContext(ctx)
 	srv.StartHub()
 
+	// Retry-then-escalate scheduler for role-typed tasks (internal/role,
+	// internal/scheduler). Only `serve` runs this background loop — the
+	// one-shot `run` command has no long-lived process to host it in.
+	sch := &scheduler.Scheduler{
+		Store:  store,
+		Pool:   pool,
+		Logger: logger,
+	}
+	if accountant != nil {
+		// Only assign when non-nil: a nil *budget.Accountant boxed into the
+		// scheduler.BudgetGate interface would be a non-nil interface value
+		// whose methods panic on the nil receiver — same reason pool.Budget
+		// above is only ever assigned inside the `if accountant configured`
+		// branch, never unconditionally.
+		sch.Budget = accountant
+	}
+	go sch.Run(ctx, cfg.Scheduler.PollInterval())
+
 	httpSrv := &http.Server{
 		Addr:    addr,
 		Handler: srv.Handler(),