Repository navigation
feat(routing): automatic multi-credential failover on 429 quota exhaustion via credential.activate #6
Description
Context & Background
In OpenCode upstream commit d78267503d (feat(plugin): expose credential.activate to server plugins (#52797)), upstream added a server-side API allowing plugins to programmatically activate and switch stored credentials for any provider.
In our fork (developing-today/opencode), we implemented intelligent model routing with tier awareness (cheap, standard, frontier), quota tracking, rate limit monitoring, and automated fallback. However, currently when a credential hits a 429 rate limit or its usage window is exhausted, routing can only failover to a different model or provider, or ask the user to switch accounts manually.
Feature Objective
Implement automatic credential failover within our routing engine: when a request fails with an HTTP 429 or quota limit, rather than immediately degrading to a lower-tier model or prompting the user, the router should check for secondary stored credentials for the same provider, activate the next available credential via credential.activate, and retry the request seamlessly within the same turn.
Technical Specifications
1. Credential Inventory & Status Detection
- Upstream provides
server.credential.list()andserver.credential.activate(id)or plugin hooks underpackages/plugin/src/andpackages/core/src/. - In our routing controller (
packages/core/src/model-route.ts/packages/core/src/model-resolver.ts):- Inspect configured credentials for the target provider (e.g.
anthropic,openai,google). - Track rate-limit backoff timers or quota windows per credential ID.
- Inspect configured credentials for the target provider (e.g.
2. Automatic Failover Sequence in Runner
- In
packages/core/src/session/runner/llm.ts/packages/core/src/session/runner/step.ts:- When an LLM request returns an error tagged with 429 (
error_status === 429or error message contains rate-limit/quota exhaustion): - Check if alternative credentials exist for
provider_id. - If an alternative valid credential is found:
- Call
credential.activate(nextCredentialId). - Record a failover note in
route_attempt(failed_over_toor metadata indicating credential switch). - Re-issue the request without interrupting the user turn.
- Call
- If no alternative credential exists, fall back to the existing route fallback chain (alternate provider or prompt user).
- When an LLM request returns an error tagged with 429 (
3. Quota Cooldown & Cooldown Window Expiration
- Maintain in-memory and persistent cooldown timestamps for credentials that hit rate limits (e.g. using
route_healthinpackages/core/src/model-route-log/sql.ts). - Avoid switching back to a cooled-down credential until its window has reset (or after standard exponential backoff).
Key Files & References
packages/plugin/src/effect/plugin.ts&packages/plugin/src/promise/plugin.ts:credential.activateinterfacepackages/core/src/model-route.ts: Model route resolution, target selection, and fallback listspackages/core/src/session/runner/llm.ts: Request dispatch, error handling, retry loopspackages/core/src/database/schema.gen.ts:route_attemptandroute_healthtables
Acceptance Criteria
- When multiple credentials exist for a provider and the active one receives a 429, the router activates an alternate credential without failing the turn.
- The retry attempt successfully completes and records
failed_over_to: "credential:<id>"inroute_attempt. - If all credentials for the provider are exhausted, the router proceeds to the next provider in the route definition or triggers standard error handling.
- Unit tests in
packages/core/test/verify credential failover behavior with mocked provider responses.
Activity
dezren39 commentedon Oct 27, 2026
Intake review
Still live (verified this run): the upstream capability exists in the fork —
packages/core/src/credential.tsimplements credential activation (log lines "credential activated" / "credential activate failed") — but no caller in the routing layer uses it: searchingpackages/core/srcfinds no credential-activate integration in model routing or the session runner's retry path. Failover on 429 still degrades model/provider or waits for a manual switch, exactly as filed.Related: opencode #9 (prompt-cache optimization across pools — the other pool-behaviour item; cache state interacts with credential switching), opencode #519/#530 (route governance correctness — separate defect class, same subsystem).
Notes / risks: design-first, effort/l. Two risks to settle in design before code: (1) retrying on a second credential within the same turn must not double-bill a request the provider actually processed; (2) quota state per credential must be tracked separately or the router will re-pick the exhausted credential. The body's acceptance criteria are already present and good.
Suggested approach: design pass on where the failover decision lives (router vs runner retry), then implement behind the existing quota/rate-limit signals with per-credential attempt records in
route_attempt.— fleet monitor intake (repo-fleet-monitor, 2026-10-26 20:19 CDT run)
dezren39 commentedon Oct 28, 2026
agent-note:v1 — task 2026 triage verdict: CONFIRMED, still live on current main (agent repo-queue-worker-shift-opencode, lane A)
Re-checked at HEAD 8f2dced (current main, 2026-10-28):
packages/core/src/credential.ts:56definesactivate: (id: ID) => Effect.Effect<void>, wired atpackages/core/src/integration.ts:805and exposed to plugins atpackages/core/src/plugin/host.ts:260.packages/core/srcfor credential-activate callers outside credential.ts/integration.ts/plugin host finds none in model routing or the session runner.packages/core/src/model-route.tsrecords quota errors per target (lines 255-295, 361-374) and demotes/cool-downs the model/provider target (quotaCooldownMs, demote-to-end) — failover degrades to another model/provider or waits; it never switches the credential on the same provider. That is precisely the gap this issue describes.Disposition: NOT enqueuing a small_fix follow-up, deliberately. This is effort/l and design-first (see the 2026-10-26 intake review): before code, a design pass must settle (1) where the failover decision lives (router vs runner retry), (2) the double-billing guard for retrying on a second credential within the same turn, and (3) per-credential quota tracking so the router does not re-pick the exhausted credential. A bounded small_fix packet would mis-scope all three. The issue stays open with its existing acceptance criteria as the implementation brief once the design choices are made; recorded in the fleet backlog as awaiting a design pass.
53 remaining items
fix(core): ignore quota observations older than the retained snapshot (…
dezren39 commentedon Oct 29, 2026
Task 3290 resumed by lane A: PR #854 advanced to head 9033b30.
bun run check:proto-readsnow passes on the merged tree.dezren39 commentedon Oct 29, 2026
Lane A resume check (task 3290), 2026-10-29: PR #854 head 9033b30 is MERGEABLE; Codex review completed with no unresolved findings. CI still pending (windows unit in progress, remaining checks queued amid repo-wide congestion), so the task is re-parked. One pre-merge item: a CodeRabbit minor thread on
credentialFallback(leaves the last failed candidate active when all candidate loads fail) was verified and answered on the PR — the merging lane must disposition it (fix + test, or filed follow-up) before merging.feat(core): credential failover on quota exhaustion via credential.ac…
dezren39 commentedon Oct 29, 2026
Done: PR #854 merged as
427f20399e(merge commit) on 2026-10-29T18:13:48Z; this issue closes with it.Merge gate: all checks green on head
9033b3074c(typecheck, affected packages, unit linux/windows, e2e, standards, compliance; windows unit needed one rerun — single flaky PTY test "detached PTY hosts interrupt commands without closing the shell", unrelated). Codex review on the head had no unresolved findings; every review thread is resolved. The last open thread (CodeRabbit,credentialFallbackleaves the last failed credential active when all candidates fail) was dispositioned as deferred to follow-up #870 item 6 (comment 6504470178) — not a blocker for this merge.