Survive provider outages: opt-in fallback model chain (--fallback-model, fallback_models) that switches models after retries on 429/5xx/network errors are exhausted #117

Summary

Let a run survive a provider outage instead of ending with Error: an opt-in, ordered list of fallback models (--fallback-model <provider>/<model>, repeatable; config fallback_models). When a model request still fails after tinyagent's existing retries on a transient error (HTTP 429, 5xx, or a network error), the run switches to the next fallback model and re-sends the same request, keeping the conversation, step/cost totals and tools. Every switch is reported on stderr, in --json / --log-db events, and in the trajectory. Without the option, behavior is unchanged.

Problem and opportunity

tinyagent is built for unattended runs (--json, --output-schema, --resume, stop hooks, --worktree). A long run is only as available as one model endpoint.

Verified facts from the repository (main at 1860a12):

• internal/model/provider.go post() retries network errors, 429 and 5xx up to maxAttempts = 6 with exponential backoff (honoring Retry-After), then returns giving up after 6 attempts: .... Other 4xx responses are returned right away.
• internal/agent/loop.go handles a failed Model.Query by compacting and retrying once on context overflow. Any other error ends the run with StatusError (exit 1). There is no second model to try.
• README, "Resume an unfinished task": an Error checkpoint can be resumed, but "an explicitly different --model is rejected", so a run that hits an outage can only be resumed on the model that just failed, after it recovers.
• Define named model providers in config (endpoint, API-key env var, headers) with per-model price and context-window metadata #93 (named providers) listed "Provider fallback chains or automatic model routing" as a non-goal of that issue. That kept the scope small. It was not a decision against fallback. Named providers make fallback more useful because a run can now fall back to a different endpoint (for example OpenRouter, a local Ollama, or a self-hosted gateway).

Opportunity (interpretation): the provider-neutral model.Message history, the Model interface, per-provider adapters, named providers, and per-model price/context metadata already exist. Moving a run to another model at a step boundary fits the current architecture without changing how tools or the loop work.

Research and evidence

Comparable implementations (primary sources, read 2026-10-25):

• Claude Code, --fallback-model: "Enable automatic fallback to the specified model(s) when the primary model is overloaded or not available ... Accepts a comma-separated list tried in order." Its model-config docs ("Fallback model chains") say authentication, billing, rate-limit, request-size and transport errors never trigger a switch, chains are capped at three models, a notice is shown on each switch, and the chain also covers compaction (skipping fallbacks with a smaller context window) and subagents. https://code.claude.com/docs/en/cli-reference, https://code.claude.com/docs/en/model-config
• Gemini CLI, "Model routing": "automatically switches to a fallback model in case of a model failure (for example, due to quota or server errors)". Interactive use prompts by default. https://geminicli.com/docs/cli/model-routing/ (page dated Apr 29, 2026). Gemini CLI PR #29800 tells permanent quota exhaustion apart from transient capacity errors when deciding to switch. fix(core): fall back across model tiers only when Auto quota is exhausted google-gemini/gemini-cli#29800
• OpenRouter, models parameter: server-side fallback across models "if the primary model's providers are down, rate-limited, or refuse to reply". Billing uses the model that actually answered, which the response's model field names. This only works within OpenRouter. https://openrouter.ai/docs/guides/routing/model-fallbacks

User pain points (public trackers; each one is a single report and does not prove broad demand):

• OpenCode #7602 (opened 2026-01-10), "Native Model Fallback / Failover Support": "long-running agent workflows ... fail on transient provider/model issues and forces users to rely on external routers or proxies". It asks to switch on rate limits, unavailability and 5xx, but "not retry on prompt or validation errors". [FEATURE]: Native Model Fallback / Failover Support anomalyco/opencode#7602
• OpenCode #25150 (opened 2026-04-30), a separate request for the same capability. [FEATURE]:Add model fallback/failover support anomalyco/opencode#25150
• OpenCode PR #26292 implemented a fallback chain with per-model cooldowns. Its rationale: "routing through LiteLLM degrades tool-call quality". It was closed without merging because it targeted an older codebase. feat(opencode): add LLM provider fallback chain anomalyco/opencode#26292
• A third-party OpenCode plugin exists only to add this behavior: https://github.com/Smart-Coders-HQ/opencode-model-fallback

Interpretation: at least three agent CLIs or their users treat model-level failover as needed for long or unattended runs. They agree on the trigger (transient/availability errors only, never auth or bad requests) and on making each switch visible. Where they differ is whether the switch lasts one turn (Claude Code) or until a cooldown ends (OpenCode PR). tinyagent's non-interactive design rules out Gemini CLI's prompt-before-switch default.

Users and use cases

• CI and scripted runs (--json, --output-schema, -o): a provider 529/503 burst or a regional outage should not fail a pipeline job that had already done 30 steps of work.
• Rate-limited accounts: a user whose primary provider returns 429 for longer than the retry window (five backoff waits of 1, 2, 4, 8 and 16 seconds plus up to 50% jitter, unless Retry-After says otherwise) can continue on a second provider or a local Ollama model.
• Named-provider users (Define named model providers in config (endpoint, API-key env var, headers) with per-model price and context-window metadata #93): fall back from a self-hosted gateway to a public API, or the reverse.
• Parallel --worktree runs that exhaust one provider's rate limit together.

Goals

• Opt-in ordered fallback chain from a flag or the user config, validated before the first request.
• Switch only after the active model's existing retries are exhausted on a transient error (429, 5xx, network). Never switch on auth, bad-request, other 4xx, context overflow, interruption or limits.
• Re-send the same request (same messages, tools and schema) to the fallback model, without losing the conversation or totals.
• Correct accounting: each response is priced with the model that served it, and --cost-limit, --step-limit and --time-limit keep applying to the whole run.
• Full visibility: stderr notice, --json / --log-db event, summary line, trajectory record.
• Resumable: a checkpoint written after a switch can be resumed, and the fallback chain can be given (or changed) on --resume.

Non-goals

• Cost- or quality-based routing ("cheap model first, escalate"), or content/refusal-based fallback.
• Load balancing or round-robin across models.
• Fallback for subagents (delegate); they keep subagents.model and today's error behavior. Could be added later.
• Changing the retry policy itself (attempt count, backoff) or making it configurable.
• Interactive prompts before switching.
• Reading the chain from repository files. Like providers, mcp_servers and hooks, it comes only from the user config, -c, or flags.
• Using OpenRouter's server-side models parameter. That is provider-specific and can be configured separately.

Proposed behavior

Configuration

• Config: fallback_models: [anthropic/claude-sonnet-4-5, ollama/qwen3-coder], a list of model specs using the same <provider>/<model> syntax and resolution as -m, including named providers.
• Flag: --fallback-model <spec>, repeatable. When given, it replaces the config list, the same way --sandbox-allow-domain replaces sandbox_allowed_domains. A way to disable the config list from the command line (for example --fallback-model= or --no-fallback) is part of the design. See open questions.
• Default: empty list, so behavior is exactly as today.

Validation (before any model request; exit 2 on failure)

• Each entry must parse as a model spec and name a known built-in or named provider. A missing API-key environment variable is reported naming the entry, the same way the primary model's setup errors are.
• An entry equal to the primary model, or a duplicate entry, is rejected.
• At most a small fixed number of entries (Claude Code caps at 3; see open questions).
• With a positive --cost-limit, the existing unpriced-model warning is printed for each unpriced fallback entry, naming it.

Switching

1. The active model's request fails after the existing retry loop with a retryable cause (http_429, http_5xx, network; the same causes provider.retry reports).
2. If the run context was cancelled or a limit is reached, the run ends exactly as today (no switch).
3. Otherwise tinyagent prints model fallback: <from> -> <to> (<reason>) at step N to stderr, emits a model.fallback event, and sends the same request to the next entry. That request goes through its own normal retry loop.
4. If the fallback also fails with a retryable cause, the next entry is tried. When the chain runs out, the run ends with Error and the error names every model tried with its last reason.
5. A non-retryable error from a fallback (for example HTTP 400 because the model rejects a parameter) ends the run with Error as today, naming the model. It does not move further down the chain.
6. After a successful switch, later steps keep using the model that answered. Whether and when the run returns to the primary is an open question; the behavior must be deterministic and documented.

Request translation

• The conversation is sent in the fallback provider's format by its existing adapter. Provider-specific reasoning state stored in Message.Extra (Anthropic thinking blocks, Codex/Responses encrypted reasoning) is only replayed to a provider family that can accept it. Foreign state is left out of the request but kept in the trajectory.
• reasoning_effort and max_output_tokens carry over only when the fallback uses the same provider as the primary. This mirrors the existing subagents.model rule in README "Subagents".
• If the fallback's known context window is smaller than the current history, the existing overflow handling (compaction, then Error) applies on the fallback model.

Accounting and limits

• Each response's cost uses the price of the model that produced it, including that model's cache pricing. The summary, trajectory and result event totals stay cumulative.
• The failed attempts on the previous model are not counted as steps.
• Compaction requests made after a switch use the active model.

Observability

• --json (and so --log-db): model.fallback event with step, from, to, reason (http_<status> or network, never a response body), and attempts on the failed model. run.started includes fallback_models only when non-empty.
• Summary line: ..., N model fallbacks (now <model>).
• Trajectory (-o): fallback_models (the configured chain) and model_switches (step, from, to, reason). The existing model field keeps the primary model. Each assistant message (or step record) identifies the model that produced it, so audits can tell responses apart.

Resume

• --resume keeps the checkpoint's primary model as today. The checkpoint records which model was active. The fallback chain comes from the current flags/config, so a run that ended in Error after an outage can be resumed with --fallback-model added. Whether a resumed run starts on the primary or on the last active model is an open question.

Compatibility

• Without fallback_models or --fallback-model, requests, events, trajectories and exit codes are byte-for-byte what they are today.
• Older tinyagent versions reject a config file containing the new fallback_models key (unknown keys are rejected). This must be documented, as was done for providers and subagents.

Acceptance criteria

• Given no fallback configured, when the primary returns 503 on every attempt, then the run ends with Error after the existing 6 attempts, with no model.fallback event and no new trajectory fields (existing tests unchanged).
• Given --fallback-model B, when primary A returns 503 (or 429, or a network error) on all attempts for step N and B succeeds, then step N's response comes from B, stderr shows one model fallback: A -> B (http_503) at step N line, --json emits one model.fallback event with step, from, to, reason, attempts, and the run continues to Submitted/Completed with exit 0.
• Given A returns 400, 401, 403 or 404, when the step fails, then no fallback happens and the run ends with Error as today.
• Given A returns a context-overflow error, then the existing compaction-and-retry path runs on A. Fallback is used only if a later failure has a retryable cause.
• Given Ctrl-C or a step/cost/time limit during A's retry backoff, then the run ends with Interrupted / the limit status and no fallback request is sent.
• Given chain [B, C] where B also fails retryably and C succeeds, then two model.fallback events are emitted in order (A->B, B->C) and step N is answered by C.
• Given every model in the chain fails retryably, then the run ends with Error, exit 1, and the error message names each model tried and its last reason.
• Given a fallback fails with a non-retryable error, then the run ends with Error naming that model and no further entries are tried.
• Given a fallback entry with an unknown provider, a malformed spec, a missing API-key env var, a duplicate, an entry equal to the primary, or more entries than the cap, then tinyagent exits 2 before any model request, with a message naming the entry. Key values are never printed.
• Given --cost-limit 1 and an unpriced fallback model, then a warning naming that model is printed before the first request.
• Given A and B with different prices, then each response's cost uses its own model's price, totals are cumulative, and --cost-limit stops the run on the combined total.
• Given A is Anthropic with thinking blocks in history and B is an OpenAI/Chat Completions model, then the request to B is accepted by B's adapter without Anthropic-specific fields (verified with a local stub server), and the trajectory still contains the original thinking blocks.
• Given A and B use different providers, then reasoning_effort and max_output_tokens are not sent to B. Given the same provider, they are.
• Given -o run.json and a switch, then the trajectory records fallback_models and model_switches, keeps model as the primary, and identifies the model behind each response. run.started includes fallback_models.
• Given a trajectory that ended in Error after A failed, when --resume run.json -o run2.json --fallback-model B is run, then the resume is accepted (not refused as a model change) and can continue on B.
• Given --log-db, then model.fallback events are recorded like other events, with no response bodies or credentials.
• Given fallback_models in a config file, then the README documents that older versions reject the key.
• --help golden output and README (flags table, Config section, --json event table, resume rules, troubleshooting) are updated. gofmt -l ., go build ./..., go vet ./..., go test -count=1 ./..., and GOOS=linux go build ./... && GOOS=linux go vet ./... pass. All tests use local stub servers, with no live credentials.

Technical context and constraints

Facts (paths on main at 1860a12):

• internal/model/provider.go: Model interface (Query(ctx, messages, tools) (Message, Cost, error)), post() retry loop, httpError, Retry / WithRetryObserver (backs the existing provider.retry event), price table. Message.Extra holds provider-specific state.
• internal/model/{openai,responses,anthropic,ollama,codex,custom}.go: per-provider adapters, including named providers.
• internal/agent/loop.go: the a.Model.Query call site and the context-overflow compaction retry. Non-overflow errors call a.finish(t, StatusError, ...).
• internal/agent/compact.go, internal/agent/subagent.go: other model users.
• internal/trajectory/trajectory.go: trajectory schema (model, messages, totals).
• cmd/tinyagent/main.go, cmd/tinyagent/providers.go: flag parsing, model/provider resolution, warnUnpriced, usage text.
• internal/config/config.go: strict YAML config (unknown keys rejected).

Constraints:

• The only way to tell retryable from non-retryable errors today is inside post() (httpError.status, network errors). After the 6 attempts it wraps the last error in a plain fmt.Errorf. A typed or errors.Is/errors.As-detectable retry-exhausted error will be needed. Streaming paths should be checked for the same classification.
• CONTRIBUTING.md asks for no new dependencies without strong justification. None are expected.
• Unknown-model pricing is $0 (README Notes), so unpriced fallbacks weaken --cost-limit.

Proposed design (not decided): wrap the primary and fallbacks in one Model that the agent loop sees, so loop.go changes as little as possible. Report switches through a context observer like WithRetryObserver.

Decisions for maintainers are listed under Open questions.

Alternatives considered

• External router/proxy (LiteLLM, OpenRouter models): works today with no tinyagent change, but adds infrastructure, is limited to one gateway, and per the OpenCode PR #26292 rationale can degrade tool calling. It also hides which model answered from tinyagent's pricing and trajectory.
• Resume on a different model after failure: allowing --resume --model B on Error checkpoints is smaller, but needs a manual or scripted second invocation and loses the run's flow. Allowing the fallback chain on resume (proposed) covers this case anyway.
• More retries / longer backoff: does not help with long outages or retired models, and makes runs slower.
• Per-step return to primary (Claude Code's per-turn model): each step during an outage would first spend the full retry backoff on the primary. Kept as an open question, not the default.

Cross-cutting considerations

• Security/privacy: a fallback can send the whole conversation (code, tool output) to a different provider. Users choose this explicitly, and the README must say so. Credentials come from env vars as today and are never logged. Events carry only status codes or network, never response bodies.
• Reliability: switching happens only at a step boundary after a failed request, so no tool call runs twice. Partially streamed output from a failed request is thrown away before the retry. Implementation should verify that a stream that fails midway does not leave partial text on stdout or in the history.
• Cost: prompt-cache benefits are lost on switch. Pricing must follow the serving model.
• Back-compat: default off. New config key rejected by older versions (documented). Trajectory additions are additive. Whether the trajectory schema_version needs to change is for maintainers.
• Observability: stderr, --json, --log-db, summary, trajectory, as above.
• Maintenance: every new provider adapter must work as a fallback target with foreign Extra state in the history. Tests should cover each adapter pair that is supported.

Test and verification expectations

• Unit tests in internal/model for error classification (429 / 5xx / network → retryable-exhausted; 4xx → not), using the existing retryBase override for fast tests.
• Agent-loop tests with stub models: single switch, chain walk, chain exhaustion, non-retryable stop, cancel/limit during backoff, cost accounting with different prices, compaction after a switch.
• Cross-adapter tests with local httptest servers for at least Anthropic→Chat Completions and Responses→Anthropic history translation.
• CLI tests: validation exits (2), flag/config precedence, --help golden regenerated from the binary, --json event shape, trajectory fields, resume with an added chain.
• No live provider calls. No results are claimed here.

Documentation and rollout considerations

• README: flags table, Config example, new "Model fallback" section (triggers, non-triggers, data-sharing note, accounting, resume), --json event table (model.fallback, run.started.fallback_models), Troubleshooting row, Changelog entry, and the "older versions reject fallback_models" note.
• Rollout: opt-in only, so no migration is needed. Rollback is removing the config key or flag.

Dependencies and related work

• Define named model providers in config (endpoint, API-key env var, headers) with per-model price and context-window metadata #93: named providers. Listed fallback chains as a non-goal of that issue. This issue builds on its provider resolution and per-model metadata.
• Detect output-token truncation: never treat cut-off responses as success or run their tool calls, and make the output-token budget configurable #83: output-token truncation (OutputTruncated, --max-output-tokens). A truncated response is not an error and must not trigger fallback.
• Resume unfinished tasks from saved trajectory checkpoints #42 / Durable mid-task resume: per-tool write-ahead checkpoints, compacted-run resume, and a persistent plan/todo tool #69: resume and durable checkpoints. Resume rules are extended here.
• Delegate read-only investigations to a subagent with its own context: opt-in delegate tool that returns only the child's report #107: subagents (subagents.model). Precedent for per-provider carry-over of reasoning_effort / max_output_tokens. Subagent fallback is a non-goal here.
• Add opt-in JSONL event stream output for scripting and CI integrations #60 / Add opt-in SQLite event log (runs, events, tool calls) mirroring --json output #70: --json events and --log-db, which must carry the new event.

Open questions

1. After a switch, should later steps stay on the fallback for the rest of the invocation (simplest), return to the primary after a cooldown (OpenCode PR #26292 style), or try the primary at every step (Claude Code per-turn style)?
2. On --resume of a checkpoint that had switched, should the run start on the primary or on the last active model?
3. Maximum chain length: follow Claude Code's cap of 3, or another value?
4. Should model-not-found (HTTP 404) count as "unavailable" and trigger fallback? Claude Code switches for retired models, but tinyagent currently treats 404 as a setup error with a hint: line.
5. Flag form for clearing the config chain from the command line (--fallback-model= vs --no-fallback).
6. Does adding fallback_models / model_switches to trajectories require a trajectory schema_version bump?