feat(routing): prompt cache optimization across model pools via compatibility.cacheFormat

Context & Background

In OpenCode upstream commit 69a71426ce (feat(core): configure Anthropic cache breakpoints with compatibility.cacheFormat (#54945)), upstream introduced compatibility.cacheFormat for Anthropic and AWS Bedrock models to control how prompt caching breakpoints are structured and placed.

In our fork (developing-today/opencode), model routes (experimental.model_routes) balance requests across pools of models according to tiers (cheap, standard, frontier). Prompt cache read hits (tokens_cache_read) provide up to 90% cost savings compared to base prompt tokens (tracked in our route_attempt table). However, currently cache configuration is not unified across routed pools, causing cache fragmentation or missed breakpoint opportunities when requests cycle through different models in a pool.

Feature Objective

Introduce cache format optimization into model route definitions:

Allow model routes to declare cache format strategies (compatibility.cacheFormat) to maximize prompt caching efficiency.
Track and report cache savings ($ saved via cache hits) in route_stats and route_usage.

Technical Specifications

1. Route Schema Support for Cache Optimization

In packages/schema/src/config/model-routes.ts and packages/core/src/model-route.ts:

Add optional cache_format setting to route target definitions:

"experimental": {
  "model_routes": {
    "frontier_pool": {
      "mode": "ordered",
      "tier": "frontier",
      "cache_format": "anthropic-breakpoints",
      "targets": [
        { "provider": "anthropic", "model": "claude-3-5-sonnet-20241022", "cache_format": "anthropic-breakpoints" },
        { "provider": "bedrock", "model": "anthropic.claude-3-5-sonnet", "cache_format": "bedrock-converse" }
      ]
    }
  }
}

2. Cache Format Application in Provider Adapter

When resolving the target model in packages/core/src/session/runner/model.ts:
- Ensure the resolved route target injects the appropriate compatibility.cacheFormat into the request options before invoking the provider transport.
- Verify that Anthropic cache breakpoints (cache_control: { type: "ephemeral" }) are preserved when switching between supported targets in the same pool.

3. Cache Savings Metrics in `route_usage` & `route_stats`

In packages/core/src/model-usage.ts and packages/core/src/tool/plugin/route-usage.ts:
- Compute cost_saved_usd based on difference between base input pricing and cached read pricing:
```
savings = tokens_cache_read * (price_input - price_cache_read)
```
- Display cache hit ratio and total USD saved in route_usage tool output.

Key Files & References

packages/core/src/config/plugin/model-routes.ts: Route schema and normalization
packages/core/src/session/runner/model.ts: Route resolution and model options injection
packages/core/src/model-usage.ts: Usage tracking, token pricing, and cost calculations
packages/core/src/tool/plugin/route-usage.ts: Usage reporting tool

Acceptance Criteria

Route targets can specify cache format preferences that are applied to outgoing LLM requests.
route_attempt records non-zero tokens_cache_read when prompt caching is active.
route_usage reports total USD savings achieved through cache hits.
Unit tests in packages/core/test/ verify cache format propagation and savings calculations.

Intake review

Still live (verified this run): compatibility.cacheFormat does not appear in the model-routes schema or the core routing code (packages/schema/src/config/model-routes.ts, packages/core/src/model-route*) — routes still cannot declare a cache strategy, as filed. (The field exists in the broader models.dev/compat surface — packages/core/src/model-resolver.ts references compatibility data — but not in route definitions.)

Related: opencode #6 (multi-credential failover — credential switches interact with cache locality), opencode #7/#8 (cost reporting family; cache savings would surface through the same route_stats).

Notes / risks: priority classed low: the savings are real (cache reads are already tracked in route_attempt) but nothing is broken — this is optimization, and its value depends on how much pool rotation actually fragments caches in practice. Measuring current cache-hit rates per route before building would size the win; the body's acceptance criteria already cover the reporting half.

Suggested approach: first report per-route cache-read ratios from existing route_attempt data; if fragmentation is material, add the schema field and breakpoint placement, then the savings rollup.

— fleet monitor intake (repo-fleet-monitor, 2026-10-26 20:19 CDT run)

Triage verdict: CONFIRMED (queue task 2040)

Verified on current main b0633cc697760085022fb5656ed92fac9ce49b9c (2026-10-28), not from the issue text alone:

git grep cache_format|cacheFormat across every route file this issue names — packages/schema/src/config/model-routes.ts, packages/core/src/model-route.ts, packages/core/src/config/plugin/model-routes.ts, packages/core/src/session/runner/model.ts, packages/core/src/model-usage.ts, packages/core/src/tool/plugin/route-usage.ts — returns zero hits. The route schema's TargetObject (packages/schema/src/config/model-routes.ts:18, targets at :221) has no cache field, so a route/target still cannot declare a cache strategy.
The target surface exists and is live: packages/schema/src/model-compatibility.ts:29 defines cacheFormat: CacheFormat.pipe(optional), and packages/ai cache-policy / bedrock-cache code already consumes it — this is purely a routing-layer gap.
tokens_cache_read is already recorded (packages/core/src/model-route-log.ts), so cache activity is observable once formats can be set. The USD-savings reporting (acceptance 3) is also absent (cost_saved|savings: zero hits in model-usage/route-usage) and is split out below.

Routing: follow-up small_fix enqueued as queue task 2255 — route/target cache_format declaration (existing CacheFormat values, invalid values rejected at config decode) + propagation into the resolved target's compatibility.cacheFormat, tests pinning route-level, target-level, and unset behavior. The route_usage USD-savings slice is explicitly excluded from task 2255 and still owed as a separate follow-up.

Remainder routed (issue-sweep-rotating, 2026-10-30): the part-3 slice now has its own tracking issue — #952 — filed with the evidence re-verified on main 01cf577, and a small_fix task enqueued under a fresh dedupe key (repo-health-opencode-pilot task 3836, packet packet-952-fix.json, queue/queued applied to #952). This resolves outbox OB-2026-10-30-001 exactly as its post-steps prescribed. Parts 1–2 remain landed; #9 itself stays open pending the overall feature, with its remainder now owned by #952's queue task.

🤖 Agent note · picked up

Agent: queue worker (repo-worker-opencode), OpenChamber scheduled lane
Session / ID: queue task 7031
Found via: atomic queue claim
Task: Verify feat(routing): prompt cache optimization across model pools via compatibility.cacheFormat #9 against current main; the 2026-11-03 coverage packet may be stale because prior work split across feat(core): declare cache_format on model routes and targets #749 and feat(routing): show cache savings in route_usage — the #9 part-3 remainder #952.
Expectation: confirm the route cache-format and route-usage savings criteria on current main, record any remaining blocker (or none), and close out with evidence.
Links: PR feat(core): declare cache_format on model routes and targets #749 (cache-format routing), PR feat(core): report cache savings and hit ratio in route_usage (#952) #956 / issue feat(routing): show cache savings in route_usage — the #9 part-3 remainder #952 (cache-savings reporting)

Triage verdict: COMPLETE on current main — no remaining blocker

Verified against origin/main 94939076d9969770f942e2c6c8ee0f289b3da2e3 on 2026-11-03. The 2026-11-03 coverage packet was stale: the full issue scope has landed across two merged PRs.

Route/target cache format and resolver propagation: PR feat(core): declare cache_format on model routes and targets #749, merge f1878bd03eb72f0244c74d5947805d10299d5e66. Current code has cache_format in packages/schema/src/config/model-routes.ts:71,279, route normalization in packages/core/src/config/plugin/model-routes.ts:616-802, and per-target compatibility.cacheFormat injection in packages/core/src/model-resolver.ts:559-570,887,979.
Cache-read/savings reporting: PR feat(core): report cache savings and hit ratio in route_usage (#952) #956, merge 4548c43aee1ff2e7baf79dcec8883b4c51084ace; tracking issue feat(routing): show cache savings in route_usage — the #9 part-3 remainder #952 is closed. Current code aggregates cache_savings_usd in packages/core/src/model-route-log.ts:718-762 and reports cache-read tokens, hit ratio, and saved USD in packages/core/src/tool/plugin/route-usage.ts:228-261.
Current-main tests: bun run test test/model-usage.test.ts test/model-route-log.test.ts test/model-resolver.test.ts test/model-routes-expand.test.ts — 188 pass, 0 fail.
CI note: merged PRs retain historical failing checks. The PR feat(core): report cache savings and hit ratio in route_usage (#952) #956 Windows unit log was rechecked this run: its sole failure is Snapshot > captures concurrently from independent processes at packages/core/test/snapshot.test.ts:189, outside that PR’s diff. PR feat(core): declare cache_format on model routes and targets #749’s completion note documents its unit/e2e failures as unrelated UI tests. No feature-specific blocker remains.

🤖 Agent note · done (already implemented)

Agent: queue worker (repo-worker-opencode), OpenChamber scheduled lane
Session / ID: queue task 7031
Finding: all acceptance criteria for feat(routing): prompt cache optimization across model pools via compatibility.cacheFormat #9 are on current main; the remaining savings slice landed via feat(core): report cache savings and hit ratio in route_usage (#952) #956/feat(routing): show cache savings in route_usage — the #9 part-3 remainder #952.
Blockers: none.
Tests: 188 pass / 0 fail on current main (command above).
Links: PR feat(core): declare cache_format on model routes and targets #749 · PR feat(core): report cache savings and hit ratio in route_usage (#952) #956 · issue feat(routing): show cache savings in route_usage — the #9 part-3 remainder #952

Program tending — roadmap drift flag (2026-11-03, via developing-today/code#1352)

This issue closed COMPLETED 2026-11-03T10:05:52Z, but the developing-today/opencode ROADMAP.md row "Prompt cache use across model pools" (Later, idea) still links only here — the row names finished work.

Recommendation: remove the row (single-link row; the work has landed). Flagged with meta/roadmap-drift from the tending pass rather than edited directly this run.

Navigation Menu

Repository navigation

feat(routing): prompt cache optimization across model pools via compatibility.cacheFormat #9

Context & Background

Feature Objective

Technical Specifications

1. Route Schema Support for Cache Optimization

2. Cache Format Application in Provider Adapter

3. Cache Savings Metrics in `route_usage` & `route_stats`

Key Files & References

Acceptance Criteria

Intake review

Triage verdict: CONFIRMED (queue task 2040)

33 remaining items

Triage verdict: COMPLETE on current main — no remaining blocker

Program tending — roadmap drift flag (2026-11-03, via developing-today/code#1352)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

feat(routing): prompt cache optimization across model pools via compatibility.cacheFormat #9

Description

Context & Background

Feature Objective

Technical Specifications

1. Route Schema Support for Cache Optimization

2. Cache Format Application in Provider Adapter

3. Cache Savings Metrics in route_usage & route_stats

Key Files & References

Acceptance Criteria

Activity

dezren39 commented on Oct 27, 2026

Intake review

dezren39 commented on Oct 28, 2026

Triage verdict: CONFIRMED (queue task 2040)

33 remaining items

dezren39 commented on Oct 30, 2026

dezren39 commented on Nov 3, 2026

dezren39 commented on Nov 3, 2026

Triage verdict: COMPLETE on current main — no remaining blocker

dezren39 commented on Nov 3, 2026

Program tending — roadmap drift flag (2026-11-03, via developing-today/code#1352)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

3. Cache Savings Metrics in `route_usage` & `route_stats`