bug: Workspace turn fails with "reached its usage limit" instead of walking every subscription — SUB-003 retries once, then hands the retry to the user #3470

Summary

A Workspace user sends a message, the agent's current subscription is at its usage limit, and the turn comes back as Failed with "The agent has reached its usage limit and can't respond right now. Please try again later." — even though the install has other subscriptions that could have served it.

Subscription exhaustion must be fully transparent to the Workspace user: when a turn hits a usage limit, the platform should switch the agent to another subscription and re-run the same turn itself, walking through every viable subscription (and the platform API-key fallback) until one answers. The user should only ever see a failure when the whole pool has been tried and nothing can serve.

Context

Today the remediation is bounded to one switch + one retry per turn, and in several paths it hands the retry back to the human:

• services/task_execution_service.py — the post-failure SUB-003 hook is guarded by a one-shot subscription_switch_attempted flag: at most one handle_subscription_failure (or fallback_to_api_key) and one re-issue of the turn. If that second attempt also rate-limits — the alternative was itself near its wall — the turn lands FAILED although a third subscription may be healthy.
• services/subscription_auto_switch.py — select_best_alternative_subscription honours a 2h kind-blind skip-list (has_recent_subscription_failures), so "try the others" can decline alternatives that would serve now. The bug(subscriptions): a Workspace message on a rate-limited subscription fails instead of switching and completing — close the SUB-003 gaps on the turn path #2638 readmission override only partially relaxes this.
• services/chat_execution_service.py — the sync /chat path switches and then answers "Subscription auto-switched to '…'. Please retry." with retry_after: 15: the switch happened, but the user's message was not re-sent.
• client_portal/service.py — the AUTH/BILLING branch either says "it was moved onto '…'. Send that again and it should go through." (auth_switched, retryable) or, when nothing was switched, emits the _usage_limit_detail 502 shown above (auth, not retryable). Both make the person do the platform's job; the second gives up on a pool that was never fully walked.

SUB-003 was designed around schedules, where one user-visible failure per switch was acceptable. In the Workspace an external client is sitting in front of a chat box: "try again later" against a pool with headroom is a product failure, not a degraded mode.

Acceptance Criteria

• A Workspace turn that hits a usage-limit (rate-limit or auth/billing class) failure is re-run on another subscription by the platform, with no user action — the person sees a normal reply, not a Failed card.
• The retry loop walks the whole viable pool before giving up: each attempt moves to the next-best subscription (headroom-ranked, per auto-switch picks an alternative subscription without checking headroom — it can move agents onto a nearly-exhausted one #2409), excluding only subscriptions that failed during this same turn; the 2h skip-list must not block a candidate that is the last untried one. The platform API-key fallback (bug(subscriptions): a Workspace message on a rate-limited subscription fails instead of switching and completing — close the SUB-003 gaps on the turn path #2638) is the final rung.
• The loop is bounded by the turn's remaining wall-clock budget (bug: SUB-003 post-switch auto-retry is capped at 300s regardless of the agent's execution timeout — long turns die mid tool-use #2789 — never by a second ceiling) and by the number of candidates, so an exhausted pool terminates promptly with one honest failure instead of thrashing.
• Only when every candidate has refused does the Workspace show the usage-limit message, and that message states the earliest known reset time when the headroom sampler has one (bug(subscriptions): a Workspace message on a rate-limited subscription fails instead of switching and completing — close the SUB-003 gaps on the turn path #2638 AC#4 copy stays).
• The same behaviour applies on the sync /chat and streaming turn paths used by the Workspace — the "auto-switched … Please retry" / "Send that again" hand-backs are replaced by the platform's own retry; they remain only as the last-resort copy if a retry is impossible (e.g. budget exhausted).
• Each attempt is recorded on the execution (subscription_switch trace / attempts list) so the Executions view and operator notifications show which subscriptions were tried and why each was skipped, and the failed attempts' cost is rolled up (Async chat_with_agent: long execution silently fails with null response (reader-thread) #678 R2) rather than absorbed.
• The per-agent switch lock (bug: SUB-003 auto-switch has no per-agent lock — concurrent 429s race the restart #799) still serialises concurrent switches; a second concurrent turn on the same agent sees the already-switched subscription rather than racing its own restart.
• Regression tests: (a) two of three subscriptions rate-limited → the turn succeeds on the third with no user-visible failure; (b) all exhausted → exactly one failure, reset time named; (c) attempts never exceed the candidate count or the remaining budget; (d) existing test_subscription_auto_switch_pingpong.py thrash contract still holds for schedules.

Technical Notes

• Entry points: task_execution_service.py (post-failure switch block around the subscription_switch_attempted one-shot and the bug(subscriptions): a Workspace message on a rate-limited subscription fails instead of switching and completing — close the SUB-003 gaps on the turn path #2638 pre-dispatch check), chat_execution_service.py (switch-then-"please retry" responses), client_portal/service.py (AUTH/BILLING ladder and _usage_limit_detail), subscription_auto_switch.py (handle_subscription_failure, select_best_alternative_subscription, fallback_to_api_key).
• Read docs/memory/architecture/execution.md and docs/memory/architecture/workspace.md first — the terminal-status CAS / dispatch-activity close rules (bug: recovery paths write execution terminals without closing the paired activity — agent shows as running for up to 2h, then logs a fabricated 120-min failure #1804) apply to every extra attempt, and a retry must reuse the same execution_id so one-shot triggers and idempotency keys keep one row.
• Each retry restarts the container on the new subscription; the readiness probe is the retry (_SWITCH_RETRY_DELAY_S), so the loop should stay sequential per agent.
• Related: bug(subscriptions): a Workspace message on a rate-limited subscription fails instead of switching and completing — close the SUB-003 gaps on the turn path #2638 (one switch + API-key fallback + reset-time copy), bug: SUB-003 post-switch auto-retry is capped at 300s regardless of the agent's execution timeout — long turns die mid tool-use #2789 (retry budget), bug: Workspace blames the chosen model (and clears the preference) for any unclassified turn failure, incl. a post-switch retry timeout #2790 (model blamed for a post-switch retry timeout), bug: content-filter refusal ("Output blocked by content filtering policy") reported as OOM, classified as auth, triggers SUB-003 seat switch and a false "usage limit" portal message #2909 (content-filter refusal misclassified as a usage limit — the classifier must stay correct or this loop will churn subscriptions on a non-quota error), bug: chat_with_agent reports a usage-limit 429 as agent_busy although the turn already ran, so a retry repeats its side effects #3244 (usage-limit 429 reported as agent_busy on the MCP surface).

Part 2 — the API-key fallback (#2638) can strand an agent on a key that cannot serve

Observed on a live install: an agent's subscription hit its limit mid-run, the switcher found no viable alternative, and fallback_to_api_key fired — it cleared the subscription assignment, turned on use_platform_api_key, and restarted the container (killing the run in flight). Every turn since then goes to the platform key and fails in seconds with "Credit balance is too low". A limit that resets in a few hours became a permanent outage.

Confirmed in services/subscription_auto_switch.py::fallback_to_api_key:

• The only viability check is if not api_key — a configured key with no credit passes.
• The assignment is cleared, not remembered (documented as deliberate: no return path), so nothing moves the agent back when the window resets.
• Once on the API key, handle_subscription_failure's precondition (agent has a subscription) and the fallback's own if not old_sub_id: return None both no-op — the agent is stuck; a BILLING failure on the key has no remediation at all.
• _restart_agent is unconditional, so a fallback triggered by one turn kills whatever other work the agent was doing.

Acceptance Criteria (Part 2)

• The API-key fallback is a rung in the same retry loop, not a permanent reassignment: the turn is attempted on the key; if the key fails with an auth/billing-class error, that counts as one more exhausted candidate and the turn fails honestly with the pool's earliest reset time.
• A key that is known-refused (recent BILLING/AUTH failure recorded against it) is not selected — the same "recently refused by the provider" exclusion that auto-switch picks an alternative subscription without checking headroom — it can move agents onto a nearly-exhausted one #2409 applies to subscriptions.
• The agent's subscription assignment is preserved across the fallback (the fallback is per-turn routing, or the original assignment is recorded and automatically restored once the subscription's headroom sampler reports it can serve again). Operator reassignment remains a deliberate act, but the platform must not need one to recover from a transient limit.
• The "Switched to the platform API key" notification stays, and gains a second notification when the agent returns to its subscription.
• Regression tests: (a) subscription exhausted + key with no credit → one honest failure, assignment unchanged, agent not restarted into a dead state; (b) subscription exhausted + healthy key → turn served, assignment preserved, later turns return to the subscription once it can serve; (c) a key that failed BILLING is skipped on the next turn.