Repository navigation
Meta provider sessions fail with client_attestation_denied (403) despite healthy quota picture #818
Description
Problem
OpenChamber sessions seated on the Meta provider (meta/muse-spark-1.3-contributor) intermittently — then terminally — fail with a provider auth error, even though the Meta quota read reports the pool healthy (Muse Code High Usage plan, 5h 1% used at 2026-10-28 22:32 CDT).
Error (provider.auth, HTTP 403, code client_attestation_denied):
This API key is billed to a Muse Code subscription, and this request does not carry Muse Code's builtin tool declarations. Send it from Muse Code, or use a pay-as-you-go API key for other clients.
Evidence (2026-10-28 ~22:36–22:44 CDT, laptop OpenChamber 2.5.0)
- Dispatcher launch 1 for queue task 3029 (issue Standalone serve: meta provider turns hang with zero tokens (embedded build works) #431): session
ses_e5e7a44eacbffeudHD6csBC9v6— first assistant turn errored with the above, outcomefailed, zero repo work. - Dispatcher launch 2 (the one allowed same-pool retry): session
ses_e5e7a43b020ffeSWSrbb07yxHJ— engaged briefly (orientation tool calls, announced the task), then a subsequent turn errored with the sameclient_attestation_deniedand the session went idle/failed. - Meta-seated sessions dispatched earlier the same evening (tasks 2913/feat(routes): per-element (per-target) context limits #710, 2982/feat(codemode): versioned program-source libraries + std v1 pure helpers (bind, chunk, uniqBy, keyBy, pick, omit) (design #800, phase 1) #802) completed successfully on the same model/provider, so this is a regression or an intermittent attestation path, not a standing config error.
- Workaround used: rotated the dispatch to
opencode-go/gpt-6-luna, which engaged and is working task 3029.
Acceptance
- Root cause named: which request shape/tool-declaration difference triggers
client_attestation_deniedfor OpenCode's Meta provider calls. - Meta-seated OpenChamber sessions complete multi-turn tool-using work again, verified with a scripted multi-turn session, OR the Meta provider configuration is changed so the failing request shape is no longer sent.
- The failure mode is surfaced somewhere visible (session outcome / quota or provider health) instead of a healthy quota picture masking a dead route.
Acceptance criteria
- Root cause named: which request shape / tool-declaration difference triggers
client_attestation_deniedfor OpenCode's Meta provider calls. - Meta-seated OpenChamber sessions complete multi-turn tool-using work again, verified with a scripted multi-turn session, OR the Meta provider configuration is changed so the failing request shape is no longer sent.
- The failure mode is surfaced somewhere visible (session outcome / quota or provider health) instead of a healthy quota picture masking a dead route.
Activity
developing-today commentedon Oct 29, 2026
OPENCODE-179
dezren39 commentedon Oct 29, 2026
Intake review
🤖 Fleet monitor intake enrichment, 2026-10-29. Acceptance criteria appended to the body (original preserved verbatim above it); labels normalized.
dezren39 commentedon Oct 29, 2026
Root cause found; route verified healthy; closing.
What actually happened (from the server's own message store): the
client_attestation_deniedfailures were real but bounded to a ~22-minute window. The store contains exactly 5 denied outcomes, all between 2026-10-29T03:36:41Z and 03:58:06Z (Oct 28, 22:36–22:58 CDT) — the two dispatcher sessions named in this issue plus their retries. In each session the first step completed onmeta/muse-spark-1.3-contributorand the next step was denied, which matches a Meta-side attestation enforcement switching on mid-window (a progressive rollout is exactly "intermittent, then terminal"). No client-side change preceded it: provider config, request construction, and credentials are identical to the ones that ran meta sessions successfully earlier the same evening.Why it looked down for ~7.5 more hours: after 03:58:06Z the store contains zero denials — and zero completed meta messages until a verification session today at 11:33Z. The fleet never retried inference. The "still down" signal was the quota read (
/api/quota/meta→No quota data in response). The meta quota provider reads plan usage fromPOST https://api.meta.ai/muse-code/keywith the Muse CLI's account OAuth token — a different endpoint from inference. That read kept failing (HTTP 200 with nosubs_usageandis_subs_activenot true) long after inference recovered, and the dispatch gate treats a failed quota read as a closed pool, so the outage became self-sustaining: no sessions seated → inference never re-checked → the failing read stayed the only signal.Verification today (2026-10-29 ~11:22–11:40Z), all with the unchanged plan key (sha256 prefix
61a2ae5f; the pay-as-you-go key was never touched):https://api.meta.ai/v1:GET /models, chat completions, Responses, and Messages shapes — plain, streaming, with tools, and full two-turn tool round-trips (reasoning + function_call + function_call_output replay) — all HTTP 200.ses_e5e788ff595ffeXDNEy5WvkG2A) seated onmeta/muse-spark-1.3-contributorran a bash tool call and completed with the correct output; outcomesucceeded./api/quota/metareturnsok: true— 5h window 1% used, weekly 23%, plan "Muse Code High Usage", account admin@drewrypope.com.~/.config/muse/auth.jsonis untouched since 2026-10-26, its account token (sha256 prefix5b8afcae) authenticates to the key endpoint (HTTP 200,is_subs_active: true, and the endpoint returns the same minted inference key), andmuse execcompletes tool-using work. No re-login was needed or performed. Inference requests present the minted key, not the OAuth token, so a CLI re-login would not have changed the denied requests anyway.Not determinable from outside: exactly what Meta's gate keyed on inside the window. The error text points at tool declarations, but today the identical declarations pass in every shape, including the full production tool set through the real server. If the window recurs, the evidence to capture is a denied request's full body from within the window.
Acceptance mapping: (1) root cause named — Meta-side attestation window 03:36:41–03:58:06Z, plus the fleet-side quota-gate masking that extended it; (2) meta-seated sessions complete multi-turn tool-using work again — verified above on the production server; (3) failure-mode surfacing — the inverse masking also happened during the window (quota showed a healthy 1% while inference was being denied). A follow-up is filed in openchamber for the quota-read side: a failed or empty plan-usage read must surface its failure class and age instead of standing in for route state.
— operator investigation, 2026-10-29
dezren39 commentedon Oct 29, 2026
Follow-up for the quota-masking half: developing-today-forks/openchamber#787 — surface the meta quota read's failure class and last-success age so a failed usage read never stands in for route state. Closing this issue as completed on the verification above.