Repository navigation
Fleet queue unreachable: Supabase PostgREST 503 PGRST002 (schema cache) + 522, 2026-10-30 ~09:19 CDT #2767
Description
Problem
The fleet queue's Supabase PostgREST endpoint (muse-dev, used by ~/tools/repo-queue.py for all repo-health-* queues) is returning errors, so no queue reads or writes succeed:
- Repeated
HTTP 503from PostgREST:{"code":"PGRST002","message":"Could not query the database for the schema cache. Retrying."} - One
HTTP 522(Cloudflare origin error) in the same window.
Evidence (2026-10-30, dispatcher run 09:15–09:33 CDT)
- First failure ~09:19 CDT on
repo-queue.py healthand a direct tasks query (queued/claimed/parked acrossrepo-health-openchamber-pilot,repo-health-opencode-pilot,repo-health-nix-pilot). - Retries at ~09:24, ~09:26 (x3, incl. one 522), ~09:31 CDT all failed the same way — outage persisted at least ~12 minutes across the whole run.
- The laptop/OpenChamber side is healthy in the same window:
/api/version+/api/session-activity+ full/api/quota/*reads all succeeded over~/tools/laptop-ssh.sh. This is isolated to the Supabase/PostgREST side.
Impact
- The OpenChamber dispatcher could not inventory queued/claimed tasks, so it dispatched nothing and could not perform executor-saturated hand-backs this run (dispatcher-log 2026-10-30 09:15 run).
- Presumably every queue consumer (lanes, workers, fleet monitor, babysitter queue checks) is equally blocked while this lasts.
Acceptance
- PostgREST schema-cache/origin health restored:
repo-queue.py healthand a tasks query succeed again. - A note here recording when it recovered and, if known, the cause (schema-cache reload failure vs origin outage), so a recurrence is recognisable from this issue instead of rediscovered per-run.
Activity
dezren39 commentedon Oct 30, 2026
OpenChamber scheduled lane cross-reference at 15:44Z: outage now continuous ~65 minutes (first 14:39Z, still 503 PGRST002 at 15:44Z, one 522 at ~15:18Z). This lane failed its health gate and claimed nothing, so no stranded claims from it. Details and the 401-vs-503 edge check are on #2768: #2768 (comment)
dezren39 commentedon Oct 30, 2026
Intake review
🤖 Agent note · handed off
Still live? Yes — verified this run at ~16:12Z:
repo-queue.py healthreturned HTTP 503 PGRST002 ("Could not query the database for the schema cache") for ALL FOUR pilot queues (code, opencode, openchamber, nix). The issue's own## Acceptancesection stands as its acceptance criteria, so no duplicate section was appended.Notes / risks: #2768 covers the same outage window from a second run's evidence — the two should be reconciled (one kept as the canonical outage record) once recovery lands, not before. Until health returns, every queue verb (enqueue/claim/park/complete/cancel/bind) is blocked fleet-wide; GitHub labels are the only truthful state surface in the meantime, and claim leases will keep lapsing back to queued — that is expected outage behaviour, not worker evaporation.
Suggested approach: on recovery — post the recovery timestamp + cause here, land stranded terminal verbs from checkpoints, then run the monitor's divergence repair across all queues.
dezren39 commentedon Nov 3, 2026
🤖 Agent note · handed off
opencodefor task 3956 (triage of issue Fleet queue unreachable: Supabase PostgREST 503 PGRST002 (schema cache) + 522, 2026-10-30 ~09:19 CDT #2767: 2026-10-30 Supabase PostgREST 503 PGRST002 incident). Fresh check 2026-11-03T01:02Z: issue Fleet queue unreachable: Supabase PostgREST 503 PGRST002 (schema cache) + 522, 2026-10-30 ~09:19 CDT #2767 OPEN, no hard-stop labels. Handing off — the laptop opencode claimant launches the bound session; this lane does not work the task directly. Context for the session: the queue has been operating normally since (this very claim transacted against it), so the triage should weigh stale/resolved-with-evidence against any residual hardening follow-up.dezren39 commentedon Nov 3, 2026
🤖 Agent note · picked-up
dezren39 commentedon Nov 3, 2026
🤖 Agent note · done (triage)
Triage verdict: NOT CONFIRMED NOW — resolved with evidence
The reported outage is historical, not currently active. At the start of this run (2026-11-03 ~01:40Z),
python3 ~/hatch/tools/repo-queue.py healthreturned successfully for all queues; this task's claim and subsequent checkpoints also successfully read/wrote through the same queue backend, with no 503/522. Companion issue #2768 independently recorded successful queue operations on 2026-11-03 ~00:40–00:55Z and was closed as resolved.The exact recovery time is unknown; it occurred after the last reported 503 on 2026-10-30 ~16:12Z and no later than the successful operations on 2026-11-03 ~00:40Z. Cause is unknown; the evidence points to a transient upstream Supabase/PostgREST health incident, not a fleet-code defect. No follow-up fix packet is warranted. Closing this incident as resolved; reopen if the queue backend fails again.