Repository navigation
Implement repo-fleet-monitor snapshot/SQL-first fetch (digest-judged runs, ≤3M input/run) #2154
Description
Implementation follow-on from the #2134 triage (queue task 3164, 2026-10-29). Parent: #2134 (design recorded there); fleet epic: #2055.
The 2h cadence half of #2134 is already live. This issue is the remaining half: rebuild repo-fleet-monitor's state fetch to be snapshot/SQL-first so a run costs ≤3M input (from 7.2M at check-in #5) with no duty dropped.
Design (full text on #2134):
- Deterministic precompute, run-start fresh. A script (extend the existing
hidden_files/monitor-fetch.py/monitor-query.pyprototypes in the goal workspace) runs at the start of every monitor run and emits ONE compact digest: live queue tasks across all four pilot queues (PostgREST, one query per queue, live statuses only) joined against a bounded GitHub pull (only issues carryingqueue/*orwork/*labels and PRs bound to live tasks). The digest pre-flags divergences mechanically: claimed-but-unlabelled, labelled-but-no-live-task, expired leases still claimed, expired parks, PR merged/closed but task open, issue closed but task open, dead-in-queue (max_attempts reached, still queued). - The agent judges only the digest. Full task/issue lists never enter the agent's context; a flagged divergence is investigated with a targeted single-item fetch. Intake enrichment stays bounded (newest-first sample, as today) but draws its candidate list from the digest.
- Freshness is binding: the digest is generated in-run; judging from a snapshot older than the run is forbidden (the stale 2026-10-28 prototype snapshot JSONs in hidden_files are the anti-pattern — delete or overwrite them in-run, never read them as evidence).
- No duty dropped — each protocol duty maps to digest fields: reconciliation + auto-repair (divergence flags), dead-in-queue alerts (attempts flag), intake enrichment (unenriched-issue list), in-progress tracker (claimed tasks + label join), protocol checks (work/* without a fresh agent note = flag).
Acceptance:
- Monitor instruction rewritten to the digest flow; precompute script committed in the goal workspace and invoked by the job body
- One full monitor run executed on the new flow with all five duties evidenced in its output
- Input per run ≤3M measured over the following three daily token check-ins, with divergence-repair counts not below the pre-rewrite run average
- Slim repo-fleet-monitor: snapshot/SQL-first state fetch + 2h cadence (target ≤3M/run) #2134 closed or updated with the measurement; epic: fleet expansion & placement program (2026-10-28 directive, LIST 1–5) #2055 linked
Activity
developing-today commentedon Oct 29, 2026
CODE-1049
dezren39 commentedon Oct 30, 2026
🤖 Agent note · done
Triage verdict — scope CONFIRMED, with one correction that changes the acceptance
Premise holds. The remaining half of #2134 is real and untouched: no branch matching
2154orfleet-monitorexists, andgh pr list --state all --search '2154 in:body,title'returns nothing, so no parallel attempt is in flight.Live state verified fresh (2026-10-30T04:45Z)
repo-fleet-monitorenabledevery: 2h, anchor2026-10-29T03:12:10, timeout 3600s — the #2134 cadence half is live, so this issue must not touch itgrep -c 'monitor-fetch|monitor-query'over the definition returns 0monitor-fetch.pyistasks?select=*across the four queues (no GitHub join, no flags);monitor-query.pyis the compact task projection usable as the digest's task halfmonitor-tasks-snapshot.json(975 KB, 2026-10-28 10:35),monitor-gh-snapshot.json(95 KB, 10:45)The correction: seven duty blocks, not five. The issue body maps four design points onto five duties. The live definition — which is the thing being rewritten — now carries seven blocks:
labelled-but-no-live-task,issue-closed-but-task-openclaimed-but-unlabelled,expired-lease-still-claimed,expired-park,dead-in-queuepr-merged-but-task-openmeta/enrichedabsent)work/*with no freshagent-note:v1expedite/next-upissues with live task id + priorityDuty 1B is the one that ended the duplicate-PR failure class on 2026-10-28, and Duty 5 is the newest Drew-ordered duty. A digest that only serves the five duties named in this issue's body would silently stop both. "No duty dropped" therefore means all seven, and that is what the implementation packet requires.
What actually costs the 7.2M/run (named so the rewrite has a target, not a guess): the in-run
snapshot.pypulls every live task row for 4 queues × 4 statuses plusgh issue list --limit 1000andgh pr list --limit 500for all four repos, writingissues-developing-today-code.json(4.86 MB),tasks.json(2.85 MB),prs-developing-today-code.json(332 KB), then printing 62 KB of analysis. The digest replaces that whole shape: bounded pull (queue/*+work/*labelled issues only, PRs bound to live tasks), everything else derived by join.One implementation gotcha worth recording: the job body addresses the goal workspace as
~/workspace/goals/...(the agent sandbox mount), while the versioned copy on the execution host is~/hatch-personal/workspace/goals/...(a git repo —crons/andGOAL.mdare tracked). The digest script has to be committed in the versioned copy and invoked by a path the job actually resolves at run time. That has to be decided and proved inside the one full run, not assumed.Routing
Implementation packet enqueued: task 3809,
repo-health-code-pilot, priority 90 (Batch 3 top priority per Drew's #2134 directive), deduperepo-health:v1:fix:developing-today/code:issue:2154, timebox 45 min, max 2 attempts. It carries this issue's acceptance criteria verbatim plus the seven-duty mapping above. The ≤3M target stays as written: measured over the three check-ins after the rewrite lands, with divergence-repair counts not below the pre-rewrite average.36 remaining items
dezren39 commentedon Nov 4, 2026
agent-note:v1
state: parked (task 3809)
/home/user/git/fleet/3809-2154-monitor-digest-correctness, branchqueue/3809-2154-monitor-digest-correctness; correctness changes are local/uncommitted. Four unit tests, syntax parsing, andgit diff --checkpass.work/assignedremains.dezren39 commentedon Nov 4, 2026
🤖 Agent note · blocked
The run-start digest (
monitor-digest.py) FAILED this run: exit 4,repo-queue: curl failed: curl: (28) Operation timed out after 60001 milliseconds with 0 bytes received— the PostgREST read path is hanging again (tracked as the queue outage on issue #4875). No digest was written; the on-disk digest remains the 2026-11-04T07:27Z one from the previous run and was NOT used as evidence (stale-digest rule).Fallback used this run, per the monitor body: per-duty commands only —
ghfor the GitHub side (works, ~5s/call) andsupabase-exec.py(Management API) for queue reads, which is itself degraded: aggregate count queries succeed after ~1–2 min, row-level queries mostly return HTTP 544 "Connection terminated due to connection timeout", and the few that succeed take ~2 min for a single row. Queue mutations viarepo-queue.py(enqueue/cancel/signal/claim) are therefore effectively unavailable this run; any queue-side repairs that could not be verified are reported as pending in the run summary, not silently dropped.dezren39 commentedon Nov 4, 2026
🤖 Agent note · blocked
Failure:
monitor-digest.pyexited 4 on two attempts (04:12 and 04:13 CST). Error:repo-queue: HTTP 503 from PostgREST: {"code":"PGRST002","message":"Could not query the database for the schema cache. Retrying."}— project keihfchslgjetkmnmzra.supabase.co.repo-queue.py healthfails identically, so the queue control plane is unreachable this run; all queue-side duties (reconciliation, dead-in-queue repair, PR binding, expedite raises) are blocked on the DB, and the monitor is falling back to GitHub-side per-duty checks only. No digest JSON was produced; no stale digest was used as evidence.dezren39 commentedon Nov 4, 2026
🤖 Agent note · blocked
monitor-digest.pyfails immediately withFileNotFoundError: /home/hatch/tools/repo-queue.py— the Python helper was removed in the 2026-11-04 binary-direct migration (code#5362), and the digest script still imports it via importlib.shift-certifier.py(full mode) fails the same way; only its--summary-onlypath (jsonl-only) works. This is the same missed-consumer class as code#5422 / code#5414.dezren39 commentedon Nov 4, 2026
Update on the digest breakage noted here:
monitor-digest.pyimported the removed~/tools/repo-queue.pymodule for its queue reads, so every monitor run since the binary-direct migration (#5362) lost its run-start digest and Duty 6 certifier input. Repaired 2026-11-04: the import is replaced with a self-contained read of muse-dev2 — the tasks query runs through the Management API SQL endpoint (the supabase-exec path) via curl with a slim projection of the fields the digest actually consumes, paged at 200 rows, because direct PostgREST reads from the sandbox truncate mid-chunk unpredictably. Verified with a full run today: digest written (165,948 bytes) with per-repo flag counts for all four repos. The next scheduled monitor run picks it up unchanged.dezren39 commentedon Nov 4, 2026
🤖 Agent note · parked
FileNotFoundError: ~/tools/repo-queue.py); the script repair posted above is confirmed structurally (no remaining import of the removed module; direct PostgREST reads) but is unproven until a scheduled monitor run executes it.dezren39 commentedon Nov 4, 2026
Monitor repair status — 2026-11-04 (fleet-binary exception close-out, code#5412)
What broke: the 2026-11-04 06:12 CST run's digest step failed with
FileNotFoundError: [Errno 2] No such file or directory: '/home/hatch/tools/repo-queue.py'— the digest builder (monitor-digest.py, deployed 2026-11-03, rollout record:~/workspace/goals/keep-the-four-repos-healthy/hidden_files/monitor-digest-rollout.md) still spawned the deleted Python helper for its queue-wide pull. The run proceeded on the filed-failure fallback path.Repair (done 2026-11-04, earlier today):
monitor-digest.pynow reads PostgREST directly — Management API SQL endpoint for the queue-wide pull (plain urllib reads truncate mid-body on this host), slim projection, credentials fetched from Bitwarden by name only. Norepo-queue.pydependency remains in the digest path. The related Duty-6 helpershift-certifier.pyhad the same break and was repaired the same day (code#5505): self-contained PostgREST reads via curl (key passed in curl's stdin config, never argv/disk); judging logic untouched; verified CERTIFIED output for all lanes.Verification — four consecutive judged runs on the repaired digest:
ok, each generating a fresh in-run digest (latest: 2026-11-04T20:12:31Z, 152,179 bytes) and evidencing all duties (per-issue checks across all four repos, seven worker bindings incl. shift-certifier CERTIFIED x6, fleet + supervisor + watchdog duties).What of this issue's spec remains:
repo-queuealias name (works via the~/tools/bin/repo-queuesymlink to the fleet binary); migrating the body's invocations tofleet queueis tracked in the migration ledger (~/workspace/goals/keep-the-four-repos-healthy/hidden_files/repo-queue-to-fleet-migration-2026-11-04.md) — deferred in this pass because the live body could not be safely re-based from the copy available to the migration agent; it needs a follow-up edit from a verified copy.agent-note:v1 — monitor repair verification + remaining spec, 2026-11-04.