Skip to content

Full-quota utilization: burn-rate projections, cascade plan, and 4–6h audibles for every plan #2173

Closed

Description

@dezren39

What this is

Drew, 2026-10-29: OpenCode work burns mostly DeepSeek, so other plans that do not offer DeepSeek (Gemini, ChatGPT, eventually Claude again, others) risk ending their periods under-used. Standing rule from this issue: if a plan is not on track to be used in full by its reset, queue more useful work. Leftover quota is waste; useful work done sooner is not a bad thing.

The requirement

At all times there is a theoretical full-utilization plan for every quota — for each provider's largest window (weekly or monthly): current %, reset instant, the burn rate that lands it at ~100% exactly at reset, and where it fits in the cascade.

  • Burn rates come from history: per-model/per-pool run rates over time from the data we already keep (Supabase agent.quota_records, aggregator quota series, agent.mirror_token_usage_daily/hourly joined to agent models). Rates are measured per pool, not assumed.
  • Cascade modeling: when pool A exhausts at its projected date, route pools fall through to the next provider — project that provider's exhaustion in turn, across ALL pools and across consecutive resets, not just the next one. A plan that hits 100% for this reset but strands the following period is not a plan.
  • Audibles are planned, with dates: when projections say a pool will finish under-used or over-paced, the plan names the change in advance — route-pool priority/weight changes on a specific date, work-class shifts, or more work queued — rather than discovering the miss at the reset.
  • Check cadence: every 4–6 hours. Each check recomputes projections from fresh quota reads + recent burn, compares against the standing plan, and either confirms on-track or escalates the approach dynamically (more work queued, weights adjusted per the plan).
  • Over-pace flag: when a pool projects to exhaust materially early, raise the flag to Drew: slow down, or keep progressing? Default bias per Drew: keep doing useful work — but the flag is asked, not assumed.

Build

  1. Projection model over the recorded series (per pool: % trajectory, per-model burn attribution, reset calendar incl. the reset after next).
  2. The standing plan document (per pool: target trajectory, cascade position, dated audibles) — regenerated each check, diffed against the previous plan.
  3. The 4–6h check itself: evolve the existing go-plan-expiry-watch (already 6h, already projects in its daily report) or a sibling job; integrate with pool-quota-steward rather than duplicating it. Any job cadence change is Drew's call per the standing rate rule — this issue carries his explicit 4–6h direction for THIS check.
  4. Under-pace output must name the work to queue (which backlog/queue, which job class fits the stranded provider's models); over-pace output is the flag to Drew.

Acceptance

  • A current written plan exists covering every pool's largest window, with cascade projections across at least two resets.
  • Two consecutive 4–6h checks ran, recomputed from fresh reads, and either confirmed the plan or executed/escalated an audible.
  • One real under-pace pool was caught by the check (not by Drew) with work queued against it, or a written proof that every pool was on track.

Activity

  1. developing-today commented on Oct 29, 2026

    @developing-today
    Contributor
  2. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    Plan v1 — built 2026-10-29 from the recorded series (full doc: ~/workspace/research_notes/quota-burn-plan-20261029/plan.md)

    Sources: laptop quota-gatherer series (599 rows through 2026-10-29T07:15Z) + quota-current.json (07:30Z), Supabase agent.quota_records, agent.mirror_token_usage_daily. Fleet demand still climbing: 5.07B input tokens on Oct 28.

    Pool state (largest window each)

    Over-pace:

    • Go Lab monthly 87.2%, burning +12.5 to +18.6 pts/day → exhausts ~Oct 30–31, ~7 days before its Nov 7 reset.
    • ClinePass monthly 91%, accelerating (+25 pts/day last 24h) → exhausts ~Oct 29–30, ~6–7 days early (deliberate finishing mode, but a real cascade cost).

    Under-pace:

    • Meta weekly 23.2% at +6.6/day vs the 20.8/day needed → lands ~48% at its Nov 2 reset, ~52 pts stranded.
    • OpenAI/Codex weekly 27% — base case (72h rate +5.9/day) lands ~62% at Nov 4, ~38 pts stranded; BUT the last ~6h ran at ~25 pts/day, which exhausts it ~Nov 1 instead. The one genuine slow-down-or-keep-progressing flag (raised to Drew in chat with this comment).

    Composite flag — the Nov 2–5 DeepSeek trough: Go monthly spent, ClinePass monthly spent until Nov 5, Go Personal expired → three DeepSeek doors closed at once. Cascade cover: Meta (resets Nov 2) + Codex (resets Nov 4) + Antigravity.

    Dated audibles

    • A1 — Oct 30 (or first check with Lab monthly ≥95%): rotate the laptop's active Go workspace Lab → Personal and pre-queue Flash-fit work, rescuing Personal's ~16 pts before the cancelled plan ends Nov 2 21:02 CT (currently burning ~0/day because Lab is the active workspace — the openchamber#736 gap in action).
    • A2 — Oct 31 → Nov 1: Meta burn-up — shift workhorse overflow + subsessions onto meta seats (weight/priority change pre-filed as a dated PR) to land Meta ~95–100% at its Nov 2 reset instead of ~48%.
    • A3 — Nov 2 (at the resets): trough posture — hold DeepSeek-only queue classes, point the workhorse default at fresh Meta, schedule Luna/Sol-fit backlog work into Nov 2–4 to pull Codex toward ~100% by Nov 4; re-point scout to reopened OpenRouter.
    • A4 — Nov 4–5: pre-spread the returning DeepSeek backlog across Nov 5–14 (~10 pts/day per pool) ahead of the ClinePass (Nov 5) and Go Lab (Nov 7) resets, so cycle 2 doesn't repeat the binge-and-starve shape.
    • A5 — standing: Command Code + Open Design micro-burn (~9 pts/day of small CLI-shaped jobs) — only after a fresh Command Code read exists; do not burn blind.

    Named data gaps (not guessed)

    • Antigravity weekly — no gatherer rows since Oct 27 06:00Z, yet sonnet/opus-plan route through it: trough capacity is a blind spot.
    • Go Personal burn rate — one console read only; the gatherer reads only the active workspace (openchamber#736).
    • Command Code — last good read Oct 27; gatherer auth now failing (~28 pts projected stranded, provisional).
    • Meta freshness — file-export series stale since Oct 29 04:00Z and erroring at 07:30Z.
    • Claude — free tier + provider disabled + reads erroring: no pool to project.

    The check

    Proposal (in plan.md §5): evolve the existing go-plan-expiry-watch (already 6h, already projecting) into this issue's 4–6h full-quota check — two-rate + post-reset-segment computation, cascade pass across two resets, plan-diff output, and an execute-vs-escalate split (operations like A1's rotation and queueing execute directly under standing authorization; route-config weight changes are pre-filed as dated PRs; only over-pace flags, stale-source warnings, and out-of-authority audibles reach Drew).

  3. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    INVESTIGATION RESULT (Drew's order, 2026-10-29): what consumed the Codex plan. Full findings: ~/workspace/research_notes/codex-usage-investigation-20261029/findings.md (fleet workspace). Sources: laptop OpenCode DB (route_attempt/session_v2), Codex logs_2.sqlite, quota-gatherer series, provider config. No routes changed.

    Verdict: the flagged burn did not come from OpenCode routes, Luna/Sol selection, or any local Codex surface. The overnight Luna burn billed OpenCode Go, not ChatGPT. The Codex weekly meter is being drained by an unidentified account-side consumer at a metronome ~1.1 pts/hour, 24/7.

    Attribution of the +6 pts (21% @ Oct 29 01:15Z → 27% @ 07:15Z):

    The overnight Luna (269 attempts / 19.3M input tokens in-window) was five fleet queue-worker sessions on opencode issues (#431, #193, #813, #784, #819; agent build, worktrees in ~/git/fleet) pinned directly to opencode-go/gpt-6-luna — no route_id, no pool decision. Per the config's own comments, the Go door of Luna burns Go quota ("Go quota burns exclusively through luna-plan's gpt-6-luna leaves"), and the Go Lab series confirms it (5h window 1.7% → 39.7% in the same hours). Pool-routed luna-plan/sol-plan attempts in the last 3 days: zero — all Luna/Sol use found was direct session pins. The Oct 26–27 ChatGPT-door traffic behind the plan's 72h average was an OpenChamber review swarm (Oct 26, 25.1M Luna tokens) plus two build sessions (Oct 27) — it stopped Oct 27 ~12:00Z.

    The meter's shape is the tell: +1 pt every 45–75 min, same rate before/after the reset, through fleet-dead hours (05:00–07:00Z), unperturbed by real door sessions. At ~25–27 pts/day it exhausts ~Nov 1 no matter what the fleet routes — the over-pace scenario is real, but Luna/Sol routing is not its driver. Naming the consumer requires the ChatGPT web UI usage view (breaks usage down by surface); that login was not attempted here. Candidates consistent with a 24/7 metronome: a ChatGPT-side scheduled task/agent feature, a server-side integration (e.g. Codex cloud), another signed-in surface, or Aster-side standing work.

  4. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    Codex-usage investigation, leg 2 (account side, 2026-10-29): BLOCKED at a Drew-only step, exactly located. Working on the laptop desktop as directed, the staged Helium profile copy turned out to be signed out of both ChatGPT and Google: chatgpt.com loads logged out, the Google account chooser for the OpenAI sign-in lists drewrypope@gmail.com as "Signed out", and selecting it lands on Google's password challenge ("Hi Drewry — Enter your password", with a passkey alternative under "Try another way"). No credential was entered or improvised and nothing on the account was viewed or changed; the laptop tab was left sitting on that password screen, mid-OAuth, so once Drew types the password there the leg resumes immediately — enumerate the ChatGPT Scheduled page (names + cadences; an hourly-ish task matches the metronome's ~1 pt per 45–75 min) and the Codex usage views for Oct 28–29. Until then the standing 24/7 consumer behind the Codex weekly climb remains unnamed: leg 1 already excluded every local system, so it is account-side, and its control panel is behind that sign-in. Detail appended to ~/workspace/research_notes/codex-usage-investigation-20261029/findings.md (Leg 2 section).

  5. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    CORRECTION (2026-10-29, Drew + the 2026-10-26 census investigation): the plan's 'session DB tops out Oct 6' data gap is WRONG, and the 'two databases' framing with it. There is ONE OpenCode database (~/.local/share/opencode/opencode.db); around Oct 6 OpenCode 2.0 migrated storage INSIDE it from the legacy session/message/part tables to session_v2 / session_message. The legacy tables froze at the cutover — any reader querying them sees a store that 'ends Oct 6'. The live tables run Jan 18 → today. Read pattern (verified 2026-10-26/28): session_message rows, tokens at $.tokens.*, provider/model at $.model.providerID / $.model.id, type='assistant'. Tonight's stale-DB claims (this plan's attribution step; the #2174 lane note about the :13420 store) came from legacy-table reads. The quota check's job body is being corrected to name the v2 tables explicitly; its attribution step is unblocked — no exporter dependency for READ access (the #2072 exporter remains a pull from these same v2 tables).

  6. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    Leg 2b update (Codex metronome; full detail: "Leg 2b" in ~/workspace/research_notes/codex-usage-investigation-20261029/findings.md): retried per Drew's direction on the laptop. Helium route is a dead end for a mechanical reason — there is no Bitwarden button to click: the staged profile copy contains no extensions at all, the second Helium is a bot profile with only uBlock, and Drew's real Helium profile also has only uBlock installed; the Bitwarden extension exists only in Zen. Zen route got further and changed the picture: Zen's Google session is fully live (myaccount.google.com renders signed in as drewrypope@gmail.com, Google AI Pro — no password or autofill needed), and Bitwarden is installed there, but ChatGPT's Cloudflare challenge never completes for the WebDriver-driven session — /auth/login rendered the login card once, "Continue with Google" clicks produced no navigation and no popup, and later loads render an empty challenge shell (the profile's cf_clearance for chatgpt.com is stale). So the consumer behind the +1 pt / 45–75 min Codex tick is still unnamed, and the remaining step is small and specific: one human click-through in Zen (already on the laptop desktop, left running) — chatgpt.com → Log in → Continue with Google, chooser/consent only since Google is already authenticated — after which the session persists in that profile and the read-only collection (Scheduled page enumeration; Codex usage per-surface attribution, web vs API vs Codex tool, Oct 28–29) can run unattended. Nothing was changed on the account; no credential was read, printed, stored, or typed by the agent.

  7. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    Luna pin on queue-worker sessions — source found and removed (2026-10-29)

    What the pin was: the OpenChamber dispatcher (the every-15-min Muse job that launches OpenCode sessions for queued issues) was creating worker sessions with an explicit per-session model — the direct leaf opencode-go/gpt-6-luna — instead of a fleet route id (opencode-route/*). A direct leaf bypasses the route pools entirely: no quota demotion, no rotation, no fall-through. That is how five routine queue tasks burned 269 attempts / 19.3M input tokens of Go plan quota overnight (Oct 28–29).

    The source, with evidence

    The dispatcher writes a launch payload per task at ~/workspace/goals/keep-the-four-repos-healthy/hidden_files/dispatch-packets/payload-<task>.json — fields {agent, directory, model, prompt, title} — and launches it via the OpenChamber API. The five sessions' payloads:

    Task / issue Payload model Session
    2696 / opencode#193 opencode-go/gpt-6-luna ses_e7ba353b4ffeawmaxTCx84MQSv
    2916 / opencode#784 opencode-go/gpt-6-luna ses_e5e7aa33eebffeJOG9oq1WAI15
    2997 / opencode#813 opencode-go/gpt-6-luna ses_e5e7a6d6875ffeDxWg412Z5vBH
    3029 / opencode#431 opencode-go/gpt-6-luna (payload-3029-go.json, after rotation) ses_e5e7a42277cffeSgj1zZRAbM04
    3045 / opencode#819 opencode-go/gpt-6-luna ses_e5e7a36b910ffeE1A1D1gcr1iN

    The dispatcher's own binding notes state the mapping verbatim, e.g. binding-2696: "Pool: Fast (direct model opencode-go/gpt-6-luna — OPEN; mechanical, well-specified work)". For #431 the dispatcher first tried Meta Spark direct (two sessions died on provider.auth 403 client_attestation_denied, filed as opencode#818) and then "rotated" — to the Luna leaf, not to a route. Across all dispatcher payloads on disk: 31× opencode-route/workhorse, 17× direct opencode-go/gpt-6-luna, 16× direct meta/muse-spark-1.3-contributor, 6× opencode-route/deepseek-flash-plan, 1× direct opencode-go/deepseek-v4.1-flash. In each session's event log the model is set by a model-switched event at sequence 1, at the creation instant, before any user message — i.e. chosen by the launcher, never by the session.

    Why it happened: the dispatcher runbook (openchamber-dispatcher-instruction.md §4) named Drew's tiers (Workhorse / Frontier / Fast) but never bound a tier to a route id, so the dispatcher agent resolved "Fast" to a direct leaf and the habit hardened (steward notes in dispatcher-log.md even endorsed feeding pools "direct model or explicit route").

    Candidates checked and ruled out

    1. Supabase agent.tasks — there is no model column (columns verified), and the five tasks' packets (repo-task/v1) contain no model field. The model never lived in the queue table or the packet; it was chosen at dispatch time.
    2. Lane runbooks (lane-shift-instruction.md, repo-queue-worker-instruction*.md) — no model directive anywhere; the Muse lane shifts claim tasks and reconcile/harvest, they do not launch these sessions.
    3. Scheduled tasks — repo copies of the job definitions: openchamber-dispatcher enabled: true, openchamber-babysitter enabled: true, opencode Lane A shift enabled: true (Lane B disabled). The dispatcher and babysitter are live — the dispatcher is the launcher; the babysitter only nudges already-bound sessions and creates none.
    4. Per-directory config — the opencode repo/worktree .opencode agent files pin nothing for build (only duplicate-pr → claude-haiku-4-5, triage → gpt-5.4-mini); the fleet config's agent.build has no model field and the top-level default is opencode-route/workhorse.
    5. OpenChamber runtime routing.json — its research category also names opencode-go/gpt-6-luna, but it does not explain these sessions (their route attempts record variant default, routing.json says max, and the sessions' model was set explicitly at creation by the dispatch payload). Left untouched — it is the app's classifier state, not the pin.

    The removal

    Edited the source document — hidden_files/openchamber-dispatcher-instruction.md (the file the dispatcher job's body tells every run to read in full):

    • §4 now binds tiers to route ids and prohibits direct leaves: "Session models are ALWAYS fleet route ids — opencode-route/<route> — never a direct provider/model leaf. Tier → route: Workhorse → opencode-route/workhorse; Fast → opencode-route/speed; Frontier → opencode-route/specialist. … If a route fails live, rotate to the NEXT ROUTE (Workhorse → Speed → Scout), never to a direct leaf." The edit names the removed pin and this order explicitly.
    • §5 now requires fixing any prepared payload whose model names a direct leaf to the route id before launch.
    • No cron body change was needed (the dispatcher job body contains no model text; it delegates to the instruction file). No repo config change was needed (nothing there pinned Luna). Luna's seats inside the route pools are untouched, per orders — dispatched sessions can still reach Luna, but only through a pool, where quota demotion and fall-through apply.

    Verification — stated limit

    The fix took effect at 2026-10-29 08:37:25Z. The dispatcher run overlapping the edit dispatched nothing (its log: "no launch was attempted (no slot)" — the queued opencode candidates were lane-owned/capped/triage-class), and no new payload or session appeared through 08:48Z. So there is no post-fix session to inspect yet. The first dispatcher cycle certain to read the edited instruction is the next 15-minute run; the verification artifact is the next payload-<task>.json model field / binding-<task>.md Pool line, which must show an opencode-route/* id. Static verification is complete: the payload model field was the only mechanism that ever carried this pin, and its only standing instruction source now prohibits it.

  8. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    TWO DREW CORRECTIONS (2026-10-29, main chat):

    1. ChatGPT scheduled tasks: Drew clarifies he never said the fleet's OpenCode/OpenChamber scheduled tasks are canceled — we had none until today. What he said is that all the ChatGPT scheduled tasks are canceled. The line in the Luna-pin report implying his belief about scheduled tasks was wrong is itself corrected: the Luna pin's source was the (live) dispatcher job + its instruction doc, full stop. For the metronome hunt this matters: if ChatGPT's Scheduled page confirms zero active tasks, the account-side consumer is something else; leg 2's retry will verify the tasks' actual state.

    2. Go fall-through shape: Drew confirms he does NOT want the ordered [Lab, Personal] composition. What we have (pools seating all doors, round-robin spread + quota demotion) is what he wanted. No ordered-fall-through PR will be built.

  9. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    Gemini burn PR built + probe-verified (2026-10-29, Drew's order): PR #2200 — stacked on #2184 (base = its branch; merge in stack order, then the same laptop switch activates both).

    The change (one file, .opencode/opencode.jsonc, +30/−5, all inside #2184's TEMPORARY workhorse-primary wrapper): third target antigravity-agy/gemini-4-argon#high — Gemini 4 Argon, the strongest Gemini tier the Antigravity doors serve — plus a weights map: DeepSeek 1 / Spark 1 / Argon 2 → Argon takes ~half of primary picks, the other two keep ~a quarter each. Leg-only revert (delete the argon target + weights map) and the full revert are both written in the route comment. Not merged.

    Probe evidence (ran before building): one live call via the standing laptop server (opencode run --server … --model antigravity-agy/gemini-4-argon#high) → route_attempt rows 12378 + 12382, session ses_e5e791c534effeSZH2EHWrRmjE, 2026-10-29 09:00:30Z: provider antigravity-agy, model gemini-4-argon, variant high, outcome success (11,831 + 2,202 input tokens). Pool attribution: the gatherer's Antigravity read shows exactly two weekly pools — every gemini-* model at the same 5% used, resetting 2026-10-31T13:11:00Z (one shared Gemini allowance) vs Claude/GPT at 57% resetting Nov 1 — a Gemini call has nowhere else to bill. (Gatherer re-check after the probe: latest Antigravity snapshot is still the 08:13Z tick at 5% — expected; a ~14K-token probe cannot move a rounded weekly %.)

    Burn math: workhorse logged 3,670 attempts in the trailing 48h (~1,835/day, same ledger), nearly all resolving to the primary slot → ~900 Argon picks/day → ~1,800 Argon calls before the Fri 13:11Z reset, at #high with fleet-sized inputs. The gatherer publishes percent only (no absolute denominator), so the share is flow-sized: 50% is the largest cut that leaves DeepSeek and Spark a quarter each (Spark keeps burning to its Nov 2 reset), and quota demotion sheds overflow back automatically if the Gemini 5h window walls. Watch item for the next quota-check runs: the weekly Gemini % should now visibly move; if it hasn't, raise the Argon weight — a one-line change in the same wrapper.

    Validation: JSONC parses clean; OpenChamber validateModelRoutes reports no new problems vs the unmodified #2184 tip (only the pre-existing background window-key complaint, byte-identical; routes 24 → 24).

  10. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    Leg 2c update (2026-10-29): At Drew's direction the Continue-with-Google click was retried in Zen with human-grade pointer patterns (curved paths, hover dwell, press-and-hold, click-and-drag, wandering) — the attempts never engaged, because chatgpt.com in automated tabs now renders an empty Cloudflare challenge shell with no login card, widget, or button in the DOM at all. Drew then completed the sign-in himself (~04:12 CDT) — but verification here finds no authenticated ChatGPT session in the Zen profile this fleet can drive (~/.zen/xqh5fwvr.default): all seven of its chatgpt.com tabs still sit on the "Just a moment..." interstitial and /api/auth/session returns challenge HTML, so his fixed login evidently lives in another browser/profile/device (he was away from the laptop). The collection therefore still cannot run from this seat: Scheduled-task state, Codex usage detail, and billing remain unread, and the metronome consumer remains unnamed (leg 1's eliminations stand — the consumer is account-side). One click in this Zen — Google is already signed in here, chooser-only — or naming the seat he used unblocks it immediately. Full record: leg 2c in research_notes/codex-usage-investigation-20261029/findings.md.

  11. added a commit that references this issue on Oct 29, 2026
  12. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    Leg 2d — account-side collection via Helium (Drew-corrected): the Codex metronome is NAMED

    Full section in research_notes/codex-usage-investigation-20261029/findings.md ("Leg 2d — Helium"). Headlines:

    Route: Zen closed on the laptop first (verified: zero zen processes). The collection ran in the already-open Helium via CDP, signed in as drewrypope@gmail.com, planType prolite — Drew's fix had landed in Helium; no wall at any point.

    ChatGPT Tasks: none exist. chatgpt.com/scheduled is the empty state ("Schedule a task") with six dashed suggestion-template cards only (Sentinel Instant reliability report, Weekly finances, Meal plan, Flights, Sleep coaching, Email monitor). There is no task in any state — active, paused, or canceled. Drew's "the ChatGPT scheduled tasks are canceled" checks out in effect: nothing on that surface can be burning anything. The scheduled-tasks hypothesis for the metronome is closed. (Disclosure: clicking the Sentinel template card opened a local chat prefilled with its prompt — that's what template cards do; nothing was sent, no task created.)

    The usage data lives in Usage → Analytics, not the Overview and not Billing. The Overview Drew saw indeed shows no history ("Balance history isn't available yet"): weekly 69% left / 31% used, resets in 5d 20h; credits 0; one full reset available expiring Nov 6; App limits lists one app — OpenCode at 100%. Billing shows plan "ChatGPT Pro 100", renews Nov 25 2026, balance $0, and carries no usage data at all.

    Analytics, by feature, Oct 23–29: Cloud code review 75.6% · Subagents 22.4% · Unknown 2% · Chats <0.1%. Weekly period Oct 26–28: 36.1% of limit used (cloud code review 24.9 pts, subagents 10.2 pts). Tool activity 7d: 159 plugin calls, 66 skill uses, 1,398 messages.

    Assessment: the ~+1 pt / 45–75 min metronome is Codex's cloud code review — automatic PR reviews fired server-side by GitHub activity — plus Codex subagents work. It runs 24/7 because the fleet opens/updates PRs around the clock, it's reset-invariant, and it's invisible locally because no local process makes those calls. Not routes, not pins, not scheduled tasks, not local sessions. At the current pace the weekly pool still exhausts ~Nov 1; turning it down would be a Codex cloud code-review settings change on the repos, not a routing change. Nothing was modified in this leg (read-only apart from the disclosed template click); Helium left open as found.

  13. added a commit that references this issue on Oct 29, 2026
  14. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    Codex review-value audit — were the reviews worth it? (2026-10-23 → 2026-10-29)

    Full audit: ~/workspace/research_notes/codex-review-value-20261029/findings.md (raw dataset alongside it). Method: every Codex bot review comment in the window was read and classified by hand; outcome signals computed over the full set; everything paginated with explicit high limits.

    Totals

    • 1,613 PRs active in the week across code/opencode/openchamber/nix. Codex reviewed 1,129 of them (70%) — 763 ended the week with a clean latest review; 268 PRs got findings.
    • 636 inline findings (P1 110 / P2 526; median 2 per finding-PR). Cost frame: cloud code review = 75.6% of the Codex pool Oct 23–29; pool exhausts ~Nov 1 at current pace, resets Nov 4.

    Quality: 97.5% real

    Class Count
    REAL-CATCH (genuine bug/logic/security/data-loss) 620 (97.5%)
    TRIVIAL (correct, low value — mostly convention/process demands wearing P1 badges) 16 (2.5%)
    WRONG (false positive) 0
    NOISE 0

    Best catches: code#1724 WAL-migration data loss (Codex reproduced it in SQLite; lane fixed it with exactly the prescribed lock); code#1740 secret guard blind to files inside untracked directories — autocommit then pushes the credential (reproduced; not acted on, merged); code#923 cold-store hash/compress race sealing backups that can never restore (not acted on, merged); code#856 reflected XSS with a working payload (not acted on, merged, verified still on main today); code#2200 agy added to the workhorse primary pool while its bridge drops tool schemas, so tool-needing jobs silently can't complete (PR still open).

    No false positives found in 636 comments — and not one rebuttal in any thread; every response to a Codex finding was a fix.

    Attention: the failure is consumption, not quality

    • Explicit engagement (reply / thread resolved / conversation note): 54/636 = 8.5%. Upper bound incl. a later commit touching the flagged file: 199/636 = 31.3%. Reactions: 0.
    • Bimodal: ~27% of finding-PRs were shepherded well (nix#996 iterated three review rounds; code#1820 was held unmerged over four P2s). The rest landed in a void. P1s get barely more attention than P2s (33% vs 29%).
    • Realized value ≈ 8.5–31% of potential. Roughly 70–90% of what the pool bought was never consumed.

    Counterfactual

    Fleet comparison (judgment, from a 20-PR overlap sample vs CodeRabbit)

    78 PRs were reviewed by both bots. CodeRabbit is genuinely good — it independently found ~half of Codex's catches in the sample (twice on the identical line) and ~15 real defects Codex missed — but it is shallower on the hardest PRs (opencode#468: 4 findings vs Codex's 10) and noisier (6 of 7 comments on openchamber#503 were one doc nit × six locales). The fleet's own /oc-review left zero comments on all 20 sampled PRs. Running 1,129 reviews/week as fleet tasks would burn scarcer quotas plus lane attention. Verdict on substitution: fleet review would be worse on coverage, no better on quality; CodeRabbit alone loses the deepest catches. They are complements.

    Bottom line

    The reviews were worth it — the ignoring wasn't. Keep Codex review on all four repos (recommendation is analysis only; no settings changed). The pool buys the highest-precision review signal the fleet has, aimed at exactly our failure classes (secrets, backup integrity, data loss). If the pool must stretch before Nov 4, the least-bad lever is the trigger (review PRs a lane is shepherding), not repo restriction — nix, the smallest consumer, has 100% precision and the best engagement. The free fix: make Codex-findings triage a standing lane step, as the Oct 26–29 lanes already proved works.

  15. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    Follow-through on the Codex review-value audit (2026-10-29) — the audit's confirmed-live, never-acted findings are now filed as fixes with fleet tasks queued (all priority/high, queue repo-health-code-pilot):

    Defect (Codex P1, verified on main 2026-10-29) Fix issue Task
    Reflected XSS, PM aggregator /item/<subject> (flagged on PR #856) #2267 task 3422
    Secret-guard bypass: keeper misses credentials in wholly untracked dirs, autocommit pushes them (PR #1740) #2268 task 3423
    Cold-store hash/compress race sealing unrestorable backups (PR #923) #2269 task 3424
    agy bridge drops tool schemas in the workhorse primary pool (PR #2200, merged 2026-10-29 — defect now on main) #2270 task 3425

    Each issue quotes the original Codex finding, links the audit (research_notes/codex-review-value-20261029/findings.md), and carries Codex's prescribed fix direction. Duplicate checks were run first: no existing issue covered any of the four (#1751 and #865 are sibling defects in different scripts/proxies, cross-referenced on the new issues).

    The systemic evidence — recurrence. The audit found 6 groups where the same defect was re-flagged on later PRs because the first flag was never consumed, i.e. the defect lived long enough to be written again. The two worst: (1) the Jules tracking issue closed before the sync outcome is known — flagged four times (openchamber#692, #702, #716, opencode#731); (2) the main-broken watcher filing breakages for fork PR runs and duplicating issues — flagged in two repos (openchamber#753, again opencode#828).

    Lane-step change (the no-quota-cost fix the audit recommended). The fleet lane instructions now carry a standing Codex findings gate: §5 of repo-queue-worker-instruction.md and the -nix / -openchamber / -opencode variants, plus a gate paragraph in lane-shift-instruction.md §1 — before merging, a lane reads the PR's Codex review findings (all rounds) and dispositions each one (fix it in the PR, or reply on the thread why not); an undispositioned finding blocks the merge. This is a method change only — no cadence, lane-count, or queue-behavior change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions