Repository navigation
v1.91.108.0 fix(evals): census repair — the late-input and simplification fixtures stop inviting their own failures (#3153) - #3200
Merged
Conversation
|
😎 This pull request was merged. |
Contributor
E2E Evals: ✅ PASS10 executed, 0 reused rule/judge records | 0 manual accepted (unscored; no score-cache credit) | $13.27 rule/judge cost | reconcile exit: 0 Profile: pr / mode Rule and judge shards
Fail-closed reconciliationSliced lane: planner → duration-packed executors → fail-closed report. Behavior cases run a pre-registered 3-trial panel (PASS at 2/3 with no contract violation); a PASS 2/3 is shown with its failed trial, never as a clean pass. Reused results retain their original provenance and expiry. |
garrytan
added a commit
that referenced
this pull request
Oct 16, 2026
…test:health tells the truth (#3154) (#3212) * test: turn git auto-maintenance off in free fixtures, with a runner backstop git 2.29+ starts a detached `git maintenance run --auto` after every commit; it creates and deletes .git/objects/maintenance.lock after the commit returns. Diagnosed on 47d639a (git 2.55.0): - gstack-docs-candidate (ledger 37742176952/16): 4/150 replicated snapshots saw the .git listing change, every one `-objects/maintenance.lock`; 0/150 with the keys. repo() now sets maintenance.auto=false and gc.auto=0. - plan-design-sdk-fixture (ledger 37807608021/6): under commit load 18/150 runs failed the collector check; in every one Bun's rmSync(recursive, force) returned without throwing and left the scratch repo behind. 0/100 with the keys in the fixture's replacement git env, which never sees the runner's. - qa-functional-observer (ledger 37800857251/4): under load 6/200 observer setups threw ENOENT lstat .git/objects/maintenance.lock; 0/400 with the keys in the inherited env. The observer and its fixture (22 paid cases) are not edited; the runner backstop covers them. runFreeShard appends the two keys after any inherited GIT_CONFIG_* pairs of each shard child (a malformed inherited count is left for git to reject). A GIT_TRACE commit shows 3 maintenance lines with defaults and 0 in the runner child env; the new test pins that and the env merge cases. Failure messages now name the changed .git entries and surviving fixture directories; no assertion changed. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: synchronize the zombie publication-lock test on a framed PID line Ledger 38153371796/3 and 38161906980/3: line 69 `existsSync(ready)` failed. The test polled 300 x 10 ms for the Python parent's readiness file, then another 300 x 10 ms of /proc for state Z; under shard load the fixed count expired. existsSync also raced the write (an empty file read as pid 0). The parent now waits for its child with os.waitid(P_PID, WEXITED|WNOWAIT), which leaves it unreaped in state Z, and prints the PID with flush=True. The test reads exactly one newline-framed line from the parent's stdout under the test's own (unchanged) 10 s timeout, rejects a malformed or non-positive PID, and makes one /proc read that must show Z. Early exit, a malformed line or a timeout reports the parent's exit status and stderr tail; the owned parent is always killed in cleanup. Intentional assertion replacement: the readiness-file existence check and the polling loop are replaced by the framed-line requirement (a positive PID) and one /proc read; `expect(state).toBe('Z')` and every lock-recovery assertion are unchanged. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: stop the next-version smoke test from calling live GitHub Ledger 37804770972/9: "CLI runs against real repo and emits parseable JSON" failed with `JSON Parse error: Unexpected EOF` after 30032 ms. The smoke test ran the real CLI against live `gh`; bin/gstack-next-version's per-PR `gh api` calls run one after another (spawnSync inside its async workers), so 27 open PRs passed the 30 s spawn timeout and stdout was empty. Measured with a delayed fake gh: 8 PRs x 0.5 s = 4108 ms; 27 PRs x 1.2 s = 32637 ms. The CLI defect is filed as a P1 follow-up, not fixed here. The test keeps the real repository and the real CLI and puts a named fake gh first on PATH. It logs every argv, answers exactly the calls the CLI makes (auth status only when origin is not github.com, pr view exit 1, canned pr list, repo view and contents with one same-owner PR claiming 1.6.4.0) and exits 2 with a FAKE_GH_UNHANDLED marker for anything else; glab exits 1. The test now checks the exit code with stderr before JSON.parse, the exact call set, no marker, no glab call, host github, offline false, no fallback, no warnings, the canned claim, and version 1.6.5.0. Every existing assertion is kept; the spawn timeout is unchanged. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(test-health): report the Windows wall over PR runs and name the seed's real prerequisite free-wall read windows-free-tests.yml with `branch: 'main'`, but that workflow runs only on pull requests and workflow_dispatch, so the "main" runs it averaged were dispatch duration recordings. It now uses listWorkflowRuns' existing `event: 'pull_request'` filter and prints the population inline: `windows-free-tests (successful PR runs) 7.1 min (23 runs)`, or `no successful PR runs in window`. The free-tests half is byte-identical. The CEO-10 fix text stops offering a "CI recording" (none exists for the Linux seed) and names UBICLOUD_API_KEY; test:ubicloud --record-durations copies the seed back into the checkout, and contributors without the key leave the refresh to a maintainer. docs/TESTING_INTERNALS.md drops "or a CI recording" in the same change. Intentional pin change: test/test-health-report.test.ts free-wall value `windows-free-tests 0.0 min (1 runs)` -> `windows-free-tests (successful PR runs) 0.0 min (1 runs)`; the fixture now carries a PR run and an excluded dispatch run, plus a new case for the empty and mixed populations. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(timeline-read): reject missing and unparseable --since/--branch values `--since` and `--branch` read an unbound $2 under `set -u` and crashed with exit 1 when given no value, and an unparseable --since ("yesterday", "2 weeks") silently returned unfiltered history because the Bun regex was unanchored and fell through. Unknown arguments were silently ignored. --since is now parsed once in bash (no `date -d`, no bash-4 features) into a cutoff in epoch ms, and the Bun regex is deleted. Accepted forms: "<N> minute|hour|day|week|month[s] ago" (case-insensitive, as before), "<N>h|d|w" (the form gstack-global-discover documents) and ISO dates YYYY-MM-DD or YYYY-MM-DDTHH:MM:SSZ (UTC). In #1723's shape, before any read: --since must be a relative time like "7 days ago" (got no value); accepted: ... --since must be a relative time like "7 days ago" (got 'yesterday'); accepted: ... --branch must be a branch name (got flag '--limit') with "got an empty value" for ''. Unknown arguments exit 2 with the usage line; --help/-h prints it and exits 0. Tests extend the --limit table in test/timeline.test.ts and run every case on empty and populated history, plus each accepted --since form filtering old events; the quoted-branch and no-filter tests are unchanged. Behavior change: unrecognized arguments (previously ignored) now exit 2. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: record the due audit follow-ups with verdicts, and file wave-2 follow-ups docs/test-audit-2026-10.md gains a dated 2026-10-15 section from a no-spend `bun run test:health --since-days 7`, with the 2026-10-12 figures from #3154 kept as history and a verdict per target: full-fallback 45.9% (open; the audit's next step is touchfile mappings for bin/, lib/, hosts/, make-pdf/), spend per merged PR $73.52 (verdict unavailable: the pre-change baseline, recomputed with collectHealth at now=2026-10-04 behind an upper-bound client, holds no trial-outcomes artifacts because the recorder shipped with the audit), cancelled slice-minutes 1,545/week, push-to-verdict p50 11.4 min (missed against 3-5 min), flaky-pass 6.1% (missed; this wave repairs it). Census reliability is not due until 2026-10-26; the interim figure is recorded and the repair stays with #3153/#3200. TODOS.md: one appended dated section (P1 gstack-next-version serial fetch with the 4160 ms measurement, P1 standing seed recorder, P1 full-fallback/spend, P1 dated flake follow-ups, P2 createQAFunctionalFixture maintenance keys, P2 diagnostic-mode HOME/state-root artifacts). The timeline-read bullet was deleted with its fix. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore(release): tentative stamp v1.91.105.0 with its CHANGELOG entry Stamp-at-merge policy (.gstack/ship-policy.json): the branch carries a tentative PATCH stamp and its CHANGELOG section directly under `# Changelog`; the queue restamps at merge time. Base 47d639a is 1.91.101.0, but the live slot scan shows 1.91.102.0 (#3190), 1.91.103.0 (#3200) and 1.91.104.0 (#3209) claimed, so this branch takes 1.91.105.0. Written with `gstack-restamp --version 1.91.105.0 --base main --expect-base 47d639a --allow-repo-commands` (VERSION, package.json's npm translation, the agents digest via the policy's release tool, and the CHANGELOG header). * test(seed): refresh the free-suite duration seed from a Ubicloud recording (#3154) Recorded with `bun run test:ubicloud --record-durations` (standard-16, eu-central-h1) on this branch, which equals origin/main 47d639a plus this wave; the recorder exited 0 and wrote 1219 durations (1055 before, 164 new, none removed; serial 2439 s -> 3202 s). `test:health` CEO-10 now reports 0 unseeded files. Two earlier recordings did not count: one failed on an unrelated fixed-sleep race in test/plan-seed-submission.test.ts (filed P2, not edited here), the other failed free-seed-ratchet because it ran against the first attempt's pulled seed. No file was excluded. free-seed-ratchet: three files are over the 60 s ceiling in this recording and are allowlisted with a split todo (setup-install-registry 82 s, host-renders-prune 64 s, setup-codex-scope 60.4 s); the split follow-up and the plan-seed-submission race are filed in the appended TODOS section. The ratchet test now names scripts/free-test-seed-allowlist.json literally (a pin on SEED_ALLOWLIST_FILE): pregate's lanes check maps data files to tests through string literals, and the allowlist had no lane (PREGATE_NO_LANE). Additive assertion; nothing weakened. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: give the git-maintenance backstop test a Windows-safe empty global config windows-free-shard (2) on #3212 (run 38832498861) failed two cases in test/test-free-shards-git-maintenance.test.ts, both serially reproduced by the retry pass: Git for Windows rejects GIT_CONFIG_GLOBAL=os.devNull (`fatal: unable to access '\\.\nul': Invalid argument`), so the malformed-count case saw that error instead of git's GIT_CONFIG_COUNT rejection, and the traced commit's `git init` exited nonzero. Point GIT_CONFIG_GLOBAL at a file that does not exist in the scratch HOME, as test/ceo-plan-mode-fixture.test.ts does; git reads a missing global config as empty on every platform. No assertion changed. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…be read-only discovery (#3153) Census 38212265208 (main @ e23f7b8, 2026-10-12) failed ship-exploratory-late-input: "probe preceded resource read: exploratory / system-functional / system-functional". The native events show capture 001 issued in the same provider response as the Reads of ship/sections/plan-completion.md and ship/sections/review-army.md, before the Step 9 section (which holds the "never batch a probe with a Read" STOP) returned and before any QA section was read. The Oct 7 capture (mtb1007a) is the same class. Diagnosis: fixture defect, not product prose. /ship's prose never names the QA capture helper (only qa/sections/exploratory.md does); the fixture prompt supplies it up front, and the fixture's appended scheduling contract (7697fb2, 2026-10-06) told the session to batch "Reads and read-only discovery" while its never-group list omitted Read-result dependencies. The case passed on Oct 3 and Oct 5, before that contract, and failed on Oct 7 and Oct 12 after it. Repair: the contract defines read-only discovery (captures, guard starts/runs and diagnostic-client runs are probes, never discovery), which removes the fixture's own invitation to batch them. It deliberately does not restate the product's Step 9 STOP: the case exists to detect that STOP being lost. The detector (callerProbeReadErrors) is unchanged. Test: the Oct 12 native tool order is replayed inline (no new fixture file, so the paid-runner global inputs stay untouched) with a reordered negative control, and the actual runner callback's contract is checked. Fails on 47d639a at the discovery-definition assertion; passes after. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… stages only its section (#3153) Census 38212265208's pass-rate gate flagged review-army-simplification at 159 s of its 180 s session budget (88.1%, above the 85% cap; main-only panel 142/91/146 s). The three Oct 12 transcripts show where the time went: - the prompt claimed "the diff is over 100 lines", but the fixture diff is 68; every trial reconciled that (scope tool, specialist stats, force-include reasoning, coverage-gap prose in the report and final message); - each session read the whole extracted review-SKILL.md and the 18.7 KB checklist, and grepped for a "## Step 4.5" heading the carved fixture no longer has; - the report was unbounded (30-45 s of output composition); trial 3 also dispatched a subagent and re-verified its findings. Repair (fixture work only, budget unchanged at 180 s / 20 turns): the case uses the same stageReviewArmySession/reviewArmyScope staging as perf-n-plus-one and consensus (recorded scope, stats, learnings and diff; only review-army.md and simplification.md staged), selects the specialist with the real /review --simplification force flag, applies the checklist in-session as the original prompt asked, and bounds the report to 30 lines with a brief final acknowledgement. The planted-structure and disavowed-frame assertions are unchanged. Test: review-army-budget.test.ts drives the actual registered callback with a mocked runner (pass control plus two negative controls); it fails on 47d639a ("Expected to contain: DIFF_LINES: 68") and passes after. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…ation headroom, ship-exploratory-late-input) (#3153) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
garrytan
force-pushed
the
capy/fixwave-6
branch
from
October 16, 2026 20:50
df07a4e to
6099fdc
Compare
github-actions
Bot
changed the title
… census repair (#3153) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
garrytan
force-pushed
the
capy/fixwave-6
branch
from
October 16, 2026 20:51
6099fdc to
6bd61b7
Compare
github-actions
Bot
changed the title
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md # VERSION # agents-digest/gstack-AGENTS.md # package.json
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md # VERSION # agents-digest/gstack-AGENTS.md # package.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix wave 2026-10-15, PR 6 (item 18). The Oct 12 weekly census (run 38212265208, main @ e23f7b8) reported two problems: a red gate case,
ship-exploratory-late-input, and a headroom alarm onreview-army-simplification. The run's own transcripts trace both to their test fixtures, not to /ship or /review, so this PR changes no skill prose, budget, threshold or detector, and nothing undership/. It does not conflict with #3190.Refs #3153. This is not marked "Fixes" because the paid reruns below, or the next weekly census, decide whether the repairs work.
Issues
Refs #3153
Behavior changes
review-army-simplificationnow uses the same recorded-observation staging asreview-army-perf-n-plus-oneandreview-army-consensus. It stages onlyreview-army.mdandsimplification.md, selects the specialist with the real/review --simplificationforce flag, applies the checklist in-session, and caps the report at 30 lines. Its 180 s / 20-turn budget and its assertions are unchanged.Diagnosis
ship-exploratory-late-input(gate, rule) failed with "probe preceded resource read: exploratory / system-functional / system-functional".ship/sections/plan-completion.mdandship/sections/review-army.md. That was before the Step 9 section (which carries the STOP) came back, and before any QA section was read. The Oct 7 capturemtb1007ais the same class: there the probe was batched with the exploratory Read.qa/sections/exploratory.mddoes. The fixture prompt hands that command over up front, and its appended scheduling contract (added in 7697fb2, 2026-10-06) told the session to batch "Reads and read-only discovery".review-army-simplification(periodic, behavior) ran at 159 s of its 180 s budget pooled (88.1%, over the 85% cap). The Oct 12 panel on main ran 142, 91 and 146 s.The phase timings are in
docs/evals/headroom-diagnosis-2026-10.md.Tests (each failed on 47d639a first)
bun test test/qa-exploratory-callers.test.ts -t 'census 38212265208': replays census 38212265208's native tool order throughcallerProbeReadErrors, with a reordered negative control, and checks the actual runner callback's discovery definitionReceived value must be a string: undefinedtest/review-army-budget.test.ts: the registered callback with a mocked runner, one pass control and two negative controls; checks the premise against the staged 68-line diff, which files are staged, and the report boundExpected to contain: "DIFF_LINES: 68"No existing assertion was weakened and no pin changed. The simplification prompt change starts a new pass-rate series for that case.
Validation
PREGATE: requires-remote (2 uncleared) tree=6653e8aaa92c head=77c78b448fe4 checks=6. The local lane passed (156 files). The remote obligations are 1 macOS test and 92 Windows tests. That head predates two changes:bun run test: passed on head 77c78b448fe4 on Ubicloud (ubirun-fixwaveg-1792087761-0507a81a): 17/17 shards, 28,183 tests, 0 failed. The shard summaries don't print a skip count. It was not rerun after the fixture-string amend or the stamp; the integrator reruns it at freeze.bun run typecheckand thetypecheck:testratchet pass ("none new"). Also passing: the spawnSync tripwire, module-size and parity checks, the test-of-test ratchet, free-fixtures,gen:skill-docsfreshness (default and codex hosts), and the version, changelog and agents-digest tests.Paid checks (not run; awaiting owner approval)
Judge selection picks 0 of 24 judges. The PR profile (
bun run eval:select) picks 10 cases:The two repair reruns below are the ones that decide this PR. Each was priced with
gstack-eval-plan price-gateagainst a call plan bound to head dbda78a. The upper bounds price the recorded tokens as uncached input, so they overstate the real cost.GATE_PRICEupper boundbun run scripts/test-paid-shards.ts --tier gate --case ship-exploratory-late-inputbun run scripts/test-paid-shards.ts --tier periodic --case review-army-simplification --trials 3Receipts:
price-gate-e9990cfcf2e8-dbda78ad33ba.json(late-input)price-gate-5c8663feb784-dbda78ad33ba.json(simplification)A diagnostic run never forms a verdict; the next census on the repaired head decides (
docs/evals/census-red.md).Coordination
gstack-next-version, with main (v1.91.107.0, v1.91.107.0 fix: the GBRA-40 field list (one Bun for install and check,recgate approval, pregate never silent) #3190) merged in. It is the next slot after v1.91.106.0 fix: the 2026-10-15 issue wave #3209 (1.91.106.0) and v1.91.107.0 fix: the GBRA-40 field list (one Bun for install and check,recgate approval, pregate never silent) #3190 (1.91.107.0), both now on main. Under stamp-at-merge, the real version is stamped when this PR is next in the queue.recgate approval, pregate never silent) #3190 except the integrator-owned append-only set:CHANGELOG.md, VERSION, the package.json version line and the agents digest.