Repository navigation
Weekly periodic evals: red lane needs triage #3153
Description
Periodic census report: branch main @ e23f7b8, event schedule, run https://github.com/garrytan/gstack/actions/runs/38212265208
A census went red: triage with https://github.com/garrytan/gstack/blob/e23f7b807ff4788f6d7c90af2ab82960bd8c5851/docs/evals/census-red.md
- periodic reconciliation exit: 0 (slices job: success, Codex slices job: success)
- gate census reconciliation exit: 1 (census job: failure)
- pass-rate history gate exit: 1
- all-green probability (approximation, assumes independent verdicts): 0% for the latest census's 148 verdicts [0%–0% with the pooled rate's 95% interval as prior]; formula Π(1 - p_i) over its cases; rerun: bun run eval:pass-rates --reds
Periodic lane
[test:paid] VERDICT GREEN — lane periodic/full census, attempt 1
rule 103/103 · behavior 24/24 · judge 25/25 · quarantined 0 (0 failing the lane)
SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
wall 10m10s · cost $101.35 known + 74 of 200 trial(s) cost unknown
Skipped, deselected and unattributed testcases
deselected testcases, zero credit (322):
305 sibling case of a case shard (its own shard runs it)
8 Bun hook placeholder of a skipped describe block (not a test)
7 gate-tier case (this lane runs periodic)
2 isolated case (its trial shards run it)
Gate census
[test:paid] VERDICT RED — lane gate/full census, attempt 1
rule 107/108 · behavior 0/0 · judge 0/0 · quarantined 0 (0 failing the lane)
SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 2
wall 8m18s · cost $90.44 known + 54 of 108 trial(s) cost unknown
Failures and split verdicts
✗ ship-exploratory-late-input rule FAIL assertion — expect(received).toEqual(expected) · cause assertion: session completed; check failed Expected: [] · Received: [ "probe preceded resource read: exploratory", "probe preceded resource read: system-functional", "probe preceded resource read: system-functional", ] [slice 6, attempt 1] evidence: gate-census-6-a1/shards/skill-e2e-qa-callers--ship-exploratory-late-input (fetch: bun run eval:pass-rates --run 38212265208 --case ship-exploratory-late-input) after a repair: bun run scripts/test-paid-shards.ts --tier gate --case ship-exploratory-late-input
Skipped, deselected and unattributed testcases
deselected testcases, zero credit (307):
293 sibling case of a case shard (its own shard runs it)
8 periodic-tier case (this lane runs gate)
6 Bun hook placeholder of a skipped describe block (not a test)
ACTION REQUIRED (2)
test/skill-e2e-qa-callers.test.ts#ship-exploratory-late-input: failed
1 unapproved final collector failure(s)
Guide: docs/evals/census-red.md
Pass-rate history (ACTION REQUIRED)
ACTION REQUIRED (1):
[headroom] review-army-simplification session claude-p:review-army-simplification#1: max 159s of 180s (88.1%) over 3 sample(s), above the 85% cap. Cut work in the skill or fixture; budgets are never raised (TESTING_INTERNALS "Timeout policy").
Red ledger: bun run eval:pass-rates --reds --runs 10 --branch main
$ bun run scripts/eval-flake-rank.ts --reds --runs "10" --branch main
<REDACTED-pii.phone.e164> gate/full census: 0 red of 106 verdicts
<REDACTED-pii.phone.e164> periodic/full census: 3 red of 146 verdicts — class: assertion 3; cause: unrecorded 3
✗ plan-ceo-mode-routing FAIL t1 assertion/unrecorded — Mode "HOLD SCOPE" routing FAILED after sending option 2: no posture match for \b(rigor|bulletproof|hold\s*scope|maximum\s+rigor)\b.
✗ setup-deploy/SKILL.md platform setup FAIL t1 assertion/unrecorded — expect(received).toBeGreaterThanOrEqual(expected)
✗ auto-decide-preserved FAIL t1 assertion/unrecorded — AUTO_DECIDE preserve test inconclusive: outcome=timeout
<REDACTED-pii.phone.e164> gate/full census: 1 red of 106 verdicts — class: assertion 1; cause: unrecorded 1
✗ tpa-absent-darwin FAIL t1 assertion/unrecorded — expect(received).toMatch(expected)
<REDACTED-pii.phone.e164> periodic/full census: 2 red of 146 verdicts — class: assertion 2; cause: unrecorded 2
✗ plan-design-review-plan-mode FAIL t1 assertion/unrecorded — expect(received).toBe(expected)
✗ auto-decide-preserved FAIL t1 assertion/unrecorded — AUTO_DECIDE preserve test inconclusive: outcome=timeout
<REDACTED-pii.phone.e164> gate/full census: 0 red of 106 verdicts
<REDACTED-pii.phone.e164> periodic/full census: 3 red of 146 verdicts — class: assertion 3; cause: unrecorded 3
✗ plan-ceo-section-loading FAIL t1 assertion/unrecorded — Error: expect(received).toBe(expected)
✗ test/skill-e2e-auq-matrix.test.ts FAIL t1 assertion/unrecorded — plan-design-review AUQ not at plan-ceo bar:
✗ auto-decide-preserved FAIL t1 assertion/unrecorded — AUTO_DECIDE preserve test inconclusive: outcome=timeout
<REDACTED-pii.phone.e164> gate/full census: 1 red of 106 verdicts — class: timeout 1; cause: unrecorded 1
✗ shared-libs-review-revalidation FAIL t1 timeout/unrecorded — Error: Claude Code process aborted by user
<REDACTED-pii.phone.e164> periodic/full census: 2 red of 146 verdicts — class: assertion 2; cause: unrecorded 2
✗ qa/SKILL.md workflow FAIL t1 assertion/unrecorded — expect(received).toBeGreaterThanOrEqual(expected)
✗ shared-libs-plan-callers FAIL t1 assertion/unrecorded — Error: shared-libs-plan-callers actor: question expands beyond unchanged-helper reuse and its required proof: hardening
<REDACTED-pii.phone.e164> gate/full census: 0 red of 106 verdicts
<REDACTED-pii.phone.e164> periodic/full census: 2 red of 146 verdicts — class: assertion 2; cause: unrecorded 2
✗ review/SKILL.md workflow FAIL t1 assertion/unrecorded — expect(received).toBeGreaterThanOrEqual(expected)
✗ plan-ceo-section-loading FAIL t1 assertion/unrecorded — Error: expect(received).toBe(expected)
<REDACTED-pii.phone.e164> gate/full census: 0 red of 106 verdicts
<REDACTED-pii.phone.e164> periodic/full census: 2 red of 146 verdicts — class: assertion 3; cause: unrecorded 3
✗ plan-ceo-review-format-approach FAIL t1 assertion/unrecorded — expect(received).toContain(expected); t2 assertion/unrecorded — expect(received).toContain(expected)
✗ shared-libs-plan-callers FAIL t1 assertion/unrecorded — Error: shared-libs-plan-callers actor: question expands beyond unchanged-helper reuse and its required proof: hardening
<REDACTED-pii.phone.e164> gate/full census: 0 red of 106 verdicts
<REDACTED-pii.phone.e164> periodic/full census: 2 red of 146 verdicts — class: assertion 2; cause: unrecorded 2
✗ qa-only/SKILL.md workflow FAIL t1 assertion/unrecorded — expect(received).toBeGreaterThanOrEqual(expected)
✗ overlay-harness-claude-dedicated-tools-vs-bash-sonnet FAIL t1 assertion/unrecorded — expect(received).toBe(expected)
<REDACTED-pii.phone.e164> periodic/full census: 3 red of 146 verdicts — class: assertion 3; cause: unrecorded 3
✗ plan-eng-multi-finding-batching FAIL t1 assertion/unrecorded — BATCHING REGRESSION: reviewCount=0 < FLOOR=3.
✗ test/skill-e2e-auq-matrix.test.ts FAIL t1 assertion/unrecorded — plan-design-review AUQ not at plan-ceo bar:
✗ plan-eng-review-plan-mode FAIL t1 assertion/unrecorded — STOP-gate regression: outcome=timeout
<REDACTED-pii.phone.e164> periodic/full census: 0 red of 146 verdicts
<REDACTED-pii.phone.e164> gate/full census: 0 red of 106 verdicts
<REDACTED-pii.phone.e164> periodic/full census: 1 red of 146 verdicts — class: assertion 1; cause: unrecorded 1
✗ auto-decide-preserved FAIL t1 assertion/unrecorded — AUTO_DECIDE preserve test inconclusive: outcome=timeout
<REDACTED-pii.phone.e164> periodic/full census: 0 red of 148 verdicts
pooled: 1038 red of 5854 verdicts = 17.7% per verdict [95% 16.8%–18.7%]
per-case red rates (shrunk toward the pooled rate, K=4):
87.3% 22/22 codex-offered-ceo-review
87.3% 22/22 codex-offered-design-review
87.3% 22/22 codex-offered-eng-review
87.3% 22/22 codex-offered-office-hours
87.3% 22/22 office-hours-spec-review
87.3% 22/22 plan-ceo-review-benefits
80.6% 13/13 design-review-fix
66.8% 26/36 design-review-detector-shim
66.8% 26/36 design-review-detector-shim-dom
66.8% 26/36 design-review-plugin-handoff
66.8% 26/36 plan-design-review-no-ui-scope
65.1% 26/37 plan-eng-review-artifact
64.4% 27/39 plan-design-review-plan-mode
62.1% 26/39 design-consultation-core
62.1% 26/39 design-consultation-existing
all-green probability (approximation, assumes independent verdicts): 0% for the latest census's 148 verdicts [0%–0% with the pooled rate's 95% interval as prior]; formula Π(1 - p_i) over its cases; rerun: bun run eval:pass-rates --reds
guide: docs/evals/census-red.md
Session headroom: bun run eval:pass-rates --headroom --runs 10 --branch main
$ bun run scripts/eval-flake-rank.ts --headroom --runs "10" --branch main
scope: garrytan/gstack evals-periodic.yml on main + 29 pooled branch census run(s), last 10 completed run(s): <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, <REDACTED-pii.phone.e164>, 3717
headroom: slowest armed session per case (warn > 75%, alarm > 85%; insufficient below 3 samples; censored = timed out)
status session max/budget samples case wall (upper bound, not a session clock) case
over 88.1% 159s/180s 3 180s review-army-simplification [claude-p:review-army-simplification#1]
insufficient 88% 528s/600s 1 600s qa-fix-loop [claude-p:qa-fix-loop#1]
insufficient 84.7% 458s/540s 1 458s ship-section-loading [claude-p:ship-section-loading#1]
insufficient 83.4% 250s/300s 1 276s shared-libs-plan-callers [agent-sdk:shared-libs-plan-callers#1]
insufficient 82.1% 394s/480s 1 394s plan-ceo-section-loading [claude-p:plan-ceo-section-loading#1]
insufficient 73.4% 352s/480s 1 419s carve-section-loading-plan-eng-review [claude-p:plan-eng-review section-loading#1]
insufficient 72.5% 348s/480s 1 388s carve-section-loading-plan-devex-review [claude-p:plan-devex-review section-loading#1]
insufficient 62.8% 188s/300s 1 303s shared-libs-pr-coverage [claude-p:shared-libs-pr-coverage#1]
ok 61.2% 183s/300s 3 184s design-html-slop-gate [claude-p:design-html-slop-gate#1]
insufficient 61.1% 183s/300s 1 230s retro [claude-p:retro#1]
insufficient 59.8% 359s/600s 1 742s plan-ceo-mode-routing [pty:session#2]
insufficient 58.4% 281s/480s 1 281s carve-section-loading-plan-design-review [claude-p:plan-design-review section-loading#1]
insufficient 58% 278s/480s 1 360s carve-section-loading-design-html [claude-p:design-html section-loading#1]
insufficient 57.5% 311s/540s 1 311s plan-ceo-review [claude-p:plan-ceo-review#1]
insufficient 55.9% 168s/300s 1 236s qa-b8-checkout [claude-p:qa-b8-checkout#1]
ok 55.8% 67s/120s 3 107s scrape-match-path [claude-p:scrape-match-path#1]
insufficient 53.5% 321s/600s 1 595s plan-eng-review-artifact [pty:session#1]
insufficient 52.5% 252s/480s 1 336s office-hours-design-draft [claude-p:office-hours-design-draft#1]
insufficient 50.1% 150s/300s 1 199s health-reporting [claude-p:health-reporting#1]
ok 48.2% 145s/300s 3 268s shared-libs-opportunity-judgment [claude-p:shared-libs-opportunity-judgment#2]
insufficient 47.6% 143s/300s 1 300s plan-design-review-plan-mode [claude-p:plan-design-review-plan-mode#1]
insufficient 47.6% 143s/300s 1 147s design-consultation-preview [claude-p:design-consultation-preview#1]
ok 44.3% 133s/300s 3 183s review-army-red-team [claude-p:review-army-red-team#1]
insufficient 43.9% 53s/120s 1 76s context-save-routing [claude-p:context-save-routing#1]
insufficient 43.5% 209s/480s 1 265s carve-section-loading-design-consultation [claude-p:design-consultation section-loading#1]
ok 42.8% 128s/300s 3 200s review-design-lite [claude-p:review-design-lite#1]
insufficient 42.6% 128s/300s 1 149s qa-b6-static [claude-p:qa-b6-static#1]
ok 42.2% 253s/600s 3 358s office-hours-brain-writeback [claude-p:office-hours-brain-writeback#1]
insufficient 40.6% 122s/300s 1 138s design-consultation-research [claude-p:design-consultation-research#1]
insufficient 39.5% 119s/300s 1 162s codex-review [claude-p:codex-review#1]
insufficient 38.6% 232s/600s 1 405s cso-full-audit [claude-p:cso-full-audit#1]
insufficient 38.3% 115s/300s 1 320s plan-eng-review-plan-mode [pty:session#2]
ok 37.7% 113s/300s 3 300s review-army-consensus [claude-p:review-army-consensus#1]
insufficient 36% 108s/300s 1 152s plan-design-review-plan-mode-smoke [pty:session#2]
insufficient 35.7% 172s/480s 1 172s carve-section-loading-design-shotgun [claude-p:design-shotgun section-loading#1]
insufficient 35.7% 214s/600s 1 214s outside-voice-claude-code-to-codex [claude-p:outside-voice-claude-code-to-codex#1]
Full reconciliation output
test/codex-e2e-recommendation-substance.test.ts
slice 25 passed 157s test/carve-section-loading.test.ts#carve-section-loading-qa
slice 25 passed 142s test/skill-e2e-auq-matrix.test.ts
slice 25 trial passed 44s test/skill-e2e-context-skills.test.ts#context-restore-provenance-order~t1
slice 25 trial passed 142s test/skill-e2e-review-army.test.ts#review-army-simplification~t1
slice 25 passed 166s test/skill-e2e-setup-gbrain-path4-local-pglite.test.ts
slice 25 passed 112s test/skill-e2e-plan-design-plan-mode.test.ts
slice 26 trial passed 40s test/skill-e2e-context-skills.test.ts#context-restore-provenance-order~t2
slice 26 trial passed 91s test/skill-e2e-review-army.test.ts#review-army-simplification~t2
slice 26 passed 130s test/carve-section-loading.test.ts#carve-section-loading-design-shotgun
slice 26 trial passed 112s test/skill-e2e-design.test.ts#design-html-slop-gate~t1
slice 26 trial passed 70s test/skill-e2e-office-hours.test.ts#office-hours-builder-wildness~t1
slice 26 passed 193s test/skill-e2e-plan-design-finding-floor.test.ts
slice 27 trial passed 41s test/skill-e2e-context-skills.test.ts#context-restore-provenance-order~t3
slice 27 trial passed 146s test/skill-e2e-review-army.test.ts#review-army-simplification~t3
slice 27 trial passed 157s test/skill-e2e-design.test.ts#design-html-slop-gate~t2
slice 27 trial passed 103s test/skill-e2e-office-hours.test.ts#office-hours-builder-wildness~t2
slice 27 trial passed 50s test/skill-e2e-office-hours.test.ts#office-hours-forcing-energy~t1
slice 27 trial passed 118s test/skill-e2e-review-army.test.ts#review-army-red-team~t2
slice 28 trial passed 139s test/skill-e2e-design.test.ts#design-html-slop-gate~t3
slice 28 trial passed 82s test/skill-e2e-office-hours.test.ts#office-hours-builder-wildness~t3
slice 28 trial passed 58s test/skill-e2e-office-hours.test.ts#office-hours-forcing-energy~t2
slice 28 trial passed 100s test/skill-e2e-review-army.test.ts#review-army-consensus~t2
slice 28 trial passed 95s test/skill-e2e-plan.test.ts#plan-ceo-review-expansion-energy~t1
slice 28 passed 105s test/skill-e2e-design.test.ts#design-consultation-research
slice 28 passed 74s test/skill-e2e-arm-benchmark.test.ts
slice 28 trial passed 46s test/skill-e2e-office-hours-phase4.test.ts#office-hours-phase4-fork~t1
slice 29 trial passed 56s test/skill-e2e-office-hours.test.ts#office-hours-forcing-energy~t3
slice 29 trial passed 109s test/skill-e2e-review-army.test.ts#review-army-consensus~t3
slice 29 trial passed 92s test/skill-e2e-plan.test.ts#plan-ceo-review-expansion-energy~t2
slice 29 passed 81s test/carve-section-loading.test.ts#carve-section-loading-document-release
slice 29 passed 110s test/carve-section-loading.test.ts#carve-section-loading-browse
slice 29 passed 11s test/skill-e2e-design-model-smoke.test.ts
slice 29 passed 98s test/skill-e2e-setup-gbrain-remote.test.ts
slice 29 passed 94s test/skill-e2e-plan-decision-classification.test.ts
slice 30 trial passed 96s test/skill-e2e-plan.test.ts#plan-ceo-review-expansion-energy~t3
slice 30 passed 65s test/skill-routing-e2e.test.ts
slice 30 passed 80s test/carve-section-loading.test.ts#carve-section-loading-setup-gbrain
slice 30 trial passed 61s test/skill-e2e-skillify.test.ts#scrape-match-path~t1
slice 30 trial passed 40s test/skill-e2e-skillify.test.ts#scrape-prototype-path~t1
slice 30 passed 86s test/skill-e2e-safety-design-risk.test.ts
slice 30 passed 55s test/skill-e2e-sync-gbrain-readiness.test.ts
slice 30 trial passed 44s test/skill-e2e-office-hours-phase4.test.ts#office-hours-phase4-fork~t3
slice 30 passed 44s test/skill-e2e-setup-gbrain-bad-token.test.ts
slice 31 trial passed 68s test/skill-e2e-skillify.test.ts#scrape-match-path~t2
slice 31 trial passed 37s test/skill-e2e-skillify.test.ts#scrape-prototype-path~t2
slice 32 trial passed 63s test/skill-e2e-skillify.test.ts#scrape-match-path~t3
slice 32 trial passed 34s test/skill-e2e-skillify.test.ts#scrape-prototype-path~t3
slice 33 passed 60s test/codex-e2e-shared-libs.test.ts
slice 34 passed 321s test/skill-e2e-overlay-harness-opus-4-7-literal-interpretation.test.ts
slice 34 passed 93s test/skill-e2e-overlay-harness-claude-dedicated-tools-vs-bash-sonnet.test.ts
slice 34 passed 65s test/skill-e2e-overlay-harness-claude-dedicated-tools-vs-bash-opus-5-5.test.ts
slice 34 passed 64s test/skill-e2e-overlay-harness-claude-dedicated-tools-vs-bash.test.ts
slice 34 passed 48s test/skill-e2e-overlay-harness-opus-4-7-effort-match-trivial.test.ts
[test:paid] collector final outcomes: 194 executed, 0 reused; 194 passed, 0 failed, 0 manual accepted (unscored; no score-cache credit) (194 attempt records from 40 collectors; every record counts)
[test:paid] deselected testcases, zero credit (322):
[test:paid] 305 sibling case of a case shard (its own shard runs it)
[test:paid] 8 Bun hook placeholder of a skipped describe block (not a test)
[test:paid] 7 gate-tier case (this lane runs periodic)
[test:paid] 2 isolated case (its trial shards run it)
[test:paid] VERDICT GREEN — lane periodic/full census, attempt 1
rule 103/103 · behavior 24/24 · judge 25/25 · quarantined 0 (0 failing the lane)
SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
wall 10m10s · cost $101.35 known + 74 of 200 trial(s) cost unknown
[test:paid] report: every planned shard accounted and passed
sed 194s test/skill-e2e-qa-functional-fix.test.ts
slice 8 passed 156s test/skill-e2e-retro.test.ts
slice 9 passed 234s test/skill-e2e-plan.test.ts#plan-review-report
slice 9 passed 227s test/skill-e2e-qa-callers.test.ts#ship-exploratory-unavailable
slice 9 passed 176s test/skill-e2e-shared-libs-paths.test.ts#shared-libs-review-prior-coverage
slice 9 passed 183s test/skill-e2e-qa-workflow.test.ts#qa-quick
slice 9 passed 58s test/skill-e2e-review-army.test.ts#review-army-quality-score
slice 9 passed 61s test/skill-e2e-ship-hook-consent.test.ts
slice 10 passed 173s test/skill-e2e-ship-docsync.test.ts#ship-docsync-stale-before
slice 10 passed 149s test/skill-e2e-shared-libs.test.ts#shared-libs-review-lifecycle
slice 10 passed 155s test/skill-e2e-review-army.test.ts#review-army-migration-safety
slice 10 passed 240s test/skill-e2e-shared-libs.test.ts#shared-libs-read-only
slice 10 passed 70s test/skill-e2e-shared-libs.test.ts#shared-libs-unsupported-git
slice 10 passed 69s test/skill-e2e-ship-skip.test.ts
slice 11 passed 151s test/skill-e2e-ship-docsync.test.ts#ship-docsync-stale-after
slice 11 passed 131s test/skill-e2e-deploy.test.ts#land-and-deploy-review-gate
slice 11 passed 161s test/skill-e2e-qa-callers.test.ts#ship-exploratory-small-cli
slice 11 passed 164s test/skill-e2e-shared-libs-paths.test.ts#shared-libs-review-index-flags
slice 11 passed 112s test/skill-e2e-deploy.test.ts#benchmark-workflow
slice 11 passed 161s test/skill-e2e-review-army.test.ts#review-army-perf-n-plus-one
slice 12 passed 128s test/skill-e2e-ship-docsync.test.ts#ship-docsync-late-result
slice 12 passed 112s test/skill-e2e-plan-ceo-plan-mode.test.ts
slice 12 passed 111s test/skill-e2e-docsync-spawned.test.ts
slice 12 passed 97s test/skill-e2e-ship-docsync.test.ts#ship-docsync-missing-asset
slice 12 passed 121s test/skill-e2e-plan-devex-plan-mode.test.ts
slice 12 passed 115s test/skill-e2e-deploy.test.ts#land-and-deploy-first-run
slice 12 passed 102s test/skill-e2e-design.test.ts#design-review-detector-shim-dom
slice 12 passed 86s test/skill-e2e-design.test.ts#design-review-plugin-handoff
slice 13 passed 94s test/skill-e2e-safety-ship-evidence.test.ts
slice 13 passed 92s test/skill-e2e-review-attribution.test.ts
slice 13 passed 78s test/skill-e2e-office-hours-auto-mode.test.ts
slice 13 passed 87s test/skill-e2e-bws.test.ts
slice 13 passed 74s test/skill-e2e-third-party-actions.test.ts
slice 13 passed 74s test/skill-e2e-plan-tune.test.ts
slice 13 passed 58s test/skill-e2e-coverage-audit.test.ts
slice 13 passed 64s test/skill-e2e-triage.test.ts
slice 13 passed 81s test/skill-e2e-ship-docsync.test.ts#ship-docsync-timeout-unsettled
slice 13 passed 57s test/skill-e2e-ship-docsync.test.ts#ship-docsync-missing-marker
slice 13 passed 46s test/skill-e2e-ask-user-question-format-compliance.test.ts
slice 13 passed 57s test/skill-e2e-investigate-owned-termination.test.ts
slice 13 passed 7s test/skill-e2e-hermetic-canary.test.ts
slice 14 passed 62s test/skill-e2e-ship-docsync.test.ts#ship-docsync-launch-failure
slice 14 passed 54s test/skill-e2e-session-intelligence.test.ts
slice 14 passed 65s test/skill-e2e-design.test.ts#design-review-detector-shim
slice 14 passed 38s test/skill-e2e-deploy.test.ts#canary-workflow
slice 14 passed 37s test/skill-e2e-ship-hook-refresh.test.ts
slice 14 passed 52s test/skill-e2e-design.test.ts#plan-design-review-no-ui-scope
slice 14 passed 51s test/skill-e2e-review-army.test.ts#review-army-delivery-audit
slice 14 passed 47s test/skill-e2e-investigate-owned-completion.test.ts
slice 14 passed 40s test/skill-e2e-safety-pair-agent.test.ts
slice 14 passed 45s test/skill-e2e-qa-workflow.test.ts#qa-bootstrap
slice 14 passed 39s test/skill-e2e-learnings.test.ts
slice 14 passed 29s test/skill-e2e-diagram.test.ts
[test:paid] collector final outcomes: 97 executed, 0 reused; 96 passed, 1 failed, 0 manual accepted (unscored; no score-cache credit) (97 attempt records from 69 collectors; every record counts)
[test:paid] deselected testcases, zero credit (307):
[test:paid] 293 sibling case of a case shard (its own shard runs it)
[test:paid] 8 periodic-tier case (this lane runs gate)
[test:paid] 6 Bun hook placeholder of a skipped describe block (not a test)
[test:paid] VERDICT RED — lane gate/full census, attempt 1
rule 107/108 · behavior 0/0 · judge 0/0 · quarantined 0 (0 failing the lane)
SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 2
wall 8m18s · cost $90.44 known + 54 of 108 trial(s) cost unknown
[test:paid] failures and split verdicts:
✗ ship-exploratory-late-input rule FAIL assertion — expect(received).toEqual(expected) · cause assertion: session completed; check failed Expected: [] · Received: [ "probe preceded resource read: exploratory", "probe preceded resource read: system-functional", "probe preceded resource read: system-functional", ] [slice 6, attempt 1] evidence: gate-census-6-a1/shards/skill-e2e-qa-callers--ship-exploratory-late-input (fetch: bun run eval:pass-rates --run 38212265208 --case ship-exploratory-late-input) after a repair: bun run scripts/test-paid-shards.ts --tier gate --case ship-exploratory-late-input
Policy: EVAL_POLICY and CASE_QUARANTINE in test/helpers/periodic-exclude-data.ts; history: bun run eval:pass-rates.
Activity
v1.91.105.0 fix(tests): the free suite stops needing its retry pass; …
fix(qa-callers): the caller scheduling contract no longer calls a pro…
fix(review-army): the simplification case drops its false premise and…
docs(evals): record the #3153 census diagnoses (review-army-simplific…
chore(release): tentative v1.91.107.0 stamp and CHANGELOG for the #3153…
chore(release): tentative v1.91.108.0 stamp and CHANGELOG for the #3153…
v1.91.108.0 fix(evals): census repair — the late-input and simplifica…
capy-ai commentedon Oct 16, 2026
Closing this on the evidence below. The Oct 19 census will confirm both cases in the full lane, and a red there reopens it.
The Oct 12 census flagged two cases.
review-army-simplification (headroom). Its sessions pooled 159 s of the 180 s budget, 88.1% against an 85% cap, and census run 38212265208's own panel ran 142, 91 and 146 s. The transcripts showed the fixture, not the skill, causing the extra work: the case read the full extracted /review prose, searched for a heading the carved fixture no longer has, tried to reconcile a stated "over 100 lines" with a 68-line diff, and then wrote an unbounded report.
#3200 (v1.91.108.0, merged as b0f1e65) made three changes and left the 180 s / 20-turn budget and every assertion unchanged:
stageReviewArmySession);/review --simplificationflag;A post-merge periodic panel on b0f1e65 passed 3/3: 3 trials, behavior kind, Bun 1.4.2, Claude Code 2.1.292, no retries. The trials took 57.5 s, 40.9 s and 55.4 s at 4–5 turns each, and cost $1.34 in total. The slowest used 32% of the budget.
ship-exploratory-late-input (gate red). The repair in the same PR has passed four PR-gate runs in a row, most recently $2.43 on dbda78a.