Repository navigation
fix(evals): tolerate fenced/wrapped JSON in the deterministic scorer (triage 0/5) - #811
Conversation
…(triage 0/5) skill-eval-report has reported triage holdout 0/5 — every held-out case `got null` — every day since 2026-06-15 (eval-health issue #762). Root cause: score_deterministic() in scripts/evals/run-eval.sh parses the engine output STRICTLY as a bare JSON object with NO markdown-fence stripping, while the sibling llm-judge path already strips fences. Live Haiku-tier run_triage wraps its decision in a ```json fence despite prompts/triage.md asking for none, so the strict parse yields null for every case. Offline tests passed only because their stub emits bare JSON, so CI never reproduced the live-only failure. - add a shared extract_json() helper (strips code-fence lines; `|| true` so an all-fence/empty input can't trip set -e) and use it in BOTH scorer paths (deterministic + llm-judge), replacing the judge path's inline grep. - attach a truncated `raw_excerpt` to a case result ONLY when the output failed to parse (actual=null), so a future blind null-everywhere failure is diagnosable from eval-health #762 without the (undownloadable) artifact. - tests: add a fenced-JSON stub + regression test; assert pure prose still fails (with a captured excerpt) and that passing cases carry no raw_excerpt. shellcheck clean; 26/26 bats pass. Refs #762, #583, epic #581 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GtKqCFHPEShgswWynw5bbD
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe eval runner script gains an ChangesFenced JSON tolerance and diagnostics
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Suggested labels
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Code Review
This pull request introduces a helper function extract_json to strip markdown code fences from model outputs, allowing both deterministic and LLM-judge scorers to tolerate fenced JSON payloads. It also captures a short excerpt of unparseable output (raw_excerpt) to aid in diagnosis when parsing fails, and adds corresponding regression tests. The feedback suggests optimizing the extraction of raw_excerpt in scripts/evals/run-eval.sh by replacing external processes (printf, tr, cut) with pure Bash parameter expansion.
…process) Per gemini-code-assist: replace printf|tr|cut with Bash parameter expansion; the [$'\r\n'] class also collapses carriage returns. Cold path (null-parse only); behaviour unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GtKqCFHPEShgswWynw5bbD
Dev-Lead — waiting on PR blockers (intent: fix-reviews)PR: #811 |
Dev-Lead — waiting on PR blockers (intent: review-changes)PR: #811 |
|
Note @don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/evals/run-eval.sh`:
- Around line 145-146: The jq parsing commands at lines 145-146, 166-170, and
197-199 do not enforce a single JSON document when input contains multiple JSON
objects, causing the parsed variables to become invalid multi-document text that
fails subsequent jq --argjson operations. Modify each jq invocation to use slurp
mode with array indexing to extract only the first JSON document from the input
stream before applying the filtering logic. This ensures that actual and
judge_obj variables always contain valid single JSON documents even when model
output contains multiple JSON objects, preventing hard failures in --argjson
operations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 896c52a7-4dfd-4d14-9f97-15e0cb2799d1
📒 Files selected for processing (2)
scripts/evals/run-eval.shtests/test_skill_evals.bats
Dev-Lead — fix-reviews (applied)Changes committed and pushed. |
The slurp + length==1 hardening (CodeRabbit review fix) indexes the slurped array correctly in the deterministic path, but in the llm-judge path it wrote `.[0].score` INSIDE the `.[0]|` pipe — there `.` is already the object, so `.[0]` re-indexes it (→ null) and the judge condition was always false. Every llm-judge case scored 0, failing tests/test_skill_evals.bats 12/13/15/16 (the bats CI job). Inside `.[0]|` the check is simply `.score`. 26/26 bats pass; shellcheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GtKqCFHPEShgswWynw5bbD
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-19T23:53:38Z. |
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: LOW
Reviewed commit: 8cf3af1df12498434ffa559384171c7d41d7369e
Review mode: triage-approved (single reviewer)
Summary
Hardens the eval scorer to tolerate markdown-fenced model output. Adds a shared extract_json() helper (strips ``` fences, || true so all-fence/empty input can't trip set -e) used by both the deterministic and llm-judge paths, replacing the judge path's inline grep. Captures a 160-char single-line raw_excerpt only on parse failure so a future blind null-everywhere failure is diagnosable from eval-health #762 without the artifact. Confined to eval tooling (2 files, +74/-11); no production, auth, or data-path impact; comprehensive regression tests added. Triage's low-risk assessment confirmed.
Linked issue analysis
Closes #762 (skill eval health — triage holdout 0/5). Root cause is correctly diagnosed: score_deterministic() parsed engine output strictly as bare JSON while live Haiku-tier run_triage fences its decision, yielding null for every case; offline bats passed because the stub emitted bare JSON. The fix targets the parser rather than prompts/triage.md (whose bare-JSON contract is already correct), which is the right call. Final verification (eval score > 0) is necessarily post-merge.
Findings
No blocking findings.
- extract_json correctly strips optional-whitespace-prefixed fence lines;
|| trueguards the grep-no-match exit-1 under set -e. - Switch to
jq -csewithlength==1is sound: slurped empty/multi-value/prose input falls through to the empty branch -> null actual -> fail, preserving the strict-failure semantics for genuinely malformed output. - raw_excerpt uses the original raw (not cleaned), newline-flattened and truncated to 160 chars, and is attached only when actual=null — passing/mismatch cases keep the lean shape, asserted by tests.
- Test coverage is strong: new fenced-JSON stub reproduces the live 0/5 regression; assertions confirm prose still fails (with excerpt) and passing cases carry no raw_excerpt.
Note: GitHub Secret Protection MCP scan unavailable (Advanced Security not enabled); the gitleaks CI check passed and the diff contains only fake test-stub JSON — no secrets.
CI status
All substantive checks green: ShellCheck, bats/unit-tests, CodeQL, Agent Security Scan, agent-shield, gitleaks, SonarCloud, holdout-guard, Lint, validate-agent-profiles, gh-aw-compile. dependabot/dependency-audit jobs SKIPPED (no matching ecosystems); dev-lead relay CANCELLED (review orchestration). CodeRabbit/Codex external reviews were rate-limited (advisory only, non-blocking). mergeable=MERGEABLE; mergeStateStatus=BLOCKED only on required human review.
Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.
|



Why
skill-eval-report.ymlhas reportedtriageholdout 0/5 — every held-out casegot null— every day since 2026-06-15 (eval-health tracking issue #762). This is a scorer machinery failure, not a skill regression, and it silently neutered the Phase-1 eval gate that Phase 3 (the proposer, #753/#787) depends on.Root cause
score_deterministic()inscripts/evals/run-eval.shparsed the engine output strictly as a bare JSON object with no markdown-fence stripping, while the siblingscore_llm_judge()path already strips fences.prompts/triage.mdasks for bare JSON, but the live Haiku-tierrun_triagewraps its decision in a ```json fence anyway — so the strict parse yieldednullfor every case. The offline bats tests passed only because their stub emits bare JSON, so CI never reproduced the live-only failure.Evidence: run 27822776378 — engine loaded, 5 real model calls, no auth/rate-limit error, exit 1, all 5
got null.What
extract_json()helper — strips code-fence lines (|| trueso an all-fence/empty input can't tripset -e), used in both scorer paths (deterministic + llm-judge), replacing the judge path's inlinegrep.raw_excerpt— attached to a case result only when the output failed to parse (actual=null), so the next blind null-everywhere failure is diagnosable straight from eval-health Skill eval health —triage(holdout) #762 (the raw model output otherwise lives only in the undownloadable artifact).raw_excerpt.No change to
prompts/triage.md(its contract is already correct — harden the parser, don't fight the model), the workflow, or held-out data.Verification
shellcheck --severity=warning -x scripts/evals/run-eval.sh— clean.bats tests/test_skill_evals.bats— 26/26 pass (new fenced test green; prose-fails test still holds).triage) and read Skill eval health —triage(holdout) #762 — score should become > 0; a genuine per-case miss may now surface (the actual efficacy signal), and on all-passnotify-eval-health.shauto-closes Skill eval health —triage(holdout) #762.Refs #762, #583, epic #581.
🤖 Generated with Claude Code
Generated by Claude Code
Summary by CodeRabbit