Skip to content

fix(evals): tolerate fenced/wrapped JSON in the deterministic scorer (triage 0/5) - #811

Merged
don-petry merged 6 commits into
mainfrom
claude/fix-eval-scorer-fenced-json
Jun 20, 2026
Merged

don-petry merged 6 commits into
mainfrom
claude/fix-eval-scorer-fenced-json

Conversation

@don-petry

@don-petry don-petry commented Jun 19, 2026 •

Copy link
Copy Markdown
Collaborator

Why

skill-eval-report.yml has reported triage holdout 0/5 — every held-out case got null — every day since 2026-06-15 (eval-health tracking issue #762). This is a scorer machinery failure, not a skill regression, and it silently neutered the Phase-1 eval gate that Phase 3 (the proposer, #753/#787) depends on.

Root cause

score_deterministic() in scripts/evals/run-eval.sh parsed the engine output strictly as a bare JSON object with no markdown-fence stripping, while the sibling score_llm_judge() path already strips fences. prompts/triage.md asks for bare JSON, but the live Haiku-tier run_triage wraps its decision in a ```json fence anyway — so the strict parse yielded null for every case. The offline bats tests passed only because their stub emits bare JSON, so CI never reproduced the live-only failure.

Evidence: run 27822776378 — engine loaded, 5 real model calls, no auth/rate-limit error, exit 1, all 5 got null.

What

  • Shared extract_json() helper — strips code-fence lines (|| true so an all-fence/empty input can't trip set -e), used in both scorer paths (deterministic + llm-judge), replacing the judge path's inline grep.
  • raw_excerpt — attached to a case result only when the output failed to parse (actual=null), so the next blind null-everywhere failure is diagnosable straight from eval-health Skill eval health — triage (holdout) #762 (the raw model output otherwise lives only in the undownloadable artifact).
  • Tests — new fenced-JSON stub + regression test reproducing the live 0/5; assertions that pure prose still fails (with a captured excerpt) and that passing cases carry no raw_excerpt.

No change to prompts/triage.md (its contract is already correct — harden the parser, don't fight the model), the workflow, or held-out data.

Verification

  • shellcheck --severity=warning -x scripts/evals/run-eval.sh — clean.
  • bats tests/test_skill_evals.bats — 26/26 pass (new fenced test green; prose-fails test still holds).
  • Post-merge (the real proof): dispatch Skill Eval Report (skill=triage) and read Skill eval health — triage (holdout) #762 — score should become > 0; a genuine per-case miss may now surface (the actual efficacy signal), and on all-pass notify-eval-health.sh auto-closes Skill eval health — triage (holdout) #762.

Refs #762, #583, epic #581.

🤖 Generated with Claude Code


Generated by Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Hardened evaluation scoring to accept Markdown-fenced JSON produced by models.
    • Improved failure diagnostics by recording a concise excerpt of unparseable output when scoring can’t determine a decision.
  • Tests
    • Added regression coverage for fenced JSON handling.
    • Expanded assertions to verify diagnostic fields are present only on parse failures.

…(triage 0/5)

skill-eval-report has reported triage holdout 0/5 — every held-out case
`got null` — every day since 2026-06-15 (eval-health issue #762). Root cause:
score_deterministic() in scripts/evals/run-eval.sh parses the engine output
STRICTLY as a bare JSON object with NO markdown-fence stripping, while the
sibling llm-judge path already strips fences. Live Haiku-tier run_triage wraps
its decision in a ```json fence despite prompts/triage.md asking for none, so
the strict parse yields null for every case. Offline tests passed only because
their stub emits bare JSON, so CI never reproduced the live-only failure.

- add a shared extract_json() helper (strips code-fence lines; `|| true` so an
  all-fence/empty input can't trip set -e) and use it in BOTH scorer paths
  (deterministic + llm-judge), replacing the judge path's inline grep.
- attach a truncated `raw_excerpt` to a case result ONLY when the output failed
  to parse (actual=null), so a future blind null-everywhere failure is
  diagnosable from eval-health #762 without the (undownloadable) artifact.
- tests: add a fenced-JSON stub + regression test; assert pure prose still fails
  (with a captured excerpt) and that passing cases carry no raw_excerpt.

shellcheck clean; 26/26 bats pass.

Refs #762, #583, epic #581

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GtKqCFHPEShgswWynw5bbD
@don-petry
don-petry requested a review from a team as a code owner June 19, 2026 22:05
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Jun 19, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 48c63482-576f-4eed-a065-e22cd61e909a

📥 Commits

Reviewing files that changed from the base of the PR and between 9f091b0 and 2ccc948.

📒 Files selected for processing (1)
  • scripts/evals/run-eval.sh

📝 Walkthrough

Walkthrough

The eval runner script gains an extract_json helper that strips Markdown triple-backtick code fences from raw model output before JSON parsing, used in both deterministic and llm-judge scoring paths. Deterministic scoring now captures a truncated raw_excerpt on parse failure. New bats tests cover the fenced-output regression and the raw_excerpt presence/absence contract.

Changes

Fenced JSON tolerance and diagnostics

Layer / File(s) Summary
extract_json helper and deterministic scoring with raw_excerpt
scripts/evals/run-eval.sh
Adds extract_json function to strip triple-backtick fences before jq parsing. Deterministic scoring uses it to clean model output, parses the expected escalate/risk JSON object, and on parse failure sets actual=null and captures a truncated, single-line raw_excerpt from the original output. The per-case JSON report conditionally includes raw_excerpt only when non-empty (i.e., on parse failure).
llm-judge scoring using extract_json
scripts/evals/run-eval.sh
Replaces the previous inline grep code-fence stripping of judge_raw with the shared extract_json helper, then parses a single JSON object containing a numeric score (clamped to [0,1]) and extracts reason (defaulting to null).
Stub fenced engine and raw_excerpt test assertions
tests/test_skill_evals.bats
Adds STUB_FENCED stub that emits valid JSON inside a Markdown code fence. Asserts passing cases omit raw_excerpt. Asserts non-JSON output sets raw_excerpt containing expected prose. Adds a regression test confirming fenced JSON is parsed and scored correctly for approve and escalate decisions.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Suggested labels

needs-human-review

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: fixing the deterministic scorer to handle markdown-fenced JSON output in the triage skill evaluation.
Linked Issues check ✅ Passed The PR directly addresses issue #762 by implementing the extract_json() helper to parse fenced JSON and restore scoring for all five failing triage test cases.
Out of Scope Changes check ✅ Passed All changes are scoped to fixing the JSON parsing issue in the deterministic scorer and adding related test coverage; no unrelated modifications are present.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/fix-eval-scorer-fenced-json

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a helper function extract_json to strip markdown code fences from model outputs, allowing both deterministic and LLM-judge scorers to tolerate fenced JSON payloads. It also captures a short excerpt of unparseable output (raw_excerpt) to aid in diagnosis when parsing fails, and adds corresponding regression tests. The feedback suggests optimizing the extraction of raw_excerpt in scripts/evals/run-eval.sh by replacing external processes (printf, tr, cut) with pure Bash parameter expansion.

Comment thread scripts/evals/run-eval.sh Resolved
@don-petry
don-petry enabled auto-merge (squash) June 19, 2026 22:07
…process)

Per gemini-code-assist: replace printf|tr|cut with Bash parameter expansion;
the [$'\r\n'] class also collapses carriage returns. Cold path (null-parse
only); behaviour unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GtKqCFHPEShgswWynw5bbD
@don-petry
don-petry disabled auto-merge June 19, 2026 22:07
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — waiting on PR blockers (intent: fix-reviews)

PR: #811
No changes were committed, but the PR still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews). The retry cron will re-attempt automatically. Next attempt after: 2026-06-19T22:40:24Z

@don-petry
don-petry enabled auto-merge (squash) June 19, 2026 22:10
@don-petry
don-petry disabled auto-merge June 19, 2026 22:42
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #811
No changes were committed, but the PR still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews). The retry cron will re-attempt automatically. Next attempt after: 2026-06-19T23:13:22Z

@don-petry

Copy link
Copy Markdown
Collaborator Author

Note

@don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically.
Next attempt after: 2026-06-19T23:13:22Z

@don-petry
don-petry enabled auto-merge (squash) June 19, 2026 22:43

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/evals/run-eval.sh`:
- Around line 145-146: The jq parsing commands at lines 145-146, 166-170, and
197-199 do not enforce a single JSON document when input contains multiple JSON
objects, causing the parsed variables to become invalid multi-document text that
fails subsequent jq --argjson operations. Modify each jq invocation to use slurp
mode with array indexing to extract only the first JSON document from the input
stream before applying the filtering logic. This ensures that actual and
judge_obj variables always contain valid single JSON documents even when model
output contains multiple JSON objects, preventing hard failures in --argjson
operations.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 896c52a7-4dfd-4d14-9f97-15e0cb2799d1

📥 Commits

Reviewing files that changed from the base of the PR and between 9cb66bb and 9f091b0.

📒 Files selected for processing (2)
  • scripts/evals/run-eval.sh
  • tests/test_skill_evals.bats

Comment thread scripts/evals/run-eval.sh Resolved
@don-petry
don-petry disabled auto-merge June 19, 2026 22:46
coderabbitai[bot]
coderabbitai Bot previously approved these changes Jun 19, 2026
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (applied)

Changes committed and pushed.

coderabbitai[bot]
coderabbitai Bot previously approved these changes Jun 19, 2026
@don-petry
don-petry disabled auto-merge June 19, 2026 22:49
The slurp + length==1 hardening (CodeRabbit review fix) indexes the slurped
array correctly in the deterministic path, but in the llm-judge path it wrote
`.[0].score` INSIDE the `.[0]|` pipe — there `.` is already the object, so
`.[0]` re-indexes it (→ null) and the judge condition was always false. Every
llm-judge case scored 0, failing tests/test_skill_evals.bats 12/13/15/16 (the
bats CI job). Inside `.[0]|` the check is simply `.score`.

26/26 bats pass; shellcheck clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GtKqCFHPEShgswWynw5bbD
@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-19T23:53:38Z.

@don-petry
don-petry enabled auto-merge (squash) June 19, 2026 22:53

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: 8cf3af1df12498434ffa559384171c7d41d7369e
Review mode: triage-approved (single reviewer)

Summary

Hardens the eval scorer to tolerate markdown-fenced model output. Adds a shared extract_json() helper (strips ``` fences, || true so all-fence/empty input can't trip set -e) used by both the deterministic and llm-judge paths, replacing the judge path's inline grep. Captures a 160-char single-line raw_excerpt only on parse failure so a future blind null-everywhere failure is diagnosable from eval-health #762 without the artifact. Confined to eval tooling (2 files, +74/-11); no production, auth, or data-path impact; comprehensive regression tests added. Triage's low-risk assessment confirmed.

Linked issue analysis

Closes #762 (skill eval health — triage holdout 0/5). Root cause is correctly diagnosed: score_deterministic() parsed engine output strictly as bare JSON while live Haiku-tier run_triage fences its decision, yielding null for every case; offline bats passed because the stub emitted bare JSON. The fix targets the parser rather than prompts/triage.md (whose bare-JSON contract is already correct), which is the right call. Final verification (eval score > 0) is necessarily post-merge.

Findings

No blocking findings.

  • extract_json correctly strips optional-whitespace-prefixed fence lines; || true guards the grep-no-match exit-1 under set -e.
  • Switch to jq -cse with length==1 is sound: slurped empty/multi-value/prose input falls through to the empty branch -> null actual -> fail, preserving the strict-failure semantics for genuinely malformed output.
  • raw_excerpt uses the original raw (not cleaned), newline-flattened and truncated to 160 chars, and is attached only when actual=null — passing/mismatch cases keep the lean shape, asserted by tests.
  • Test coverage is strong: new fenced-JSON stub reproduces the live 0/5 regression; assertions confirm prose still fails (with excerpt) and passing cases carry no raw_excerpt.
    Note: GitHub Secret Protection MCP scan unavailable (Advanced Security not enabled); the gitleaks CI check passed and the diff contains only fake test-stub JSON — no secrets.

CI status

All substantive checks green: ShellCheck, bats/unit-tests, CodeQL, Agent Security Scan, agent-shield, gitleaks, SonarCloud, holdout-guard, Lint, validate-agent-profiles, gh-aw-compile. dependabot/dependency-audit jobs SKIPPED (no matching ecosystems); dev-lead relay CANCELLED (review orchestration). CodeRabbit/Codex external reviews were rate-limited (advisory only, non-blocking). mergeable=MERGEABLE; mergeStateStatus=BLOCKED only on required human review.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

@sonarqubecloud

Copy link
Copy Markdown

@don-petry
don-petry merged commit 6754772 into main Jun 20, 2026
31 checks passed
@don-petry
don-petry deleted the claude/fix-eval-scorer-fenced-json branch June 20, 2026 01:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Skill eval health — triage (holdout)

3 participants