Repository navigation
Gate the re-grade on corpus identity: ids and text cannot detect a redraw - #213
Conversation
…draw AgentEval redraws corpora keeping the question_id set 100% IDENTICAL with ZERO byte-identical items, and neither corpus_id nor revision moves either -- corpus_sha256 is the only distinguishing field. For bitemporal, 27 of 60 items keep the SAME question text and carry a DIFFERENT gold. That defeats the protection the re-grade already had. The replay is position-keyed and verifies the question TEXT at each position, which catches a redraw that reworded anything and catches NOTHING when the text is stable and only the gold moved. Re-grading across such a redraw would replay the right answers against the wrong keys and report an agreement rate that means nothing -- after paying for the judge pass. So the corpus sha is now compared BEFORE any judge call. The stored value comes from the artifact's own TypedOutcomes.CorpusSha256; the current one is hashed from the embedded resource this build actually carries, rather than read from a manifest, so it is the bytes the run would use and not a claim about them. Three outcomes, deliberately distinct: a match proceeds; a MISMATCH ABORTS and names both shas; and an artifact carrying no sha at all WARNS rather than failing -- artifacts predating provenance capture are still re-gradable, but it is the one case where nothing can be verified, so it says so instead of passing quietly. Verified both directions: a matching artifact proceeds, and an artifact whose sha was moved to the current bitemporal lineage (abf2f3f4) aborts with both shas named. Also documents the adapter's blindness as a test rather than pretending it away: identical text passes the positional check no matter what the gold now says, which is precisely why corpus identity must be established one layer up. Release 0-warn; LongMemEval 734/734. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
There was a problem hiding this comment.
🟡 Changes recommended
The new gate can proceed silently when the current corpus SHA can’t be resolved and can throw on malformed/short SHA strings, undermining the intended “verify-before-spend” behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR adds a corpus-identity gate to the TypedMemEval re-grade tool to prevent re-grading artifacts against a different (redrawn) corpus with identical question ids/text, and adds a unit test documenting the replay adapter’s positional-blindness to “gold moved behind unchanged text”.
Changes:
- Abort re-grade early when the artifact’s stored
TypedOutcomes.CorpusSha256doesn’t match the corpus bytes embedded in the current build. - Warn (non-fatal) when an artifact has no recorded corpus SHA (pre-provenance artifacts).
- Add a unit test that documents the replay adapter’s inability to detect moved gold when question text is unchanged.
File summaries
| File | Description |
|---|---|
| tools/AgentMemory.LongMemEval/TypedMemEvalRegradeProgram.cs | Adds the corpus SHA gate and computes current corpus SHA from embedded resources. |
| tests/AgentMemory.Tests.Unit.LongMemEval/TypedMemEvalReplayAdapterTests.cs | Adds a unit test documenting a known limitation of position-keyed replay. |
Review details
Suppressed comments (1)
tools/AgentMemory.LongMemEval/TypedMemEvalRegradeProgram.cs:140
- If
CurrentCorpusShareturns null (resource not found or ambiguous), the code currently proceeds silently even though the doc comment says this should be treated as "cannot verify". Also, slicingstoredCorpusSha[..16]can throw if the stored value is shorter than 16 chars. Add an explicit warning branch for the null case and make the abort message length-safe.
else if (currentCorpusSha is not null &&
!string.Equals(storedCorpusSha, currentCorpusSha, StringComparison.OrdinalIgnoreCase))
{
Console.Error.WriteLine(
$"regrade: ABORT — corpus mismatch. The artifact was produced against " +
$"{storedCorpusSha[..16]}… and this build carries {currentCorpusSha[..16]}…. " +
"Question ids and text are NOT sufficient to detect this: corpora are redrawn " +
"keeping ids identical, and gold can move behind unchanged text. Re-grade with " +
"the package the artifact was produced against.");
return 5;
- Files reviewed: 2/2 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…ash once All three from Copilot's review of #213, and all three real. A non-string CorpusSha256 -- null, a number, an object on a malformed or older artifact -- would have thrown out of GetString() and killed the tool. This gate's entire job is to fail SAFELY before anything is spent, so it now checks ValueKind and treats anything that is not a non-empty string as "cannot verify", which takes the warn path rather than the crash path. Verified with a null. FirstOrDefault over manifest resource names resolved an ambiguity by manifest ordering, which is not guaranteed. A gate that picks a different corpus on a different run is WORSE than no gate, because it looks like it verified something. It now requires exactly one match and returns null otherwise. And the hash is taken straight off the resource stream instead of buffering into a MemoryStream and copying again via ToArray -- the corpora run to ~1.5 MB and that was two full copies to compute a digest. Verified after the change: a moved sha still ABORTS with both shas named; a null sha WARNS and proceeds. LongMemEval 734/734. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
Copilot's review of #214: the mismatch message sliced both shas at a fixed [..16], which throws ArgumentOutOfRangeException on a truncated or corrupt value -- in the one branch whose entire purpose is to fail SAFELY with an exit code rather than an exception. Second time on this same gate. #213 hardened the sha's TYPE (a non-string CorpusSha256 would have thrown out of GetString()) and left its LENGTH assumed. Same defensive question, same guard, asked about one property and not the next. A guard that can throw is not a guard. Verified end to end with a truncated sha: it now aborts cleanly and prints the short value as-is rather than crashing. Release 0-warn; LongMemEval 744/744. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo
#214) * Make the corpus-identity gate a tested unit, and close the id-keying audit Discharges the family-wide audit: every cache, baseline, join and tool of ours that could key on question_id/corpus_id/revision across corpus versions. RESULT: one gap, in one surface, already closed in #213 -- the --regrade cross-artifact join, whose position-and-text keying catches a reworded redraw and catches nothing when the text is stable and only the gold moves. Everything else is clean, and two of those are worth recording rather than merely ticking: - PREPARED-CORPUS REUSE already compares datasetSha256 alongside eleven identity fields, and its own header states the principle this audit is about -- "treating unknown as equal is how a check stops being able to fail". It reports unrecorded fields on older manifests as DRIFT rather than agreement. That is the standard the re-grade has now been brought up to. - No other verb reads two artifacts at all, so --regrade was the only place the hazard could live. WHAT THIS COMMIT ADDS: the gate's decision was a code path only exercisable by running the tool against a hand-mutated artifact. It is now CorpusIdentity.Verify with three verdicts and unit tests for each, including hex-casing (artifacts and manifests disagree about it, and that is not a redraw). Unverifiable stays its own verdict deliberately. Folded into Match it passes silently -- the constant-column failure this project has been bitten by three times. Folded into Mismatch it blocks every artifact older than provenance capture, making the gate obstructive enough to get bypassed. So it warns and refuses to be read as agreement. Release 0-warn; LongMemEval 742/742. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo * Stop the abort path assuming its sha is well-formed Copilot's review of #214: the mismatch message sliced both shas at a fixed [..16], which throws ArgumentOutOfRangeException on a truncated or corrupt value -- in the one branch whose entire purpose is to fail SAFELY with an exit code rather than an exception. Second time on this same gate. #213 hardened the sha's TYPE (a non-string CorpusSha256 would have thrown out of GetString()) and left its LENGTH assumed. Same defensive question, same guard, asked about one property and not the next. A guard that can throw is not a guard. Verified end to end with a truncated sha: it now aborts cleanly and prints the short value as-is rather than crashing. Release 0-warn; LongMemEval 744/744. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RM9gMsvo3SpYqEBENbaefo --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ired Bitemporal lands in this chain in a few hours, and the scoreboard would have placed it as a percentage beside the other nine. Every arm this project has ever run recorded zero :SUPERSEDED_BY edges, so that number would measure the ABSENCE of the mechanism its shapes ask about -- the one claim held here as absolutely forbidden, reintroduced by a tool that assembles itself. OFF-STATE is its own row state, distinct from VOID. Nothing is missing: the questions were asked and answered. What is missing is the feature, so there is no score to report rather than a low one. The gate reads `is not > 0`, not `is not 0`. An unrecorded probe is UNKNOWN, and unknown placed as a number makes the same claim as zero placed as a number. A test pins both the zero case and the unrecorded case, and a third pins that supersession-dependence is a named list of one: semantic and the rest also run against a store with zero edges, and voiding them would turn a real constraint into noise, which is how a gate stops being read. Writing that test also found a guard that could throw. TryGetInt32 and TryGetDateTimeOffset THROW on a JSON null rather than returning false, and the catch here covers JsonException only -- so a sidecar with a null edge count took the tool down instead of making one run unplaceable. Every accessor in the file is now ValueKind-checked before the read, GetString included. Same lesson as #213 on the re-grade gate: a guard that can throw is not a guard. 812 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju
…t carry a number (#217) * Assemble the ten-vertical scoreboard, and gate it on the corpus sha Ten verticals land over days from separate runs. The only things that make two of them comparable are the corpus they were drawn from, the judge that graded them and the arm they ran on -- and a table typed by hand records the numbers while dropping exactly those fields. Three columns, none sufficient alone: share-of-all correct over scored, the number we have always reported share-of-reachable over the corpus's own ceiling at k_ref, since three of ten corpora cap below 1.0 ranking-only restricted to shapes that can separate systems under a DENSE retriever, which is what this stack uses Columns 2 and 3 are deliberately not composed: the ceiling is declared over a whole corpus, so applying it to a shape-restricted numerator would divide a subset by a whole. A test pins that. The third column changes three readings that are already in the ledger. Episodic 72.0% is 53.3% over the two shapes that rank -- its assistant-stated 20/20 cannot separate systems. Forgetting 88.0% becomes 100.0% on a SINGLE ranking shape at n=15, which is a warning and not a triumph. LINEAGE IS THE CORPUS SHA, NEVER THE PACKAGE VERSION. The first cut gated on the release and read procedural, temporal and workingmemory as absent -- three finished verticals whose corpora are byte-identical across 0.34-0.36, which it would have sent back for an ~11h re-run that could not have changed one answer. The sha check reuses the re-grade's gate rather than growing a second one, so CurrentCorpusSha moves to a shared helper both call. A one-question stage-2 probe is not a row. Every full run here is preceded by one, in the same directory, on the same arm, against the same corpus -- and it stays the NEWEST artifact for the hours the real run takes. Without the partial-run filter the board reported prospective as 0.0% from a sample of one. States, never zeros: NOT PLACED (no run, stale draw, or unverifiable sha) and VOID (agent failures -- the 7/39 lesson) are distinct from a score, and a vertical whose measured shapes all fail to rank says NOT RANKABLE rather than 0%. 808 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju * Refuse to score bitemporal against a store where supersession never fired Bitemporal lands in this chain in a few hours, and the scoreboard would have placed it as a percentage beside the other nine. Every arm this project has ever run recorded zero :SUPERSEDED_BY edges, so that number would measure the ABSENCE of the mechanism its shapes ask about -- the one claim held here as absolutely forbidden, reintroduced by a tool that assembles itself. OFF-STATE is its own row state, distinct from VOID. Nothing is missing: the questions were asked and answered. What is missing is the feature, so there is no score to report rather than a low one. The gate reads `is not > 0`, not `is not 0`. An unrecorded probe is UNKNOWN, and unknown placed as a number makes the same claim as zero placed as a number. A test pins both the zero case and the unrecorded case, and a third pins that supersession-dependence is a named list of one: semantic and the rest also run against a store with zero edges, and voiding them would turn a real constraint into noise, which is how a gate stops being read. Writing that test also found a guard that could throw. TryGetInt32 and TryGetDateTimeOffset THROW on a JSON null rather than returning false, and the catch here covers JsonException only -- so a sidecar with a null edge count took the tool down instead of making one run unplaceable. Every accessor in the file is now ValueKind-checked before the read, GetString included. Same lesson as #213 on the re-grade gate: a guard that can throw is not a guard. 812 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju * Strip two byte-order marks this branch added to src files Both files arrived on main without a BOM and leave this branch with one, from an editor round-trip in an earlier commit here. Nothing in the diff but the first three bytes, which is what makes it worth removing: a BOM change shows up as the whole first line rewritten in every future blame and diff of a file nobody meant to touch that way. Second time on this branch. The cause is writing a file back with a utf-8-sig encoder, which ADDS a mark rather than preserving whichever the file had. Release 0-warn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju * Withhold share-of-reachable on a partial run instead of dividing by the corpus The ceiling is declared over a whole corpus, so the score line divided this run's correct count by the corpus denominator. On the one-question stage-2 probe that precedes every full run, it printed "Score 1/65 = 1.6% OF REACHABLE" -- a sample of one presented against 65. The ceiling header still prints: it is a property of the corpus and true whatever was sampled. Only the score line is withheld, and it says why rather than going quiet. Same defect the scoreboard had two commits ago, in the other direction -- there a stage-2 probe became a row, here it became a percentage. Both come from treating a sample as a measurement of the vertical. Release 0-warn, 812 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
From the SHA-only keying audit.
AgentEval redraws corpora keeping the
question_idset 100% identical with zero byte-identicalitems, and neither
corpus_idnorrevisionmoves —corpus_sha256is the only distinguishingfield. For bitemporal, 27 of 60 items keep the same question text with a different gold.
That defeats the protection the re-grade already had. The replay is position-keyed and verifies
question text per position: it catches a redraw that reworded anything, and catches nothing
when the text is stable and only the gold moved. Re-grading across such a redraw replays the right
answers against the wrong keys and reports an agreement rate that means nothing — after paying for
the judge pass.
The gate
Corpus sha compared before any judge call. Stored value from the artifact's own
TypedOutcomes.CorpusSha256; current value hashed from the embedded resource this build carries,rather than read from a manifest — the bytes the run would use, not a claim about them.
Three outcomes, deliberately distinct:
re-gradable, but it is the one case where nothing can be verified, so it says so rather than
passing quietly
Verified both ways: a matching artifact proceeds; an artifact whose sha is moved to the current
bitemporal lineage (
abf2f3f4) aborts with both shas named.Also documents the adapter's blindness as a test rather than pretending it away — identical text
passes the positional check whatever the gold now says, which is exactly why identity belongs one
layer up.
Release 0-warn; 734/734.