Repository navigation
Assemble the ten-vertical scoreboard, and refuse the cells that cannot carry a number - #217
Conversation
Ten verticals land over days from separate runs. The only things that make
two of them comparable are the corpus they were drawn from, the judge that
graded them and the arm they ran on -- and a table typed by hand records the
numbers while dropping exactly those fields.
Three columns, none sufficient alone:
share-of-all correct over scored, the number we have always reported
share-of-reachable over the corpus's own ceiling at k_ref, since three of
ten corpora cap below 1.0
ranking-only restricted to shapes that can separate systems under a
DENSE retriever, which is what this stack uses
Columns 2 and 3 are deliberately not composed: the ceiling is declared over a
whole corpus, so applying it to a shape-restricted numerator would divide a
subset by a whole. A test pins that.
The third column changes three readings that are already in the ledger.
Episodic 72.0% is 53.3% over the two shapes that rank -- its assistant-stated
20/20 cannot separate systems. Forgetting 88.0% becomes 100.0% on a SINGLE
ranking shape at n=15, which is a warning and not a triumph.
LINEAGE IS THE CORPUS SHA, NEVER THE PACKAGE VERSION. The first cut gated on
the release and read procedural, temporal and workingmemory as absent -- three
finished verticals whose corpora are byte-identical across 0.34-0.36, which it
would have sent back for an ~11h re-run that could not have changed one answer.
The sha check reuses the re-grade's gate rather than growing a second one, so
CurrentCorpusSha moves to a shared helper both call.
A one-question stage-2 probe is not a row. Every full run here is preceded by
one, in the same directory, on the same arm, against the same corpus -- and it
stays the NEWEST artifact for the hours the real run takes. Without the
partial-run filter the board reported prospective as 0.0% from a sample of one.
States, never zeros: NOT PLACED (no run, stale draw, or unverifiable sha) and
VOID (agent failures -- the 7/39 lesson) are distinct from a score, and a
vertical whose measured shapes all fail to rank says NOT RANKABLE rather than
0%.
808 tests green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju
…ired Bitemporal lands in this chain in a few hours, and the scoreboard would have placed it as a percentage beside the other nine. Every arm this project has ever run recorded zero :SUPERSEDED_BY edges, so that number would measure the ABSENCE of the mechanism its shapes ask about -- the one claim held here as absolutely forbidden, reintroduced by a tool that assembles itself. OFF-STATE is its own row state, distinct from VOID. Nothing is missing: the questions were asked and answered. What is missing is the feature, so there is no score to report rather than a low one. The gate reads `is not > 0`, not `is not 0`. An unrecorded probe is UNKNOWN, and unknown placed as a number makes the same claim as zero placed as a number. A test pins both the zero case and the unrecorded case, and a third pins that supersession-dependence is a named list of one: semantic and the rest also run against a store with zero edges, and voiding them would turn a real constraint into noise, which is how a gate stops being read. Writing that test also found a guard that could throw. TryGetInt32 and TryGetDateTimeOffset THROW on a JSON null rather than returning false, and the catch here covers JsonException only -- so a sidecar with a null edge count took the tool down instead of making one run unplaceable. Every accessor in the file is now ValueKind-checked before the read, GetString included. Same lesson as #213 on the re-grade gate: a guard that can throw is not a guard. 812 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju
Both files arrived on main without a BOM and leave this branch with one, from an editor round-trip in an earlier commit here. Nothing in the diff but the first three bytes, which is what makes it worth removing: a BOM change shows up as the whole first line rewritten in every future blame and diff of a file nobody meant to touch that way. Second time on this branch. The cause is writing a file back with a utf-8-sig encoder, which ADDS a mark rather than preserving whichever the file had. Release 0-warn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju
…he corpus The ceiling is declared over a whole corpus, so the score line divided this run's correct count by the corpus denominator. On the one-question stage-2 probe that precedes every full run, it printed "Score 1/65 = 1.6% OF REACHABLE" -- a sample of one presented against 65. The ceiling header still prints: it is a property of the corpus and true whatever was sampled. Only the score line is withheld, and it says why rather than going quiet. Same defect the scoreboard had two commits ago, in the other direction -- there a stage-2 probe became a row, here it became a percentage. Both come from treating a sample as a measurement of the vertical. Release 0-warn, 812 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju
5942473 to
f476381
Compare
There was a problem hiding this comment.
🟡 Changes recommended
One or more issues must be addressed before approval.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds AgentEval 0.36 measurement instruments and an artifact-based ten-vertical scoreboard, plus opt-in fact-to-entity linking and improved forgetting diagnostics.
Changes:
- Adds reachable-ceiling, chance-floor, retriever-sensitivity, and corpus-lineage handling.
- Adds scoreboard CLI with artifact filtering and state classification.
- Adds opt-in
:ABOUTlinking and related tests.
File summaries
| File | Description |
|---|---|
| tools/AgentMemory.LongMemEval/TypedMemEvalScoreboardProgram.cs | Updated as part of this pull request. |
| tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs | Updated as part of this pull request. |
| tools/AgentMemory.LongMemEval/TypedMemEvalRetrieverSensitivity.cs | Updated as part of this pull request. |
| tools/AgentMemory.LongMemEval/TypedMemEvalRegradeProgram.cs | Updated as part of this pull request. |
| tools/AgentMemory.LongMemEval/TypedMemEvalReachableCeiling.cs | Updated as part of this pull request. |
| tools/AgentMemory.LongMemEval/TypedMemEvalProgram.cs | Updated as part of this pull request. |
| tools/AgentMemory.LongMemEval/TypedMemEvalFloorReport.cs | Updated as part of this pull request. |
| tools/AgentMemory.LongMemEval/TypedMemEvalCorpusSha.cs | Updated as part of this pull request. |
| tools/AgentMemory.LongMemEval/Program.cs | Updated as part of this pull request. |
| tools/AgentMemory.LongMemEval/AgentMemory.LongMemEval.csproj | Updated as part of this pull request. |
| tests/AgentMemory.Tests.Unit/Extraction/FactEntityLinkingTests.cs | Updated as part of this pull request. |
| tests/AgentMemory.Tests.Unit.LongMemEval/ScoreboardTests.cs | Updated as part of this pull request. |
| tests/AgentMemory.Tests.Unit.LongMemEval/RetrieverSensitivityTests.cs | Updated as part of this pull request. |
| tests/AgentMemory.Tests.Unit.LongMemEval/ReachableCeilingTests.cs | Updated as part of this pull request. |
| tests/AgentMemory.Tests.Unit.LongMemEval/FloorReportTests.cs | Updated as part of this pull request. |
| src/AgentMemory.Core/Services/Projection/ConflictProjectionFeature.cs | Updated as part of this pull request. |
| src/AgentMemory.Core/Services/MemoryContextAssembler.cs | Updated as part of this pull request. |
| src/AgentMemory.Core/Extraction/PersistenceStage.cs | Updated as part of this pull request. |
| src/AgentMemory.Abstractions/Options/ExtractionOptions.cs | Updated as part of this pull request. |
Review details
Suppressed comments (9)
src/AgentMemory.Abstractions/Options/ExtractionOptions.cs:130
- This public option's explanatory baseline repeats
alias-then-countas 0/15, while the PR description reports 1/15. Align the user-facing option documentation with the authoritative artifact so consumers do not receive two different effectiveness claims.
/// <b>Why it matters, measured.</b> Three shapes across three verticals need the same operation —
/// resolve a referent to an entity — and their scores fall as the surface forms diverge:
/// semantic <c>co-reference</c> 11/15, episodic <c>participant-attribution</c> 4/15, conjunction
/// <c>alias-then-count</c> <b>0/15</b>. The last is immune to more evidence: three arms at 4.7x
/// the facts moved it by one question. <b>You cannot count across an identity the store never
src/AgentMemory.Abstractions/Options/ExtractionOptions.cs:142
LinkFactsToEntitiesis a new public configuration capability, but the user-facing configuration and memory-map documentation do not describe how to enable it or its exact-name-only limitation. Add the option, default-off behavior, and alias limitation to the appropriate docs and Unreleased changelog.
/// </para>
/// </remarks>
public bool LinkFactsToEntities { get; set; }
tools/AgentMemory.LongMemEval/Program.cs:159
--scoreboardis a new operator-facing evaluation command, but it is absent from the established evaluation-instrument inventory indocs/architecture.mdand from the Unreleased changelog. Document the command and its NOT PLACED/VOID/OFF-STATE/NOT RANKABLE output contract in the same change so users do not treat the table as an ordinary accuracy report.
if (args.Contains("--scoreboard", StringComparer.Ordinal))
{
// Reads stored artifacts only -- no store, no judge, no spend. Checked before the verbs
// that name a vertical because the scoreboard names all ten, and an absent one is a row
// it has to print rather than an argument it could take.
tools/AgentMemory.LongMemEval/TypedMemEvalProgram.cs:296
- These new floor results are printed only after
Persistand are not added to the report JSON or provenance sidecar. If console output is not retained, the archived artifact loses the per-shape floor, census, p-value, and underpowered verdict that this instrument is meant to make durable; persist the derived block alongside the run before relying on it for later audit.
PrintReachableCeiling(descriptor, result);
TypedMemEvalRetrieverSensitivity.Print(descriptor);
TypedMemEvalFloorReport.Print(result, options.Arm.FileToken());
tools/AgentMemory.LongMemEval/TypedMemEvalProgram.cs:626
- The printed fraction uses
CorrectQuestions/Questions, butShareOfReachabledivides byReachableQuestions. On a capped corpus this produces a denominator in the text that does not match the percentage, so the console measurement is misleading. Print the reachable denominator here (or label the whole-corpus fraction separately).
$"{header} Score {result.CorrectQuestions}/{ceiling.Questions} = "
+ $"{ceiling.ShareOfReachable(result.CorrectQuestions):P1} OF REACHABLE."));
tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs:143
- The sidecar records
sampling.oracleandsampling.control, but this reader filters only on the feature arm token. Oracle runs and Prospective control runs therefore share thedefaulttoken and can become the newest selected row, causing gold-context or control scores to be presented as the default agent measurement. Exclude those run kinds before choosing the latest artifact (and test both cases).
if (!sidecar.TryGetProperty("arm", out var arm) ||
!arm.TryGetProperty("token", out var token) ||
token.ValueKind != JsonValueKind.String ||
!string.Equals(token.GetString(), armToken, StringComparison.Ordinal))
{
return null;
}
tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs:185
- The malformed-artifact path has the same gap here:
ReadcallsTryGetPropertyon report/shape values, which throwsInvalidOperationExceptionwhen a JSON value is null or the wrong kind, but this filter catches only parse/IO exceptions. The scoreboard then crashes instead of treating the artifact as unplaceable, contrary to the surrounding comment.
catch (Exception exception) when (exception is JsonException or IOException or UnauthorizedAccessException)
{
// An unreadable artifact must not take the scoreboard down, and must not be silently
// counted either: it simply never becomes a placeable run, so its vertical reads ABSENT.
return null;
tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs:220
Intsilently converts missing, null, or non-numeric fields to zero. A truncated report with a validScoredQuestionsbut noCorrectQuestionsis therefore placed as a legitimate 0% row (and malformed shape tallies can likewise affect ranking), contrary to the stated unreadable/not-placeable behavior. Required measurement fields should makeReadreject the artifact instead of defaulting them.
SelectedQuestions: Int(report, "SelectedQuestions"),
CorpusQuestions: Int(provenance, "DatasetQuestionCount"),
ScoredQuestions: Int(report, "ScoredQuestions"),
CorrectQuestions: Int(report, "CorrectQuestions"),
AgentFailureQuestions: Int(report, "AgentFailureQuestions"),
tools/AgentMemory.LongMemEval/TypedMemEvalScoreboardProgram.cs:42
- This helper silently falls back to the default when
--artifactsor--armis the final argument, so a malformed invocation can print a different scoreboard than the one requested and still exit successfully. Match the other command-line parsers by throwing when the switch is present without a value.
private static string? Value(string[] args, string name)
{
var index = Array.IndexOf(args, name);
return index >= 0 && index + 1 < args.Length ? args[index + 1] : null;
}
- Files reviewed: 9/9 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // directory, on the same arm, against the same corpus, minutes before the real one. It is | ||
| // also the NEWEST artifact for hours afterwards. Left in, it would have reported | ||
| // prospective as 0.0% -- a sample of one presented as a vertical. | ||
| if (run.CorpusQuestions > 0 && run.SelectedQuestions < run.CorpusQuestions) continue; |
| return scored <= 0 | ||
| ? RankingOnlyScore.NotRankable(nonRanking, unclassified) |
| if (judges.Length > 1) | ||
| { | ||
| refusals.Add( | ||
| $"{judges.Length} distinct judge prompt fingerprints across placed rows — " | ||
| + "these cells were graded by different judges and cannot be ranked against each other"); | ||
| } |
| /// <summary>A release without its 40-character build metadata, which no reader needs inline.</summary> | ||
| private static string Build(string? version) | ||
| { | ||
| if (version is null) return "(unrecorded)"; | ||
| var plus = version.IndexOf('+', StringComparison.Ordinal); | ||
| return plus < 0 ? version : $"{version[..plus]}+{Short(version[(plus + 1)..])}"; | ||
| } |
Follows #216, which shipped the instruments (reachable ceiling, chance floors, the dense filter, fact→entity linking, legible-forgetting observability). This is the thing that reads them: the ten-vertical scoreboard, plus three defects it caught in itself.
The scoreboard
Ten verticals land over days from separate runs. The only things that make two of them comparable are the corpus they were drawn from, the judge that graded them and the arm they ran on — and a table typed by hand records the numbers while dropping exactly those fields.
--scoreboardreads them back out of the artifacts and refuses to assemble when they disagree. No store, no judge, no spend.Three columns, none sufficient alone:
k_ref, since three of ten corpora cap below 1.0Columns 2 and 3 are deliberately not composed: the ceiling is declared over a whole corpus, so applying it to a shape-restricted numerator would divide a subset by a whole. A test pins that.
States, never zeros.
NOT PLACED(no run, stale corpus draw, or unverifiable sha),VOID(agent failures — the 7/39 lesson),OFF-STATE, andNOT RANKABLEare each distinct from a score. A vertical whose measured shapes all fail to rank says so rather than reporting 0%.What the third column does to results already in the ledger
None of these is a new measurement. Prospective's is the widest gap: 52 of its 54 non-ranking questions are correct, while its one informative shape scores 1/18 against dense headroom 1.000.
Three defects it caught in itself
Lineage is the corpus sha, not the package version. The first gate read procedural, temporal and workingmemory as absent — three finished verticals whose corpora are byte-identical across 0.34–0.36 — an ~11h re-run that could not have changed one answer. The standing rule is compare corpora on sha, and that check already existed in the re-grade gate;
CurrentCorpusShamoves to a shared helper both call rather than a second rule that can drift.A one-question stage-2 probe is not a row. Every full run here is preceded by one, in the same directory, on the same arm, against the same corpus — and it stays the newest artifact for the hours the real run takes. Without the partial-run filter the board reported prospective as 0.0% from a sample of one. The same defect existed in the other direction in the console output, where the ceiling line divided a one-question probe's correct count by the whole corpus and printed "Score 1/65 = 1.6% OF REACHABLE"; that line is now withheld on a partial run and says why.
Bitemporal must never be scored here. Every arm this project has run records zero
:SUPERSEDED_BYedges, so a bitemporal number measures the absence of the mechanism its shapes ask about.OFF-STATEis its own row state, distinct fromVOID— nothing is missing, the questions were asked and answered; what is missing is the feature. The gate readsis not > 0, notis not 0: an unrecorded probe is unknown, and unknown placed as a number makes the same claim as zero. Supersession-dependence is a named list of one, because every vertical runs against a store with zero edges and voiding them all would turn a real constraint into noise.This landed hours before bitemporal finished, and fired on it automatically.
A guard that could throw
Writing that test found one.
TryGetInt32andTryGetDateTimeOffsetthrow on a JSON null rather than returning false, and the catch coveredJsonExceptiononly — so a sidecar with a null edge count took the tool down instead of making one run unplaceable. Every accessor in the file is now ValueKind-checked before the read,GetStringincluded. Same lesson as #213 on the re-grade gate: a guard that can throw is not a guard.Also: two stray byte-order marks
ExtractionOptions.csandConflictProjectionFeature.csarrived without a BOM and gained one from an editor round-trip. Nothing in the diff but the first three bytes — which is what makes it worth removing, since a BOM change shows as the whole first line rewritten in every future blame of a file nobody meant to touch that way.Verification
Release 0 warnings. 6,075 unit tests green (5,209 + 812 + 54).
🤖 Generated with Claude Code
https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju