Skip to content

Assemble the ten-vertical scoreboard, and refuse the cells that cannot carry a number - #217

Merged
joslat merged 4 commits into
mainfrom
feat/cd-035-floors-ceiling
Sep 15, 2026
Merged

joslat merged 4 commits into
mainfrom
feat/cd-035-floors-ceiling

Conversation

@joslat

@joslat joslat commented Sep 15, 2026 •

Copy link
Copy Markdown
Owner

Follows #216, which shipped the instruments (reachable ceiling, chance floors, the dense filter, fact→entity linking, legible-forgetting observability). This is the thing that reads them: the ten-vertical scoreboard, plus three defects it caught in itself.

The scoreboard

Ten verticals land over days from separate runs. The only things that make two of them comparable are the corpus they were drawn from, the judge that graded them and the arm they ran on — and a table typed by hand records the numbers while dropping exactly those fields. --scoreboard reads them back out of the artifacts and refuses to assemble when they disagree. No store, no judge, no spend.

Three columns, none sufficient alone:

column what it is
share-of-all correct over scored — the number every previous artifact reported
share-of-reachable over the corpus's own ceiling at k_ref, since three of ten corpora cap below 1.0
ranking-only restricted to shapes that can separate systems under a dense retriever, which is what this stack uses

Columns 2 and 3 are deliberately not composed: the ceiling is declared over a whole corpus, so applying it to a shape-restricted numerator would divide a subset by a whole. A test pins that.

States, never zeros. NOT PLACED (no run, stale corpus draw, or unverifiable sha), VOID (agent failures — the 7/39 lesson), OFF-STATE, and NOT RANKABLE are each distinct from a score. A vertical whose measured shapes all fail to rank says so rather than reporting 0%.

What the third column does to results already in the ledger

vertical share-of-all ranking-only
prospective 73.6 % 5.6 % (1/18)
episodic 72.0 % 53.3 %
forgetting 88.0 % 100.0 % — on a single shape at n=15

None of these is a new measurement. Prospective's is the widest gap: 52 of its 54 non-ranking questions are correct, while its one informative shape scores 1/18 against dense headroom 1.000.

Three defects it caught in itself

Lineage is the corpus sha, not the package version. The first gate read procedural, temporal and workingmemory as absent — three finished verticals whose corpora are byte-identical across 0.34–0.36 — an ~11h re-run that could not have changed one answer. The standing rule is compare corpora on sha, and that check already existed in the re-grade gate; CurrentCorpusSha moves to a shared helper both call rather than a second rule that can drift.

A one-question stage-2 probe is not a row. Every full run here is preceded by one, in the same directory, on the same arm, against the same corpus — and it stays the newest artifact for the hours the real run takes. Without the partial-run filter the board reported prospective as 0.0% from a sample of one. The same defect existed in the other direction in the console output, where the ceiling line divided a one-question probe's correct count by the whole corpus and printed "Score 1/65 = 1.6% OF REACHABLE"; that line is now withheld on a partial run and says why.

Bitemporal must never be scored here. Every arm this project has run records zero :SUPERSEDED_BY edges, so a bitemporal number measures the absence of the mechanism its shapes ask about. OFF-STATE is its own row state, distinct from VOID — nothing is missing, the questions were asked and answered; what is missing is the feature. The gate reads is not > 0, not is not 0: an unrecorded probe is unknown, and unknown placed as a number makes the same claim as zero. Supersession-dependence is a named list of one, because every vertical runs against a store with zero edges and voiding them all would turn a real constraint into noise.

This landed hours before bitemporal finished, and fired on it automatically.

A guard that could throw

Writing that test found one. TryGetInt32 and TryGetDateTimeOffset throw on a JSON null rather than returning false, and the catch covered JsonException only — so a sidecar with a null edge count took the tool down instead of making one run unplaceable. Every accessor in the file is now ValueKind-checked before the read, GetString included. Same lesson as #213 on the re-grade gate: a guard that can throw is not a guard.

Also: two stray byte-order marks

ExtractionOptions.cs and ConflictProjectionFeature.cs arrived without a BOM and gained one from an editor round-trip. Nothing in the diff but the first three bytes — which is what makes it worth removing, since a BOM change shows as the whole first line rewritten in every future blame of a file nobody meant to touch that way.

Verification

Release 0 warnings. 6,075 unit tests green (5,209 + 812 + 54).

🤖 Generated with Claude Code

https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju

Copilot AI lite review requested due to automatic review settings September 15, 2026 00:59
joslat and others added 4 commits September 15, 2026 03:00
Ten verticals land over days from separate runs. The only things that make
two of them comparable are the corpus they were drawn from, the judge that
graded them and the arm they ran on -- and a table typed by hand records the
numbers while dropping exactly those fields.

Three columns, none sufficient alone:

  share-of-all         correct over scored, the number we have always reported
  share-of-reachable   over the corpus's own ceiling at k_ref, since three of
                       ten corpora cap below 1.0
  ranking-only         restricted to shapes that can separate systems under a
                       DENSE retriever, which is what this stack uses

Columns 2 and 3 are deliberately not composed: the ceiling is declared over a
whole corpus, so applying it to a shape-restricted numerator would divide a
subset by a whole. A test pins that.

The third column changes three readings that are already in the ledger.
Episodic 72.0% is 53.3% over the two shapes that rank -- its assistant-stated
20/20 cannot separate systems. Forgetting 88.0% becomes 100.0% on a SINGLE
ranking shape at n=15, which is a warning and not a triumph.

LINEAGE IS THE CORPUS SHA, NEVER THE PACKAGE VERSION. The first cut gated on
the release and read procedural, temporal and workingmemory as absent -- three
finished verticals whose corpora are byte-identical across 0.34-0.36, which it
would have sent back for an ~11h re-run that could not have changed one answer.
The sha check reuses the re-grade's gate rather than growing a second one, so
CurrentCorpusSha moves to a shared helper both call.

A one-question stage-2 probe is not a row. Every full run here is preceded by
one, in the same directory, on the same arm, against the same corpus -- and it
stays the NEWEST artifact for the hours the real run takes. Without the
partial-run filter the board reported prospective as 0.0% from a sample of one.

States, never zeros: NOT PLACED (no run, stale draw, or unverifiable sha) and
VOID (agent failures -- the 7/39 lesson) are distinct from a score, and a
vertical whose measured shapes all fail to rank says NOT RANKABLE rather than
0%.

808 tests green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju
…ired

Bitemporal lands in this chain in a few hours, and the scoreboard would have
placed it as a percentage beside the other nine. Every arm this project has
ever run recorded zero :SUPERSEDED_BY edges, so that number would measure the
ABSENCE of the mechanism its shapes ask about -- the one claim held here as
absolutely forbidden, reintroduced by a tool that assembles itself.

OFF-STATE is its own row state, distinct from VOID. Nothing is missing: the
questions were asked and answered. What is missing is the feature, so there is
no score to report rather than a low one.

The gate reads `is not > 0`, not `is not 0`. An unrecorded probe is UNKNOWN,
and unknown placed as a number makes the same claim as zero placed as a
number. A test pins both the zero case and the unrecorded case, and a third
pins that supersession-dependence is a named list of one: semantic and the
rest also run against a store with zero edges, and voiding them would turn a
real constraint into noise, which is how a gate stops being read.

Writing that test also found a guard that could throw. TryGetInt32 and
TryGetDateTimeOffset THROW on a JSON null rather than returning false, and the
catch here covers JsonException only -- so a sidecar with a null edge count
took the tool down instead of making one run unplaceable. Every accessor in
the file is now ValueKind-checked before the read, GetString included. Same
lesson as #213 on the re-grade gate: a guard that can throw is not a guard.

812 tests green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju
Both files arrived on main without a BOM and leave this branch with one, from
an editor round-trip in an earlier commit here. Nothing in the diff but the
first three bytes, which is what makes it worth removing: a BOM change shows
up as the whole first line rewritten in every future blame and diff of a file
nobody meant to touch that way.

Second time on this branch. The cause is writing a file back with a utf-8-sig
encoder, which ADDS a mark rather than preserving whichever the file had.

Release 0-warn.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju
…he corpus

The ceiling is declared over a whole corpus, so the score line divided this
run's correct count by the corpus denominator. On the one-question stage-2
probe that precedes every full run, it printed "Score 1/65 = 1.6% OF
REACHABLE" -- a sample of one presented against 65.

The ceiling header still prints: it is a property of the corpus and true
whatever was sampled. Only the score line is withheld, and it says why rather
than going quiet.

Same defect the scoreboard had two commits ago, in the other direction -- there
a stage-2 probe became a row, here it became a percentage. Both come from
treating a sample as a measurement of the vertical.

Release 0-warn, 812 tests green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju
@joslat
joslat force-pushed the feat/cd-035-floors-ceiling branch from 5942473 to f476381 Compare September 15, 2026 01:01
@joslat joslat changed the title Measurement instruments for the C-D scoreboard, and the scoreboard Assemble the ten-vertical scoreboard, and refuse the cells that cannot carry a number Sep 15, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds AgentEval 0.36 measurement instruments and an artifact-based ten-vertical scoreboard, plus opt-in fact-to-entity linking and improved forgetting diagnostics.

Changes:

  • Adds reachable-ceiling, chance-floor, retriever-sensitivity, and corpus-lineage handling.
  • Adds scoreboard CLI with artifact filtering and state classification.
  • Adds opt-in :ABOUT linking and related tests.
File summaries
File Description
tools/AgentMemory.LongMemEval/TypedMemEvalScoreboardProgram.cs Updated as part of this pull request.
tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs Updated as part of this pull request.
tools/AgentMemory.LongMemEval/TypedMemEvalRetrieverSensitivity.cs Updated as part of this pull request.
tools/AgentMemory.LongMemEval/TypedMemEvalRegradeProgram.cs Updated as part of this pull request.
tools/AgentMemory.LongMemEval/TypedMemEvalReachableCeiling.cs Updated as part of this pull request.
tools/AgentMemory.LongMemEval/TypedMemEvalProgram.cs Updated as part of this pull request.
tools/AgentMemory.LongMemEval/TypedMemEvalFloorReport.cs Updated as part of this pull request.
tools/AgentMemory.LongMemEval/TypedMemEvalCorpusSha.cs Updated as part of this pull request.
tools/AgentMemory.LongMemEval/Program.cs Updated as part of this pull request.
tools/AgentMemory.LongMemEval/AgentMemory.LongMemEval.csproj Updated as part of this pull request.
tests/AgentMemory.Tests.Unit/Extraction/FactEntityLinkingTests.cs Updated as part of this pull request.
tests/AgentMemory.Tests.Unit.LongMemEval/ScoreboardTests.cs Updated as part of this pull request.
tests/AgentMemory.Tests.Unit.LongMemEval/RetrieverSensitivityTests.cs Updated as part of this pull request.
tests/AgentMemory.Tests.Unit.LongMemEval/ReachableCeilingTests.cs Updated as part of this pull request.
tests/AgentMemory.Tests.Unit.LongMemEval/FloorReportTests.cs Updated as part of this pull request.
src/AgentMemory.Core/Services/Projection/ConflictProjectionFeature.cs Updated as part of this pull request.
src/AgentMemory.Core/Services/MemoryContextAssembler.cs Updated as part of this pull request.
src/AgentMemory.Core/Extraction/PersistenceStage.cs Updated as part of this pull request.
src/AgentMemory.Abstractions/Options/ExtractionOptions.cs Updated as part of this pull request.
Review details

Suppressed comments (9)

src/AgentMemory.Abstractions/Options/ExtractionOptions.cs:130

  • This public option's explanatory baseline repeats alias-then-count as 0/15, while the PR description reports 1/15. Align the user-facing option documentation with the authoritative artifact so consumers do not receive two different effectiveness claims.
    /// <b>Why it matters, measured.</b> Three shapes across three verticals need the same operation —
    /// resolve a referent to an entity — and their scores fall as the surface forms diverge:
    /// semantic <c>co-reference</c> 11/15, episodic <c>participant-attribution</c> 4/15, conjunction
    /// <c>alias-then-count</c> <b>0/15</b>. The last is immune to more evidence: three arms at 4.7x
    /// the facts moved it by one question. <b>You cannot count across an identity the store never

src/AgentMemory.Abstractions/Options/ExtractionOptions.cs:142

  • LinkFactsToEntities is a new public configuration capability, but the user-facing configuration and memory-map documentation do not describe how to enable it or its exact-name-only limitation. Add the option, default-off behavior, and alias limitation to the appropriate docs and Unreleased changelog.
    /// </para>
    /// </remarks>
    public bool LinkFactsToEntities { get; set; }

tools/AgentMemory.LongMemEval/Program.cs:159

  • --scoreboard is a new operator-facing evaluation command, but it is absent from the established evaluation-instrument inventory in docs/architecture.md and from the Unreleased changelog. Document the command and its NOT PLACED/VOID/OFF-STATE/NOT RANKABLE output contract in the same change so users do not treat the table as an ordinary accuracy report.
        if (args.Contains("--scoreboard", StringComparer.Ordinal))
        {
            // Reads stored artifacts only -- no store, no judge, no spend. Checked before the verbs
            // that name a vertical because the scoreboard names all ten, and an absent one is a row
            // it has to print rather than an argument it could take.

tools/AgentMemory.LongMemEval/TypedMemEvalProgram.cs:296

  • These new floor results are printed only after Persist and are not added to the report JSON or provenance sidecar. If console output is not retained, the archived artifact loses the per-shape floor, census, p-value, and underpowered verdict that this instrument is meant to make durable; persist the derived block alongside the run before relying on it for later audit.
            PrintReachableCeiling(descriptor, result);
            TypedMemEvalRetrieverSensitivity.Print(descriptor);
            TypedMemEvalFloorReport.Print(result, options.Arm.FileToken());

tools/AgentMemory.LongMemEval/TypedMemEvalProgram.cs:626

  • The printed fraction uses CorrectQuestions/Questions, but ShareOfReachable divides by ReachableQuestions. On a capped corpus this produces a denominator in the text that does not match the percentage, so the console measurement is misleading. Print the reachable denominator here (or label the whole-corpus fraction separately).
            $"{header} Score {result.CorrectQuestions}/{ceiling.Questions} = "
            + $"{ceiling.ShareOfReachable(result.CorrectQuestions):P1} OF REACHABLE."));

tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs:143

  • The sidecar records sampling.oracle and sampling.control, but this reader filters only on the feature arm token. Oracle runs and Prospective control runs therefore share the default token and can become the newest selected row, causing gold-context or control scores to be presented as the default agent measurement. Exclude those run kinds before choosing the latest artifact (and test both cases).
            if (!sidecar.TryGetProperty("arm", out var arm) ||
                !arm.TryGetProperty("token", out var token) ||
                token.ValueKind != JsonValueKind.String ||
                !string.Equals(token.GetString(), armToken, StringComparison.Ordinal))
            {
                return null;
            }

tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs:185

  • The malformed-artifact path has the same gap here: Read calls TryGetProperty on report/shape values, which throws InvalidOperationException when a JSON value is null or the wrong kind, but this filter catches only parse/IO exceptions. The scoreboard then crashes instead of treating the artifact as unplaceable, contrary to the surrounding comment.
        catch (Exception exception) when (exception is JsonException or IOException or UnauthorizedAccessException)
        {
            // An unreadable artifact must not take the scoreboard down, and must not be silently
            // counted either: it simply never becomes a placeable run, so its vertical reads ABSENT.
            return null;

tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs:220

  • Int silently converts missing, null, or non-numeric fields to zero. A truncated report with a valid ScoredQuestions but no CorrectQuestions is therefore placed as a legitimate 0% row (and malformed shape tallies can likewise affect ranking), contrary to the stated unreadable/not-placeable behavior. Required measurement fields should make Read reject the artifact instead of defaulting them.
            SelectedQuestions: Int(report, "SelectedQuestions"),
            CorpusQuestions: Int(provenance, "DatasetQuestionCount"),
            ScoredQuestions: Int(report, "ScoredQuestions"),
            CorrectQuestions: Int(report, "CorrectQuestions"),
            AgentFailureQuestions: Int(report, "AgentFailureQuestions"),

tools/AgentMemory.LongMemEval/TypedMemEvalScoreboardProgram.cs:42

  • This helper silently falls back to the default when --artifacts or --arm is the final argument, so a malformed invocation can print a different scoreboard than the one requested and still exit successfully. Match the other command-line parsers by throwing when the switch is present without a value.
    private static string? Value(string[] args, string name)
    {
        var index = Array.IndexOf(args, name);
        return index >= 0 && index + 1 < args.Length ? args[index + 1] : null;
    }
  • Files reviewed: 9/9 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

// directory, on the same arm, against the same corpus, minutes before the real one. It is
// also the NEWEST artifact for hours afterwards. Left in, it would have reported
// prospective as 0.0% -- a sample of one presented as a vertical.
if (run.CorpusQuestions > 0 && run.SelectedQuestions < run.CorpusQuestions) continue;
Comment on lines +351 to +352
return scored <= 0
? RankingOnlyScore.NotRankable(nonRanking, unclassified)
Comment on lines +469 to +474
if (judges.Length > 1)
{
refusals.Add(
$"{judges.Length} distinct judge prompt fingerprints across placed rows — "
+ "these cells were graded by different judges and cannot be ranked against each other");
}
Comment on lines +437 to +443
/// <summary>A release without its 40-character build metadata, which no reader needs inline.</summary>
private static string Build(string? version)
{
if (version is null) return "(unrecorded)";
var plus = version.IndexOf('+', StringComparison.Ordinal);
return plus < 0 ? version : $"{version[..plus]}+{Short(version[(plus + 1)..])}";
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants