Repository navigation
Demote the dense flag from a filter to a three-class read - #219
Conversation
Executes the 2026-09-15 ruling, which amends SEND-36 §4.
The sidecar's `discriminates_under_dense` is measured with text-embedding
-ada-002 -- a 2022 model the sidecar did not name until 2026-09-14, when it
said only "azure-openai-embeddings", which every embedding model Azure has
ever served satisfies. Against text-embedding-3-small the flag FLIPS ON 4 OF
35 SHAPES, and neither embedder is ours. One retriever is not a condition; it
is an assumption.
So the boolean becomes three classes:
ROBUST-RANKING 24/35 discriminates under BOTH -- ranks us confidently
RETRIEVER-SENSITIVE 4/35 flips -- never load-bearing alone
NON-RANKING 7/35 fails under both -- as before
THE FOUR FLIPS RUN IN BOTH DIRECTIONS: forgetting/still-valid,
temporal/interval-position and workingmemory/distance-25 lose their flag under
3-small, while workingmemory/distance-40 GAINS one. A "sensitive" class that
only meant "demoted" would have missed the shape the newer retriever rescues.
WHAT IT DOES TO THE BOARD, and both are re-reads rather than new measurements:
forgetting 88.0% -> NO ROBUST SHAPE. Its only ranking cell, still-valid,
is retriever-sensitive; invalidated is non-ranking and
never-known is unclassified. The vertical has nothing that ranks
under both retrievers.
temporal 62.0% -> 42.9% over two robust shapes, with interval-position
held apart as sensitive.
The sensitive tail is APPENDED to the column, never merged into the score:
counting it would let a fact about an embedder move a number about the engine,
and discarding it would throw away a cell one published retriever does rank.
DERIVED, NOT PUBLISHED. The second column comes from AgentEval's own
tools/typedmemeval_retriever_compare.py at zero API cost. Two checks license
it, both re-run as tests:
- the tool's SELF-CHECK (--models X,X) reports spread exactly 0.000000, so
the comparison measures the model and not the plumbing;
- the tool's ada column reproduces the package's published
predicted_headroom_dense on ALL 35 SHAPES -- without which its second
column would describe a retrieval the sidecar never reports.
The threshold joining headroom to the boolean is NOT published, so it is
inferred: `headroom > 0.10` reproduces all 35 published booleans across a
clean gap (every FALSE at or below 0.100, every TRUE at or above 0.167). It is
labelled an inference everywhere it appears, and a test re-derives all 35 so an
upstream rule change fails loudly instead of quietly reclassifying a cell.
If AgentEval co-publish the second column, delete the derived table and read
the package: a snapshot cannot track a corpus, a published column can.
836 tests green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju
There was a problem hiding this comment.
🟡 Changes recommended
Several moderate issues remain in diagnostic consistency, validation coverage, and sensitive-state handling.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Updates TypedMemEval’s dense-retrieval handling from a binary filter to robust, sensitive, and non-ranking classifications.
Changes:
- Adds derived cross-retriever headroom and classification data.
- Scores robust shapes separately and reports sensitive results.
- Extends tests for classification and derivation behavior.
File summaries
| File | Summary |
|---|---|
tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs |
Applies robust scoring and sensitive-result reporting. |
tools/AgentMemory.LongMemEval/TypedMemEvalDenseSecondOpinion.cs |
Defines derived retriever data and three-way classification. |
tests/AgentMemory.Tests.Unit.LongMemEval/DenseSecondOpinionTests.cs |
Validates classification derivation and shape coverage. |
Review details
Suppressed comments (5)
tests/AgentMemory.Tests.Unit.LongMemEval/DenseSecondOpinionTests.cs:142
- These assertions do not pin the distribution promised by the ruling: they allow any 31/35 split between robust and non-ranking as long as both are non-empty. Since this table drives the scoreboard, a drift that changes the reported 24 robust/7 non-ranking result would still pass; assert 24 and 7 explicitly.
byClass.GetValueOrDefault(DenseRankingClass.Robust).Should().BeGreaterThan(0);
byClass.GetValueOrDefault(DenseRankingClass.NonRanking).Should().BeGreaterThan(0);
tools/AgentMemory.LongMemEval/TypedMemEvalDenseSecondOpinion.cs:24
- The XML remarks use the singular product name as a singular subject: “If AgentEval later co-publish” should be “If AgentEval later co-publishes”.
/// If AgentEval later co-publish the second column in the sidecars — which the coordinator has
tools/AgentMemory.LongMemEval/TypedMemEvalDenseSecondOpinion.cs:22
- The
<code>sample contains the documentation-comment marker///inside the shell command. Copying it produces an invalid command; split the XML comment across two lines so only the leading markers remain syntax.
/// python tools/typedmemeval_retriever_compare.py /// --models text-embedding-ada-002,text-embedding-3-small
tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs:360
- The sensitive branch is not covered end-to-end: the existing scoreboard tests exercise robust and non-ranking shapes, while the new tests only classify the hard-coded table. A regression could fold a sensitive shape into
Correct/Scoredor render its appended tail incorrectly while all current tests pass. Add a scoreboard test that assembles a run containing a sensitive shape and asserts it is excluded from the robust denominator but retained inSensitiveShapes,SensitiveCorrect, andSensitiveScored.
case DenseRankingClass.RetrieverSensitive:
sensitiveShapes.Add(shape);
sensitiveCorrect += tally.Correct;
sensitiveScored += tally.N - tally.Unrun;
break;
tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs:376
scorednow includes only robust shapes, so this branch is also reached when a run has sensitive shapes but no robust ones. It returnsRankingState.NotRankable, whose existing contract says every shape is non-ranking, while the new output isNO ROBUST SHAPEwith a sensitive tail. Introduce a distinct no-robust state or update the state/constructor contract and consumers so a sensitive-only result is not misclassified.
return scored <= 0
? RankingOnlyScore.NotRankable(nonRanking, unclassified) with
- Files reviewed: 3/3 changed files
- Comments generated: 5
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| /// <summary>Per-shape headroom under both published retrievers, keyed <c>vertical/shape</c>.</summary> | ||
| private static readonly IReadOnlyDictionary<string, DenseHeadroomPair> Headroom = | ||
| new Dictionary<string, DenseHeadroomPair>(StringComparer.Ordinal) | ||
| { |
| /// <summary>The three-class read the 2026-09-15 ruling requires.</summary> | ||
| internal DenseRankingClass Class => (DiscriminatesUnderAda, DiscriminatesUnderSmall) switch | ||
| { | ||
| (true, true) => DenseRankingClass.Robust, | ||
| (false, false) => DenseRankingClass.NonRanking, | ||
| _ => DenseRankingClass.RetrieverSensitive, | ||
| }; |
| // No second opinion for this shape: fall back to the published flag alone and say so | ||
| // by treating it as SENSITIVE, never as robust. One retriever is not a condition. | ||
| ?? (flag.Discriminates ? DenseRankingClass.RetrieverSensitive : DenseRankingClass.NonRanking); |
| internal static class TypedMemEvalDenseSecondOpinion | ||
| { | ||
| /// <summary> | ||
| /// The headroom above which AgentEval call a shape discriminating. <b>INFERRED, not published.</b> |
| writer.WriteLine($" judge {Short(scoreboard.JudgePromptFingerprint)}"); | ||
| writer.WriteLine( | ||
| " vertical share-of-all share-of-reachable ranking-only corpus"); | ||
| " vertical share-of-all share-of-reachable robust-ranking corpus"); |
Executes the 2026-09-15 coordinator ruling, which amends SEND-36 §4.
Why the boolean had to demote
The sidecar's
discriminates_under_denseis measured withtext-embedding-ada-002— a 2022 model the sidecar did not name until 2026-09-14, when it said only "azure-openai-embeddings", which every embedding model Azure has ever served satisfies. Againsttext-embedding-3-smallthe flag flips on 4 of 35 shapes, and neither embedder is ours.One retriever is not a condition; it is an assumption. So the flag becomes three classes:
The flips run in both directions.
forgetting/still-valid,temporal/interval-positionandworkingmemory/distance-25lose their flag under 3-small;workingmemory/distance-40gains one. A "sensitive" class that only meant demoted would have missed the one shape the newer retriever rescues.What it does to the board
Both are re-reads of cells already in the ledger, not new measurements.
forgetting88.0% → NO ROBUST SHAPE. Its only ranking cell (still-valid) is sensitive,invalidatedis non-ranking,never-knownis unclassified. The vertical has nothing that ranks under both retrievers. Its 100% was already flagged as a single cell at n=15; this removes even that.temporal→ 42.9% over two robust shapes,interval-positionheld apart.The sensitive tail is appended to the column, never merged into the score. Counting it would let a fact about an embedder move a number about the engine; discarding it would throw away a cell one published retriever does rank.
The derivation, and what licenses it
Numbers come from AgentEval's own
tools/typedmemeval_retriever_compare.py, at zero API cost over already-banked vectors. Two checks stand behind it, both re-run as tests:--models X,X) reports a spread of exactly0.000000— so the cross-model spread measures the model, not the plumbing. This was the ruling's gate: a compare tool is a probe, and a probe must be able to come out the other way.predicted_headroom_denseon all 35 shapes. Without that, its second column would describe a retrieval the sidecar never reports.Family-level: ada-002 closes 21.3% of the headroom BM25 leaves; 3-small closes 27.2%. The 0.035 ALLgold spread is the part of any "dense retrieval closes X" sentence that belongs to the model.
One inference, labelled as one
The sidecar ships the boolean and the headroom but not the rule joining them. The threshold
headroom > 0.10is inferred. It reproduces all 35 published booleans across a clean gap — every FALSE at or below 0.100, every TRUE at or above 0.167 — and a test re-derives all 35, so an upstream rule change fails loudly instead of silently reclassifying a cell. It is marked an inference everywhere it appears.The derived table is marked DERIVED, not published. If AgentEval co-publish the second column, delete the table and read the package: a snapshot cannot track a corpus.
Verification
836 tests green. Release verification is left to CI — a banded temporal run holds the Release binaries, and killing a 12-hour measurement to satisfy a local build would have cost more than it proved.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju