Skip to content

Demote the dense flag from a filter to a three-class read - #219

Merged
joslat merged 1 commit into
mainfrom
feat/dense-three-class-read
Sep 15, 2026
Merged

joslat merged 1 commit into
mainfrom
feat/dense-three-class-read

Conversation

@joslat

@joslat joslat commented Sep 15, 2026

Copy link
Copy Markdown
Owner

Executes the 2026-09-15 coordinator ruling, which amends SEND-36 §4.

Why the boolean had to demote

The sidecar's discriminates_under_dense is measured with text-embedding-ada-002 — a 2022 model the sidecar did not name until 2026-09-14, when it said only "azure-openai-embeddings", which every embedding model Azure has ever served satisfies. Against text-embedding-3-small the flag flips on 4 of 35 shapes, and neither embedder is ours.

One retriever is not a condition; it is an assumption. So the flag becomes three classes:

class n meaning
ROBUST-RANKING 24/35 discriminates under both — ranks us with confidence
RETRIEVER-SENSITIVE 4/35 flips — read cautiously, never load-bearing alone
NON-RANKING 7/35 fails under both — as before

The flips run in both directions. forgetting/still-valid, temporal/interval-position and workingmemory/distance-25 lose their flag under 3-small; workingmemory/distance-40 gains one. A "sensitive" class that only meant demoted would have missed the one shape the newer retriever rescues.

What it does to the board

Both are re-reads of cells already in the ledger, not new measurements.

  • forgetting 88.0% → NO ROBUST SHAPE. Its only ranking cell (still-valid) is sensitive, invalidated is non-ranking, never-known is unclassified. The vertical has nothing that ranks under both retrievers. Its 100% was already flagged as a single cell at n=15; this removes even that.
  • temporal → 42.9% over two robust shapes, interval-position held apart.

The sensitive tail is appended to the column, never merged into the score. Counting it would let a fact about an embedder move a number about the engine; discarding it would throw away a cell one published retriever does rank.

The derivation, and what licenses it

Numbers come from AgentEval's own tools/typedmemeval_retriever_compare.py, at zero API cost over already-banked vectors. Two checks stand behind it, both re-run as tests:

  • The tool's self-check (--models X,X) reports a spread of exactly 0.000000 — so the cross-model spread measures the model, not the plumbing. This was the ruling's gate: a compare tool is a probe, and a probe must be able to come out the other way.
  • The tool's ada column reproduces the package's published predicted_headroom_dense on all 35 shapes. Without that, its second column would describe a retrieval the sidecar never reports.

Family-level: ada-002 closes 21.3% of the headroom BM25 leaves; 3-small closes 27.2%. The 0.035 ALLgold spread is the part of any "dense retrieval closes X" sentence that belongs to the model.

One inference, labelled as one

The sidecar ships the boolean and the headroom but not the rule joining them. The threshold headroom > 0.10 is inferred. It reproduces all 35 published booleans across a clean gap — every FALSE at or below 0.100, every TRUE at or above 0.167 — and a test re-derives all 35, so an upstream rule change fails loudly instead of silently reclassifying a cell. It is marked an inference everywhere it appears.

The derived table is marked DERIVED, not published. If AgentEval co-publish the second column, delete the table and read the package: a snapshot cannot track a corpus.

Verification

836 tests green. Release verification is left to CI — a banded temporal run holds the Release binaries, and killing a 12-hour measurement to satisfy a local build would have cost more than it proved.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju

Executes the 2026-09-15 ruling, which amends SEND-36 §4.

The sidecar's `discriminates_under_dense` is measured with text-embedding
-ada-002 -- a 2022 model the sidecar did not name until 2026-09-14, when it
said only "azure-openai-embeddings", which every embedding model Azure has
ever served satisfies. Against text-embedding-3-small the flag FLIPS ON 4 OF
35 SHAPES, and neither embedder is ours. One retriever is not a condition; it
is an assumption.

So the boolean becomes three classes:
  ROBUST-RANKING       24/35  discriminates under BOTH -- ranks us confidently
  RETRIEVER-SENSITIVE   4/35  flips -- never load-bearing alone
  NON-RANKING           7/35  fails under both -- as before

THE FOUR FLIPS RUN IN BOTH DIRECTIONS: forgetting/still-valid,
temporal/interval-position and workingmemory/distance-25 lose their flag under
3-small, while workingmemory/distance-40 GAINS one. A "sensitive" class that
only meant "demoted" would have missed the shape the newer retriever rescues.

WHAT IT DOES TO THE BOARD, and both are re-reads rather than new measurements:

  forgetting  88.0%  ->  NO ROBUST SHAPE. Its only ranking cell, still-valid,
              is retriever-sensitive; invalidated is non-ranking and
              never-known is unclassified. The vertical has nothing that ranks
              under both retrievers.
  temporal    62.0%  ->  42.9% over two robust shapes, with interval-position
              held apart as sensitive.

The sensitive tail is APPENDED to the column, never merged into the score:
counting it would let a fact about an embedder move a number about the engine,
and discarding it would throw away a cell one published retriever does rank.

DERIVED, NOT PUBLISHED. The second column comes from AgentEval's own
tools/typedmemeval_retriever_compare.py at zero API cost. Two checks license
it, both re-run as tests:

  - the tool's SELF-CHECK (--models X,X) reports spread exactly 0.000000, so
    the comparison measures the model and not the plumbing;
  - the tool's ada column reproduces the package's published
    predicted_headroom_dense on ALL 35 SHAPES -- without which its second
    column would describe a retrieval the sidecar never reports.

The threshold joining headroom to the boolean is NOT published, so it is
inferred: `headroom > 0.10` reproduces all 35 published booleans across a
clean gap (every FALSE at or below 0.100, every TRUE at or above 0.167). It is
labelled an inference everywhere it appears, and a test re-derives all 35 so an
upstream rule change fails loudly instead of quietly reclassifying a cell.

If AgentEval co-publish the second column, delete the derived table and read
the package: a snapshot cannot track a corpus, a published column can.

836 tests green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HcipSXisrSNRoF5bjeZ6Ju
Copilot AI lite review requested due to automatic review settings September 15, 2026 06:43

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Several moderate issues remain in diagnostic consistency, validation coverage, and sensitive-state handling.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Updates TypedMemEval’s dense-retrieval handling from a binary filter to robust, sensitive, and non-ranking classifications.

Changes:

  • Adds derived cross-retriever headroom and classification data.
  • Scores robust shapes separately and reports sensitive results.
  • Extends tests for classification and derivation behavior.
File summaries
File Summary
tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs Applies robust scoring and sensitive-result reporting.
tools/AgentMemory.LongMemEval/TypedMemEvalDenseSecondOpinion.cs Defines derived retriever data and three-way classification.
tests/AgentMemory.Tests.Unit.LongMemEval/DenseSecondOpinionTests.cs Validates classification derivation and shape coverage.
Review details

Suppressed comments (5)

tests/AgentMemory.Tests.Unit.LongMemEval/DenseSecondOpinionTests.cs:142

  • These assertions do not pin the distribution promised by the ruling: they allow any 31/35 split between robust and non-ranking as long as both are non-empty. Since this table drives the scoreboard, a drift that changes the reported 24 robust/7 non-ranking result would still pass; assert 24 and 7 explicitly.
        byClass.GetValueOrDefault(DenseRankingClass.Robust).Should().BeGreaterThan(0);
        byClass.GetValueOrDefault(DenseRankingClass.NonRanking).Should().BeGreaterThan(0);

tools/AgentMemory.LongMemEval/TypedMemEvalDenseSecondOpinion.cs:24

  • The XML remarks use the singular product name as a singular subject: “If AgentEval later co-publish” should be “If AgentEval later co-publishes”.
/// If AgentEval later co-publish the second column in the sidecars — which the coordinator has

tools/AgentMemory.LongMemEval/TypedMemEvalDenseSecondOpinion.cs:22

  • The <code> sample contains the documentation-comment marker /// inside the shell command. Copying it produces an invalid command; split the XML comment across two lines so only the leading markers remain syntax.
/// python tools/typedmemeval_retriever_compare.py ///     --models text-embedding-ada-002,text-embedding-3-small

tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs:360

  • The sensitive branch is not covered end-to-end: the existing scoreboard tests exercise robust and non-ranking shapes, while the new tests only classify the hard-coded table. A regression could fold a sensitive shape into Correct/Scored or render its appended tail incorrectly while all current tests pass. Add a scoreboard test that assembles a run containing a sensitive shape and asserts it is excluded from the robust denominator but retained in SensitiveShapes, SensitiveCorrect, and SensitiveScored.
                case DenseRankingClass.RetrieverSensitive:
                    sensitiveShapes.Add(shape);
                    sensitiveCorrect += tally.Correct;
                    sensitiveScored += tally.N - tally.Unrun;
                    break;

tools/AgentMemory.LongMemEval/TypedMemEvalScoreboard.cs:376

  • scored now includes only robust shapes, so this branch is also reached when a run has sensitive shapes but no robust ones. It returns RankingState.NotRankable, whose existing contract says every shape is non-ranking, while the new output is NO ROBUST SHAPE with a sensitive tail. Introduce a distinct no-robust state or update the state/constructor contract and consumers so a sensitive-only result is not misclassified.
        return scored <= 0
            ? RankingOnlyScore.NotRankable(nonRanking, unclassified) with
  • Files reviewed: 3/3 changed files
  • Comments generated: 5
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +50 to +53
/// <summary>Per-shape headroom under both published retrievers, keyed <c>vertical/shape</c>.</summary>
private static readonly IReadOnlyDictionary<string, DenseHeadroomPair> Headroom =
new Dictionary<string, DenseHeadroomPair>(StringComparer.Ordinal)
{
Comment on lines +113 to +119
/// <summary>The three-class read the 2026-09-15 ruling requires.</summary>
internal DenseRankingClass Class => (DiscriminatesUnderAda, DiscriminatesUnderSmall) switch
{
(true, true) => DenseRankingClass.Robust,
(false, false) => DenseRankingClass.NonRanking,
_ => DenseRankingClass.RetrieverSensitive,
};
Comment on lines +346 to +348
// No second opinion for this shape: fall back to the published flag alone and say so
// by treating it as SENSITIVE, never as robust. One retriever is not a condition.
?? (flag.Discriminates ? DenseRankingClass.RetrieverSensitive : DenseRankingClass.NonRanking);
internal static class TypedMemEvalDenseSecondOpinion
{
/// <summary>
/// The headroom above which AgentEval call a shape discriminating. <b>INFERRED, not published.</b>
writer.WriteLine($" judge {Short(scoreboard.JudgePromptFingerprint)}");
writer.WriteLine(
" vertical share-of-all share-of-reachable ranking-only corpus");
" vertical share-of-all share-of-reachable robust-ranking corpus");
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants