Skip to content

Second periods, plans that begin, dates at their precision; filtered vector indexes on Neo4j 2026.x; event companions; retrim - #233

Merged
joslat merged 8 commits into
mainfrom
phase39/periods-and-retrim
Sep 28, 2026
Merged

joslat merged 8 commits into
mainfrom
phase39/periods-and-retrim

Conversation

@joslat

@joslat joslat commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

What changes

A value said again after it was replaced starts a second period

  • The bug: "Copenhagen, then Oslo, then back to Copenhagen" re-opened the first Copenhagen fact. It was live, but still ended at the Oslo move, and its first period was lost, so "as of the Oslo months" believed both cities.
  • Periods: facts carry a period_key: open ('') until a newer value supersedes them, then the fact's own id.
    • The three fact write paths (single, batch, fused) MERGE on the open period, so the triple said again is a new fact and the closed one keeps its end.
    • A restatement of a live fact still lands on it (mention count, confidence).
    • A fact closed any other way (retracted, decayed, consolidated) re-opens as before.
  • Existing stores get their keys at bootstrap, before any write. FindByTripleAsync prefers the live period.
  • One clock: FindSupersededCandidatesAsync gains a now overload, so write-time supersession judges the candidates at the instant it judged the new value.

A plan that has begun takes over from the value it was planned to replace.

  • "I'm moving to Oslo next month" replaces nothing when said. Once its start has passed, recall (live, and as-of at the valid-time instant) and the working-memory block give Oslo, not both.
  • Nothing is rewritten, so a cancelled plan simply stops being live.
  • Starts are compared at the precision they were said: "in August" is the whole month, so it is not ordered before "on 20 August". Overlapping starts are ordered by when they were said.

Owner-filtered vector indexes on Neo4j 2026.x (Neo4jOptions.FilteredVectorIndexes, off by default)

  • Bootstrap: checks the server, refusing an older one by name, and creates fact_embedding_owner_idx and entity_embedding_owner_idx (the embedding WITH [owner_key]).
  • Recall: owner-scoped fact and entity recall searches them with Cypher 25 SEARCH: the owner's rows, then shared rows, merged by score. The server takes an exact value in the filter, not a list or an OR. An owner can no longer be crowded out, so the widening and rescue scans are not needed.
  • Test server: the integration fixture takes AGENTMEMORY_TEST_NEO4J_IMAGE. The default is neo4j:5.26; the suite also passes on neo4j:2026.02.

Who an event was shared with is kept as its own fact (LlmExtractionOptions.CaptureEventCompanions, off by default)

  • Each companion becomes its own fact, e.g. "user | went hiking with | Pedro" with the event's day, however the event itself is split.
  • Every prompt builder carries the instruction; off, every prompt is byte-for-byte unchanged.

A corrected name renames in place when entity resolution already filed the new name under the old entity, for example "Miso" as an alias of "Missou". Found by the 2026.02 run; completes the rename-on-correction of #232.

agentmemory retrim [--apply]

  • On-demand repair of facts stored before predicate-echo trimming ("is a chef | chef").
  • Each fact is rewritten as it reads once trimmed, or superseded by the trimmed fact when that already exists.
  • A dry run unless --apply; never run at bootstrap.

Verification

  • Unit: 6,002 passed, 0 failed.
  • Integration: 597 passed, 0 failed on Neo4j 5.26, and the same on Neo4j 2026.02 (29 model-dependent skips each). On 2026.x the EXPLAIN sweep creates the owner-filtered indexes and checks the Cypher 25 queries; the owner-starvation checks, specific to 5.26's index, are asserted on 5.26 only.
  • New tests (each fails with its change reverted):
    • second period, live restatement, and the backfill on an existing store;
    • plan hand-over, in live recall, as-of recall and the working-memory block;
    • dates compared at their precision;
    • the one-clock candidate lookup;
    • filtered index: refused on 5.26, used on 2026.02;
    • event companions in every prompt builder;
    • retrim: rewrite, supersede, and dry run.
  • Perf gate: PASS on both profiles.
  • Samples: MinimalAgent and AgentWithMemory run end to end on b3bc0231.
  • Live check with a real model and event companions on: "Who was with me in Sintra?" was answered from the new companion fact, even though the extraction split the event.

🤖 Generated with Claude Code

https://claude.ai/code/session_0139u7CNFuidKuPJBRK7auC9

joslat and others added 8 commits September 28, 2026 18:32
Facts written before the write-time trim read "is a chef chef". The on-demand repair
(ISchemaBootstrapper.RetrimEchoedPredicatesAsync, CLI 'retrim', dry run unless --apply)
selects live facts whose predicate ends with their object (a parameterised pre-filter),
applies the same PredicateEcho rule as the write path, and rewrites each fact with its
identity keys, or supersedes it with the library's own supersede statement when the
trimmed fact is already stored. Never run at bootstrap: it rewrites what people said, so
it is asked for. Integration test (rewrite, supersede, untouched control, dry run,
idempotent), failing without the change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0139u7CNFuidKuPJBRK7auC9
…clock for supersession

"Copenhagen, Oslo, back to Copenhagen" re-opened the first Copenhagen fact (live, but still
ended at the Oslo move) and lost its first period, so as of the Oslo months both cities were
believed. Facts carry a period_key: open ('') until superseded, then the fact's own id. The
three fact MERGE sites (single, batch, fused) match the open period, so the triple said
again is a new fact and the closed one keeps its end; a live fact's restatement still lands
on it, and a fact closed any other way re-opens as before. Existing stores get their keys at
bootstrap (before any write, like the canonical keys), else live facts would be written
twice. FindByTriple prefers the live period. FindSupersededCandidatesAsync gains a 'now'
overload and the stage passes its clock, so the candidates are judged at the instant the
winner was.

Integration tests on a real store: move back (two Copenhagen periods, as-of Oslo believes
only Oslo, the lookup finds the live one), a live restatement stays one fact, and an
existing store's backfill (keys right, no duplicate on restatement); a unit test for the
clock. Each fails without its change. Query inventory 181 -> 185 (this and retrim).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0139u7CNFuidKuPJBRK7auC9
…ey were said

"I'm moving to Oslo next month" is a plan and replaces nothing when said, so once the
month came Copenhagen and Oslo were both current. ValueHandOver, applied where recall
chooses its facts (live at now, as-of at the valid-time instant): per owner, subject and
single-valued relation, among values that have begun, the one that began last is current
and the ones that began before it are dropped; plans still ahead stay. Nothing is rewritten,
so a cancelled plan just stops being live. Starts are spans at their precision (a year, a
month, a day, an instant): 'in August' is not before 'on 20 August'; overlapping starts are
ordered by when they were said. Scores filtered in lockstep.

Unit tests (begun, ahead, as-of before, overlap both ways, apart by date, multi-valued and
other owners) and an integration test through live and as-of recall; the wiring test fails
without the change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0139u7CNFuidKuPJBRK7auC9
…panions, off by default)

'Hiking in Sintra with my friend Pedro' was kept whole in one extraction and split without
Pedro in the next, so 'who was with me?' had no answer. With the option on, every prompt
builder (unified, multi-session, per-type facts) asks for, per companion, 'user | went
hiking with | Pedro' with the event's day, however the event itself is split; off, every
prompt is unchanged. An optional builder parameter like the other switches, so the
multi-session call and its token estimate carry the same prompt. Tests: each builder on and
off, off byte-identical, both extractors send it exactly when on (fails without the change).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0139u7CNFuidKuPJBRK7auC9
…ectorIndexes, off)

A 2026.x vector index can carry filter properties and filter inside the index. With the
option on, bootstrap checks the server (refusing an older one by name) and creates
fact_embedding_owner_idx and entity_embedding_owner_idx (embedding, WITH [owner_key]);
owner-scoped fact and entity recall search them with Cypher 25 SEARCH, own rows then shared
rows ('*'), merged by score (probed on 2026.02.3: the filter takes an exact value, not a
list parameter or an OR). An owner cannot be crowded out, so no widening or rescue scan.
One merge helper (ScoredMerge) now serves facts and entities. Off, nothing changes.

The integration fixture takes AGENTMEMORY_TEST_NEO4J_IMAGE (default neo4j:5.26). Tests: on
5.26 the option is refused by name; on 2026.02 the indexes come ONLINE and an owner's one
fact is found among 200 closer foreign ones, with shared merged in, through the filtered
index (fails with the option ignored). Query inventory 185 -> 186 (ServerVersion).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0139u7CNFuidKuPJBRK7auC9
…cond period and Cypher 25

Review of this branch: the working-memory block chose its stable facts on its own, so once a
planned move began it still stated both homes while recall gave one. It now applies the
same ValueHandOver (ids and created_at read with the facts); the text changes only when a
hand-over applies. Integration test, failing without it.

The R5 re-assert test pinned the re-opened node; its concern (a re-asserted triple is
visible to live recall) now holds through the second period, and the first keeps its end.
The EXPLAIN sweep skips Cypher 25 queries (G-16, Neo4j 2026.x) on a server without it and
EXPLAINs them on one that has it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0139u7CNFuidKuPJBRK7auC9
…d and created_at

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0139u7CNFuidKuPJBRK7auC9
…ty; tests aware of the server

Found by the full suite on Neo4j 2026.02: when entity resolution attaches the corrected name
to the old entity as an alias (a close embedding), 'Miso' finds 'Missou' itself and the
rename did nothing, leaving the old name as the entity's own. It now renames that entity in
place (the new name its own, the old one an alias). Integration test, failing without it.

On 2026.x: the EXPLAIN sweep creates the owner-filtered indexes for its duration (EXPLAIN
resolves the index a SEARCH names); the owner-starvation witnesses of OwnerFirstRecall and
ShortRescueYieldArm hold on 5.26's global index only (2026.x's finds the owner), so they are
asserted there and reported elsewhere. The fixture gains ServerMajorAsync, used by these and
the G-16 test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0139u7CNFuidKuPJBRK7auC9
Copilot AI lite review requested due to automatic review settings September 28, 2026 17:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate findings affect period backfill, tenant isolation, temporal recall, retrim, and maintenance behavior.

Review effort: Lite
Findings: 3 High severity · 6 Medium severity · 1 Low severity

Open (10)
What changed in this PR

This PR adds temporal fact periods, plan handover, optional Neo4j 2026 filtered indexes, event companions, corrected-name handling, and the retrim maintenance command.

Changes:

  • Adds period-aware supersession and precision-aware plan handover.
  • Adds filtered vector indexes and companion extraction.
  • Adds retrim support, schema updates, and integration coverage.
File Review summary
tools/​AgentMemory.Cli/​Program.cs Registers retrim.
tools/​AgentMemory.Cli/​Commands/​MemoryCommands.cs Implements the retrim command.
tools/​AgentMemory.Cli/​CliArgs.cs Documents retrim usage.
tests/​AgentMemory.Tests.Unit/​TestSupport/​StoreFakes.cs Updates repository fake signatures.
tests/​AgentMemory.Tests.Unit/​Repositories/​Neo4jFactRepositoryTests.cs Updates period-aware merge assertions.
tests/​AgentMemory.Tests.Unit/​Repositories/​Neo4jFactRepositoryDeduplicationTests.cs Covers period-aware deduplication.
tests/​AgentMemory.Tests.Unit/​Queries/​CypherQuerySnapshotTests.cs Updates query inventory coverage.
tests/​AgentMemory.Tests.Unit/​Queries/​CypherQuerySnapshot.snap Updates Cypher snapshots.
tests/​AgentMemory.Tests.Unit/​Memory/​ValueHandOverTests.cs Tests plan handover and date precision.
tests/​AgentMemory.Tests.Unit/​Extraction/​WriteTimeSupersessionTests.cs Tests synchronized supersession clocks.
tests/​AgentMemory.Tests.Unit/​Extraction/​WithinExtractionDedupTests.cs Updates supersession setup.
tests/​AgentMemory.Tests.Unit/​Extraction/​UserNameResolutionTests.cs Updates repository call setup.
tests/​AgentMemory.Tests.Unit/​Extraction/​EventCompanionsInstructionTests.cs Tests companion prompt behavior.
tests/​AgentMemory.Tests.Unit/​Extraction/​ChangesOfMindTests.cs Updates supersession tests.
tests/​AgentMemory.Tests.Integration/​Repositories/​ShortRescueYieldArmIntegrationTests.cs Adjusts version-specific expectations.
tests/​AgentMemory.Tests.Integration/​Repositories/​RetrimEchoedPredicatesIntegrationTests.cs Tests retrim repair.
tests/​AgentMemory.Tests.Integration/​Repositories/​OwnerFirstRecallIntegrationTests.cs Adjusts filtered-index expectations.
tests/​AgentMemory.Tests.Integration/​Repositories/​FilteredVectorIndexIntegrationTests.cs Tests Neo4j 2026 filtered indexes.
tests/​AgentMemory.Tests.Integration/​Repositories/​FactRepositoryIntegrationTests.cs Tests second fact periods.
tests/​AgentMemory.Tests.Integration/​Repositories/​DatePrecisionIntegrationTests.cs Tests working-memory handover.
tests/​AgentMemory.Tests.Integration/​Queries/​CypherQueryExecutionSweepTests.cs Validates Cypher 25 queries.
tests/​AgentMemory.Tests.Integration/​Fixtures/​Neo4jIntegrationFixture.cs Adds configurable Neo4j images.
tests/​AgentMemory.Tests.Integration/​Extraction/​SecondPeriodIntegrationTests.cs Tests periods and plans end to end.
tests/​AgentMemory.Tests.Integration/​Extraction/​RenameOnCorrectedNameIntegrationTests.cs Tests in-place renaming.
src/​AgentMemory.Neo4j/​Services/​Neo4jWorkingMemoryService.cs Moderate: handover occurs after the stable-fact slot cap, so begun plans can fail to replace old values.
src/​AgentMemory.Neo4j/​Repositories/​ScoredMerge.cs Adds scored-result merging.
src/​AgentMemory.Neo4j/​Repositories/​Neo4jFactRepository.cs Critical: * owner IDs collide with the shared sentinel. Moderate: as-of recall still uses the global index and can suffer owner starvation.
src/​AgentMemory.Neo4j/​Repositories/​Neo4jEntityRepository.cs Critical: * owner IDs collide with the shared sentinel. Moderate: as-of entity recall bypasses filtered indexes. Nit: filtered entity recall lacks integration coverage.
src/​AgentMemory.Neo4j/​Queries/​WorkingMemoryQueries.cs Projects temporal fact metadata for handover.
src/​AgentMemory.Neo4j/​Queries/​SchemaQueries.cs Nit: XML documentation is attached to the wrong members.
src/​AgentMemory.Neo4j/​Queries/​FusedPersistenceQueries.cs Adds period-aware fused writes.
src/​AgentMemory.Neo4j/​Queries/​FactQueries.cs Critical: period backfill can misclassify live reopened facts. Moderate: retrim excludes invalidated facts and preserves stale embeddings. Nit: XML documentation placement is incorrect.
src/​AgentMemory.Neo4j/​Queries/​EntityQueries.cs Adds filtered entity-search queries.
src/​AgentMemory.Neo4j/​Infrastructure/​SchemaBootstrapper.cs Moderate: dry-run pagination stops after the first 500 rows and undercounts larger stores.
src/​AgentMemory.Neo4j/​Infrastructure/​Neo4jOptions.cs Nit: required architecture, setup, and schema documentation is incomplete.
src/​AgentMemory.Neo4j/​Infrastructure/​ISchemaBootstrapper.cs Exposes retrim maintenance operations.
src/​AgentMemory.Neo4j/​Infrastructure/​CypherBuilder.cs Moderate: filtered SEARCH queries lack structural fingerprints for telemetry and performance accounting.
src/​AgentMemory.Extraction.Llm/​LlmUnifiedMemoryExtractor.cs Wires companion prompts.
src/​AgentMemory.Extraction.Llm/​LlmMultiSessionUnifiedMemoryExtractor.cs Propagates companion prompts.
src/​AgentMemory.Extraction.Llm/​LlmFactExtractor.cs Adds companion prompt support.
src/​AgentMemory.Extraction.Llm/​LlmExtractionOptions.cs Defines companion capture configuration.
src/​AgentMemory.Extraction.Llm/​ExtractionPromptSemantics.cs Adds companion instructions.
src/​AgentMemory.Core/​Services/​MemoryContextAssembler.cs Moderate: handover after top-K selection can leave superseded values in recall.
src/​AgentMemory.Core/​Memory/​ValueHandOver.cs Implements temporal value selection.
src/​AgentMemory.Core/​Extraction/​PersistenceStage.cs Uses one supersession clock and handles renames.
src/​AgentMemory.Abstractions/​Repositories/​IFactRepository.cs Adds a clock-aware candidate lookup overload.
docs/​configuration/​memory-options.md Nit: canonical architecture, setup, capability, and schema documentation remains incomplete.
CHANGELOG.md Records user-visible changes.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

MATCH (f:Fact)
WHERE f.period_key IS NULL
WITH f LIMIT $limit
SET f.period_key = CASE WHEN EXISTS { (f)-[:SUPERSEDED_BY]->(:Fact) } THEN f.id ELSE $open END
Comment on lines +281 to +285
var own = await QueryAsync(topK, cancellationToken, filteredKey: scope!.OwnerId).ConfigureAwait(false);
var filtered = includeShared
? ScoredMerge.ByScore(own, await QueryAsync(topK, cancellationToken, filteredKey: Neo4jFactRepository.OwnerKeyShared).ConfigureAwait(false),
entity => entity.EntityId, limit)
: own;
Comment on lines +539 to +542
var own = await QueryAsync(topK, cancellationToken, filteredKey: scope!.OwnerId).ConfigureAwait(false);
var filtered = includeShared
? MergeByScore(own, await QueryAsync(topK, cancellationToken, filteredKey: OwnerKeyShared).ConfigureAwait(false), limit)
: own;
Comment on lines +129 to +131
"CYPHER 25",
$"MATCH ({nodeAlias}:{label})",
$"SEARCH {nodeAlias} IN (VECTOR INDEX {indexName} FOR {embeddingParam} WHERE {nodeAlias}.owner_key = {ownerKeyParam} LIMIT {topK}) SCORE AS score",
Comment on lines +300 to +301
// Applied rows leave the selection; a dry run sees the same page again. Stop when a page adds nothing new.
if (!progressed || rows.Count < CanonicalKeyBackfillBatchSize) break;
/// </summary>
public const string SelectEchoedPredicates = @"
MATCH (f:Fact)
WHERE f.invalidated_at IS NULL
Comment on lines +62 to +64
SET f.predicate = $predicate, f.object = $object,
f.predicate_key = $predicateKey, f.object_key = $objectKey,
f.updated_at = datetime($now)
Comment on lines +537 to +546
if (hasOwner && _filteredOwnerIndex)
{
var own = await QueryAsync(topK, cancellationToken, filteredKey: scope!.OwnerId).ConfigureAwait(false);
var filtered = includeShared
? MergeByScore(own, await QueryAsync(topK, cancellationToken, filteredKey: OwnerKeyShared).ConfigureAwait(false), limit)
: own;
activity?.SetTag("memory.vector.owner_scoped", true);
activity?.SetTag("memory.vector.owner_first", "FilteredIndex");
activity?.SetTag("memory.vector.returned", filtered.Count);
return filtered.Select(r => (r.Item1, r.Item2)).ToList();
OccurredOn = Neo4jDateTimeHelper.ReadNullableDateTimeOffset(r["occurredOn"]),
OccurredOnPrecision = DatePrecisionProperty.FromStored(r["occurredOnPrecision"] as string),
}).ToList();
return AgentMemory.Core.Memory.ValueHandOver.Current(stable, now)

| Option | Default | What it does |
|---|---|---|
| `Neo4jOptions.FilteredVectorIndexes` | `false` | Neo4j 2026.x and later: bootstrap creates owner-filtered vector indexes for facts and entities (the embedding, filtered on `owner_key`), and owner-scoped recall searches them, so an owner is never crowded out of a global top-K by other owners' rows. Refused at bootstrap on an older server. |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants