Skip to content

Give event reads an opt-in to archived events - #899

Merged
jeremydmiller merged 3 commits into
mainfrom
fix/archived-events-opt-in
Nov 2, 2026
Merged

jeremydmiller merged 3 commits into
mainfrom
fix/archived-events-opt-in

Conversation

@jeremydmiller

@jeremydmiller jeremydmiller commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Polecat filtered archived events out of event reads but gave callers no way in. This adds the opt-in, with Marten's semantics and Fisher's shape from JasperFx/fisher#511.

What changes for callers

IsArchived starts matching rows it never matched before. The default is_archived = 0 used to sit beside any predicate, so x.IsArchived could only ever match nothing. A predicate naming IEvent.IsArchived now replaces that term and answers for itself.

  • MaybeArchived() is new, in Polecat.Events.Archiving, for live and archived events together.
  • A query that never names IsArchived is unchanged, and so is !x.IsArchived on its own.
Predicate over IEvent Before Now
x.IsArchived, x.IsArchived == true matched nothing archived events
x.IsArchived || y live events matching y archived events, and any event matching y
!x.IsArchived || y live events live events, and archived events matching y

Masking is unaffected: #888 landed first and gave masking its own read that keeps archived events, so IncludeEvents doesn't depend on this opt-in, and a filter naming IsArchived still narrows it. Merging main into this branch kept #888's includeArchived provider flag beside the new check.

Reads this covers

  • QueryAllRawEvents() and QueryRawEvents(): both opt-ins work, in any Where and in a terminal's inline predicate (CountAsync, AnyAsync, FirstOrDefaultAsync). AggregateToAsync folds what its query matched.
  • QueryEventData<T> and QueryRawEventDataOnly<T>: these share the provider and keep the default filter. Their predicates are over the event body, so an IsArchived member on a body says nothing about the row, and there is no opt-in on them.
  • QueryStreamStates(): unchanged, because it has its own provider and no implicit filter. This is now pinned by a test.
  • OrderBy, Select: not an opt-in, because they filter nothing.

Not the saved patch

The 25-line patch noted on #1104 turns the compliance facts green, but it walks the whole expression, so OrderBy(x => x.IsArchived) and Select(x => x.IsArchived) silently returned archived events. It had no MaybeArchived() either. Here LinqQueryParser records the predicates that filter rows, and the check only looks at those. The check itself, ArchivalPredicates, is Fisher's. I ran the new tests with the patch's whole-expression walk swapped in to confirm: the OrderBy and Select tests fail.

A nested lambda is not the row either. The check first accepted any IEvent-typed parameter. In captured.Where(e => e.IsArchived).Select(e => e.Id).Contains(x.Id) the parser evaluates the list up front, but the check still walked into it, dropped is_archived = 0 and returned archived rows with no opt-in. It now matches only the predicate's own parameter. captured.Any(e => e.IsArchived) never got that far, because the parser refuses it. Fisher's ArchivalPredicates in JasperFx/fisher#511 was the source of this check and has the same shape.

The IsMaybeDeleted guard on the default term is gone. It could never fire, because soft-delete operators are refused over events.

Validation

Compliance, with a local pack. v2.97.0, the version Polecat pins, plus #1104's StreamArchivingCompliance.cs, packed as 2.97.0-local.1104. A real 3.0 pack can't reach Polecat yet. #1104's other two files are comment-only and did not apply to the tag, so they were left out.

Polecat stream_archiving_compliance
origin/main 25 passed, 2 failed: events_of_an_archived_stream_report_themselves_as_archived, a_predicate_naming_is_archived_reads_only_the_archived_events
this branch 27 passed, 0 failed

Tests in this PR. There are 17 tests across archived_events_are_opt_in_tests and the conjoined tenancy class, passing on net9.0 and net10.0 against SQL Server. With the old condition restored, 11 of the first 16 fail, and the other 5 pin behaviour that must not change. The 17th is the nested lambda test, which failed with 2 archived rows before the check was narrowed.

Affected classes. 938 passed, 0 failed on net10.0: all of Polecat.Tests.Linq, plus the archive, raw event, event data, stream state, masking, AggregateTo and event query classes.

After the nested lambda fix. At the head of this branch, stream_archiving_compliance is 27 passed, 0 failed, using v2.97.0's suite with #1104's StreamArchivingCompliance.cs applied through ComplianceSourceDir.

Polecat.Tests.Events and Polecat.Tests.Compliance on net10.0: 1,375 passed, 5 skipped, 3 failed. The 3 are tenant_partitioned_unseen_tenant_tests.fetch_for_writing_of_a_new_stream_in_a_session_holding_its_transaction (RepeatableRead, Serializable, Snapshot). They fail the same way on origin/main at 8545249, because the test runs ALTER DATABASE CURRENT and the default local connection points at master. This branch did not cause them.

The whole suite was not run locally. It takes about 2 hours, so that is left to CI.

Merging with #888

#888 is still open, so this branches from main. Either can land first. They conflict on one line, the default-term condition, and the resolution is:

if (!_includeArchived && !parser.RowPredicates.Any(x => x.SpecifiesEventArchivalStatus()))

I tested that merge, again after the nested lambda fix: 169 passed, 0 failed across the masking, compaction, archive and raw event classes, with #1104's suite applied. That includes #888's own IncludeEvents(e => e.IsArchived) and !e.IsArchived tests.

🤖 Generated with Claude Code

jeremydmiller and others added 2 commits November 2, 2026 06:38
Polecat filtered archived events out of event LINQ but left no way in: its
is_archived = 0 sat beside a predicate naming IEvent.IsArchived, so x.IsArchived
matched nothing, and there was no public MaybeArchived().

A predicate that filters rows now opts in by calling MaybeArchived() or by naming
IsArchived, which replaces the default term, as on Marten and Fisher. OrderBy and
Select name the member without filtering, so they leave the default on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ArchivalPredicates accepted any IEvent-typed parameter, so a nested lambda
over an in-memory list counted as the event being filtered. In
`captured.Where(e => e.IsArchived).Select(e => e.Id).Contains(x.Id)` the
parser evaluates the list up front, and the check still walked into it and
dropped is_archived = 0, returning archived rows with no opt-in.

Compare against the predicate's own parameter instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit 6fa3098 into main Nov 2, 2026
9 of 10 checks passed
jeremydmiller added a commit that referenced this pull request Nov 2, 2026
… full text seeks (#875, #877) (#913)

* LINQ answers what .NET would for spaces, objects and projections (#875, #877)

Members a query names but the JSON does not:

- An operator after a Select reads the projection. Select(x => x.Address)
  .Where(a => a.City == "x") read $.city and Select(x => new { N = x.Name })
  .OrderBy(y => y.N) read $.n. A second Select reads the first. A member the
  tree gives no value for is refused.
- Select(x => x.Address) read JSON_VALUE, NULL for an object, for every row.
  It is projected in memory now. An event query has nowhere to do that, so it
  refuses instead.
- A null check on an object or array read JSON_VALUE too, so x.Address == null
  matched every row and != none of them. Comparing one to a value is refused.
- An event's columns were matched by name, so ((Body)e.Data).Id read the id
  column. Only a member of the event itself can name one.
- AliasLocator aliased the literal "id" and nothing built around it, so
  x.Id.Length, x.Id.ToLower() and x.Id != null were ambiguous in a join.
- x.Created.DayOfWeek wrote a number into JSON under EnumStorage.AsString.

String equality:

- = pads, so == "abc" matched "abc  " and == "" matched spaces. A DATALENGTH
  residual beside the seekable = fixes it without costing the seek. Lists and
  collection elements get a sentinel on each side, because no one length
  belongs to a list.
- Measured on the document, the residual also stops an indexed == matching a
  value that nvarchar(max) storage truncated to the column's width.

String chains threw InvalidCastException. They translate or are refused now,
and comparing two members is refused rather than failing in the compiler.

Full text: term was varchar, so CJK and everything else outside the code page
was stored as ?, and a search scanned because the term binds as nvarchar.
Binding varchar seeks too, and turns the miss into a false match, so the column
is what changes. Nothing widens an existing table at first use: 8M tokens took
108 seconds, which would fail every request for the type. Applying the schema
does it, and a rebuild refuses until then.

Docs: child collections, operators and JSON indexes claimed what they do not do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Backfill a new full-text index a batch of documents at a time (#877)

Declaring an index over 8,000 documents of 7,000 characters timed out at 30
seconds in the first request, rolled back, and did the same in every request
after it, so the type could not be queried. It now takes 63 seconds in the
first request and 442ms in the second.

The list of documents is taken once and the batches work down it. Asking the
token table what is missing per batch took 133 seconds over 2,000 documents
against 12 for the one statement, because it is an anti-join over a heap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bind a string to a varchar locator as varchar, so its index seeks (#877)

#223's seek did not hold for a string member. A default Index(x => x.Name) is a
varchar(250) computed column, and an nvarchar parameter puts CONVERT_IMPLICIT on
it under a SQL collation: an Index Scan at 451 reads over 100,000 rows, against
an Index Seek at 3. x.Id == "abc" on a string id scanned its primary key the
same way. Only an ASCII value is bound as varchar, because a value outside the
code page would change.

Such a value goes to the document instead. The column stored it as ?, so
comparing through it found nothing, with no error.

A short IN stays nvarchar. Twenty-five of them over a varchar primary key ran
the optimizer out of resources (Msg 8623) on a shape that scanned happily.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* A nested lambda's own parameter isn't the row a query is reading (#875)

#899's nested lambda facts run over a captured list in memory, and the new
refusal took their parameter for the document.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant