Repository navigation
Give event reads an opt-in to archived events - #899
Merged
Merged
Conversation
Polecat filtered archived events out of event LINQ but left no way in: its is_archived = 0 sat beside a predicate naming IEvent.IsArchived, so x.IsArchived matched nothing, and there was no public MaybeArchived(). A predicate that filters rows now opts in by calling MaybeArchived() or by naming IsArchived, which replaces the default term, as on Marten and Fisher. OrderBy and Select name the member without filtering, so they leave the default on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ArchivalPredicates accepted any IEvent-typed parameter, so a nested lambda over an in-memory list counted as the event being filtered. In `captured.Where(e => e.IsArchived).Select(e => e.Id).Contains(x.Id)` the parser evaluates the list up front, and the check still walked into it and dropped is_archived = 0, returning archived rows with no opt-in. Compare against the predicate's own parameter instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jeremydmiller
added a commit
that referenced
this pull request
Nov 2, 2026
… full text seeks (#875, #877) (#913) * LINQ answers what .NET would for spaces, objects and projections (#875, #877) Members a query names but the JSON does not: - An operator after a Select reads the projection. Select(x => x.Address) .Where(a => a.City == "x") read $.city and Select(x => new { N = x.Name }) .OrderBy(y => y.N) read $.n. A second Select reads the first. A member the tree gives no value for is refused. - Select(x => x.Address) read JSON_VALUE, NULL for an object, for every row. It is projected in memory now. An event query has nowhere to do that, so it refuses instead. - A null check on an object or array read JSON_VALUE too, so x.Address == null matched every row and != none of them. Comparing one to a value is refused. - An event's columns were matched by name, so ((Body)e.Data).Id read the id column. Only a member of the event itself can name one. - AliasLocator aliased the literal "id" and nothing built around it, so x.Id.Length, x.Id.ToLower() and x.Id != null were ambiguous in a join. - x.Created.DayOfWeek wrote a number into JSON under EnumStorage.AsString. String equality: - = pads, so == "abc" matched "abc " and == "" matched spaces. A DATALENGTH residual beside the seekable = fixes it without costing the seek. Lists and collection elements get a sentinel on each side, because no one length belongs to a list. - Measured on the document, the residual also stops an indexed == matching a value that nvarchar(max) storage truncated to the column's width. String chains threw InvalidCastException. They translate or are refused now, and comparing two members is refused rather than failing in the compiler. Full text: term was varchar, so CJK and everything else outside the code page was stored as ?, and a search scanned because the term binds as nvarchar. Binding varchar seeks too, and turns the miss into a false match, so the column is what changes. Nothing widens an existing table at first use: 8M tokens took 108 seconds, which would fail every request for the type. Applying the schema does it, and a rebuild refuses until then. Docs: child collections, operators and JSON indexes claimed what they do not do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Backfill a new full-text index a batch of documents at a time (#877) Declaring an index over 8,000 documents of 7,000 characters timed out at 30 seconds in the first request, rolled back, and did the same in every request after it, so the type could not be queried. It now takes 63 seconds in the first request and 442ms in the second. The list of documents is taken once and the batches work down it. Asking the token table what is missing per batch took 133 seconds over 2,000 documents against 12 for the one statement, because it is an anti-join over a heap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bind a string to a varchar locator as varchar, so its index seeks (#877) #223's seek did not hold for a string member. A default Index(x => x.Name) is a varchar(250) computed column, and an nvarchar parameter puts CONVERT_IMPLICIT on it under a SQL collation: an Index Scan at 451 reads over 100,000 rows, against an Index Seek at 3. x.Id == "abc" on a string id scanned its primary key the same way. Only an ASCII value is bound as varchar, because a value outside the code page would change. Such a value goes to the document instead. The column stored it as ?, so comparing through it found nothing, with no error. A short IN stays nvarchar. Twenty-five of them over a varchar primary key ran the optimizer out of resources (Msg 8623) on a shape that scanned happily. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * A nested lambda's own parameter isn't the row a query is reading (#875) #899's nested lambda facts run over a captured list in memory, and the new refusal took their parameter for the document. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Polecat filtered archived events out of event reads but gave callers no way in. This adds the opt-in, with Marten's semantics and Fisher's shape from JasperFx/fisher#511.
What changes for callers
IsArchivedstarts matching rows it never matched before. The defaultis_archived = 0used to sit beside any predicate, sox.IsArchivedcould only ever match nothing. A predicate namingIEvent.IsArchivednow replaces that term and answers for itself.MaybeArchived()is new, inPolecat.Events.Archiving, for live and archived events together.IsArchivedis unchanged, and so is!x.IsArchivedon its own.IEventx.IsArchived,x.IsArchived == truex.IsArchived || yyy!x.IsArchived || yyMasking is unaffected: #888 landed first and gave masking its own read that keeps archived events, so
IncludeEventsdoesn't depend on this opt-in, and a filter namingIsArchivedstill narrows it. Merging main into this branch kept #888'sincludeArchivedprovider flag beside the new check.Reads this covers
QueryAllRawEvents()andQueryRawEvents(): both opt-ins work, in anyWhereand in a terminal's inline predicate (CountAsync,AnyAsync,FirstOrDefaultAsync).AggregateToAsyncfolds what its query matched.QueryEventData<T>andQueryRawEventDataOnly<T>: these share the provider and keep the default filter. Their predicates are over the event body, so anIsArchivedmember on a body says nothing about the row, and there is no opt-in on them.QueryStreamStates(): unchanged, because it has its own provider and no implicit filter. This is now pinned by a test.OrderBy,Select: not an opt-in, because they filter nothing.Not the saved patch
The 25-line patch noted on #1104 turns the compliance facts green, but it walks the whole expression, so
OrderBy(x => x.IsArchived)andSelect(x => x.IsArchived)silently returned archived events. It had noMaybeArchived()either. HereLinqQueryParserrecords the predicates that filter rows, and the check only looks at those. The check itself,ArchivalPredicates, is Fisher's. I ran the new tests with the patch's whole-expression walk swapped in to confirm: theOrderByandSelecttests fail.A nested lambda is not the row either. The check first accepted any
IEvent-typed parameter. Incaptured.Where(e => e.IsArchived).Select(e => e.Id).Contains(x.Id)the parser evaluates the list up front, but the check still walked into it, droppedis_archived = 0and returned archived rows with no opt-in. It now matches only the predicate's own parameter.captured.Any(e => e.IsArchived)never got that far, because the parser refuses it. Fisher'sArchivalPredicatesin JasperFx/fisher#511 was the source of this check and has the same shape.The
IsMaybeDeletedguard on the default term is gone. It could never fire, because soft-delete operators are refused over events.Validation
Compliance, with a local pack.
v2.97.0, the version Polecat pins, plus #1104'sStreamArchivingCompliance.cs, packed as2.97.0-local.1104. A real 3.0 pack can't reach Polecat yet. #1104's other two files are comment-only and did not apply to the tag, so they were left out.stream_archiving_complianceorigin/mainevents_of_an_archived_stream_report_themselves_as_archived,a_predicate_naming_is_archived_reads_only_the_archived_eventsTests in this PR. There are 17 tests across
archived_events_are_opt_in_testsand the conjoined tenancy class, passing on net9.0 and net10.0 against SQL Server. With the old condition restored, 11 of the first 16 fail, and the other 5 pin behaviour that must not change. The 17th is the nested lambda test, which failed with 2 archived rows before the check was narrowed.Affected classes. 938 passed, 0 failed on net10.0: all of
Polecat.Tests.Linq, plus the archive, raw event, event data, stream state, masking,AggregateToand event query classes.After the nested lambda fix. At the head of this branch,
stream_archiving_complianceis 27 passed, 0 failed, usingv2.97.0's suite with #1104'sStreamArchivingCompliance.csapplied throughComplianceSourceDir.Polecat.Tests.EventsandPolecat.Tests.Complianceon net10.0: 1,375 passed, 5 skipped, 3 failed. The 3 aretenant_partitioned_unseen_tenant_tests.fetch_for_writing_of_a_new_stream_in_a_session_holding_its_transaction(RepeatableRead, Serializable, Snapshot). They fail the same way onorigin/mainat 8545249, because the test runsALTER DATABASE CURRENTand the default local connection points atmaster. This branch did not cause them.The whole suite was not run locally. It takes about 2 hours, so that is left to CI.
Merging with #888
#888 is still open, so this branches from
main. Either can land first. They conflict on one line, the default-term condition, and the resolution is:I tested that merge, again after the nested lambda fix: 169 passed, 0 failed across the masking, compaction, archive and raw event classes, with #1104's suite applied. That includes #888's own
IncludeEvents(e => e.IsArchived)and!e.IsArchivedtests.🤖 Generated with Claude Code