Repository navigation
Keep a stream that folds to null, and mask archived and skipped events - #5797
Merged
Merged
Conversation
…sing archived rows Both bugs came out of reading the source for the JasperFx 3.0 lift and had never been reproduced. These fail against the code as it stands, reading the stored rows with SQL where the read under test could not see them: - A stream that folds to no aggregate has its whole history replaced by a Compacted<T> holding null. - An archiver closed over IDocumentSession is skipped, and anything Marten cannot call is skipped too, before the events it existed to keep are deleted. - Masking leaves archived rows out of IncludeStream and IncludeEvents, and skipped rows out of IncludeStream, and reports success. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cannot run Compaction wrote Compacted<T>(null) and deleted the rest, so a stream deleted by its own aggregate lost its history to an empty snapshot. Fisher's guard is the model: no aggregate means nothing to compact, so nothing is queued and the archiver isn't called. The archiver only matched IEventsArchiver<IDocumentOperations>. The interface is invariant, so one closed over IDocumentSession was skipped and the events were deleted anyway. Both closures are accepted now, as Polecat does, and anything else throws before a delete is queued. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t them Masking read through FetchStreamAsync and the default raw event query, and both leave archived rows out, so protected data on an archived stream stayed in place while the pass reported success. FetchStreamAsync also leaves skipped events out when skipping is on, which had the same result. QueryAllRawEvents with MaybeArchived reaches all of them. It loses the identity check FetchStreamAsync makes, where a Guid against a string store would match nothing and report success, so the stream sources make that check themselves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 2, 2026
Closed
This was referenced Nov 12, 2026
This was referenced Nov 12, 2026
Closed
This was referenced Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes two bugs in released Marten, one data loss and one privacy failure. Both came from reading the source during the JasperFx 3.0 lift, and neither had been reproduced. The first commit is tests only, so you can check it out to see them fail: 14 of 20 are red there, against real Postgres. The other 6 pin behaviour that has to survive the fix.
The 3.0 branch has corrected shared versions of both, but those only reach users when the stores adopt them in the .NET 11 train.
Compaction deleted the history of a stream that folds to null
CompactStreamAsyncwrotenew Compacted<T>(aggregate!, ...)with no null check, then replaced the last event and deleted the rest. A stream deleted by its own aggregate, throughShouldDeleteorDeleteEvent<T>, lost its whole history to an empty snapshot. A partial compaction whose cutoff landed on the delete did the same to the events below the cutoff, even if the stream carried on afterward.Now nothing is queued, the archiver isn't called, no watermark is recorded, and an information line is logged.
The archiver could be bypassed. Only
IEventsArchiver<IDocumentOperations>matched. The interface is invariant, so one closed overIDocumentSessionwas skipped and the events were deleted anyway. Both closures run now, and anything else throwsInvalidOperationExceptionbefore anything is queued. That is a behaviour change, but only for an archiver that was never being called.Masking skipped archived data and reported success
IncludeStreamread throughFetchStreamAsync, andIncludeEventsthrough the default raw event query. Both leave archived rows out, so protected data on an archived stream stayed in place.QueryAllRawEvents().Where(x => x.MaybeArchived())does work on this query: on archived, partitioned, conjoined and skipped rows, and beside a filter namingIsArchived. It has one catch. It loses the identity checkFetchStreamAsyncmakes, so a Guid on a string store matched nothing and reported success. The stream sources now make that check themselves, and a test pins it.Skipped events had the same problem. With
EnableEventSkippingInProjectionsOrSubscriptionson,IncludeStreammissed them too. The new read reaches them.Behaviour change:
IncludeEventsnow includes archived events. A filter that needs only live events should add!x.IsArchived.What users should re-run
Masking: anyone who relied on masking for data that was archived or skipped when they ran it should run the same
ApplyEventDataMaskingcall again after upgrading. Until then, that data is still in place in plain text.That call also runs the rules again over everything the first pass reached. If a rule isn't safe to run twice, such as one that draws a new pseudonym, add
x.IsArchivedto anIncludeEventsthat still names the same subject:Never use
x.IsArchivedas the whole filter here, because that masks every archived event in the store.Compaction: there is nothing to re-run, because the deleted events can't be rebuilt. To find affected streams, replace
<schema>:Those need restoring from a backup, or from wherever an archiver copied them. Anyone with an archiver closed over
IDocumentSessionshould also know that archiver never ran for past compactions, so that copy won't exist.Both notes are also in
docs/events/compacting.mdanddocs/events/protection.md.Testing
Run locally on net10.0, on Postgres 15.3:
MaybeArchived().EventSourcingTests: 2,705 passed, 5 failed, 28 skipped. All 5 failures aretesting_projections, which hard-codesDatabase=marten_testing. That database doesn't exist on this server, and the failure is3D000: database "marten_testing" does not exist.TenantPartitionedEventsTests: 287 passed, 2 skipped.Not run locally: net9.0, or any suite beyond those two.
🤖 Generated with Claude Code