Skip to content

Keep a stream that folds to null, and mask archived and skipped events - #5797

Merged
jeremydmiller merged 3 commits into
masterfrom
fix/compaction-null-and-archived-masking
Nov 2, 2026
Merged

jeremydmiller merged 3 commits into
masterfrom
fix/compaction-null-and-archived-masking

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Fixes two bugs in released Marten, one data loss and one privacy failure. Both came from reading the source during the JasperFx 3.0 lift, and neither had been reproduced. The first commit is tests only, so you can check it out to see them fail: 14 of 20 are red there, against real Postgres. The other 6 pin behaviour that has to survive the fix.

The 3.0 branch has corrected shared versions of both, but those only reach users when the stores adopt them in the .NET 11 train.

Compaction deleted the history of a stream that folds to null

CompactStreamAsync wrote new Compacted<T>(aggregate!, ...) with no null check, then replaced the last event and deleted the rest. A stream deleted by its own aggregate, through ShouldDelete or DeleteEvent<T>, lost its whole history to an empty snapshot. A partial compaction whose cutoff landed on the delete did the same to the events below the cutoff, even if the stream carried on afterward.

Now nothing is queued, the archiver isn't called, no watermark is recorded, and an information line is logged.

The archiver could be bypassed. Only IEventsArchiver<IDocumentOperations> matched. The interface is invariant, so one closed over IDocumentSession was skipped and the events were deleted anyway. Both closures run now, and anything else throws InvalidOperationException before anything is queued. That is a behaviour change, but only for an archiver that was never being called.

Masking skipped archived data and reported success

IncludeStream read through FetchStreamAsync, and IncludeEvents through the default raw event query. Both leave archived rows out, so protected data on an archived stream stayed in place.

QueryAllRawEvents().Where(x => x.MaybeArchived()) does work on this query: on archived, partitioned, conjoined and skipped rows, and beside a filter naming IsArchived. It has one catch. It loses the identity check FetchStreamAsync makes, so a Guid on a string store matched nothing and reported success. The stream sources now make that check themselves, and a test pins it.

Skipped events had the same problem. With EnableEventSkippingInProjectionsOrSubscriptions on, IncludeStream missed them too. The new read reaches them.

Behaviour change: IncludeEvents now includes archived events. A filter that needs only live events should add !x.IsArchived.

What users should re-run

Masking: anyone who relied on masking for data that was archived or skipped when they ran it should run the same ApplyEventDataMasking call again after upgrading. Until then, that data is still in place in plain text.

That call also runs the rules again over everything the first pass reached. If a rule isn't safe to run twice, such as one that draws a new pseudonym, add x.IsArchived to an IncludeEvents that still names the same subject:

x.IncludeEvents(e => e.StreamId == streamId && e.IsArchived)

Never use x.IsArchived as the whole filter here, because that masks every archived event in the store.

Compaction: there is nothing to re-run, because the deleted events can't be rebuilt. To find affected streams, replace <schema>:

select stream_id, version, tenant_id
from <schema>.mt_events
where mt_dotnet_type like 'JasperFx.Events.Compacted%'
  and 'null'::jsonb in (data->'Snapshot', data->'snapshot');

Those need restoring from a backup, or from wherever an archiver copied them. Anyone with an archiver closed over IDocumentSession should also know that archiver never ran for past compactions, so that copy won't exist.

Both notes are also in docs/events/compacting.md and docs/events/protection.md.

Testing

Run locally on net10.0, on Postgres 15.3:

  • New tests: 20 of 20 pass. Every masking assertion on archived rows reads the stored row with SQL, except the tenant test, which reads through MaybeArchived().
  • EventSourcingTests: 2,705 passed, 5 failed, 28 skipped. All 5 failures are testing_projections, which hard-codes Database=marten_testing. That database doesn't exist on this server, and the failure is 3D000: database "marten_testing" does not exist.
  • TenantPartitionedEventsTests: 287 passed, 2 skipped.
  • The detection query was checked on a real row, plus inline samples of null and healthy snapshots in both key casings.

Not run locally: net9.0, or any suite beyond those two.

🤖 Generated with Claude Code

jeremydmiller and others added 3 commits November 1, 2026 06:32
…sing archived rows

Both bugs came out of reading the source for the JasperFx 3.0 lift and had never been reproduced. These
fail against the code as it stands, reading the stored rows with SQL where the read under test could not
see them:

- A stream that folds to no aggregate has its whole history replaced by a Compacted<T> holding null.
- An archiver closed over IDocumentSession is skipped, and anything Marten cannot call is skipped too,
  before the events it existed to keep are deleted.
- Masking leaves archived rows out of IncludeStream and IncludeEvents, and skipped rows out of
  IncludeStream, and reports success.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cannot run

Compaction wrote Compacted<T>(null) and deleted the rest, so a stream deleted by its own aggregate lost
its history to an empty snapshot. Fisher's guard is the model: no aggregate means nothing to compact, so
nothing is queued and the archiver isn't called.

The archiver only matched IEventsArchiver<IDocumentOperations>. The interface is invariant, so one closed
over IDocumentSession was skipped and the events were deleted anyway. Both closures are accepted now, as
Polecat does, and anything else throws before a delete is queued.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t them

Masking read through FetchStreamAsync and the default raw event query, and both leave archived rows out,
so protected data on an archived stream stayed in place while the pass reported success. FetchStreamAsync
also leaves skipped events out when skipping is on, which had the same result.

QueryAllRawEvents with MaybeArchived reaches all of them. It loses the identity check FetchStreamAsync
makes, where a Guid against a string store would match nothing and report success, so the stream sources
make that check themselves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit 8111911 into master Nov 2, 2026
32 checks passed
@jeremydmiller
jeremydmiller deleted the fix/compaction-null-and-archived-masking branch November 2, 2026 15:17
This was referenced Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant