Skip to content

fix(seed): simulation profile cannot validate above HistoryDays≈107 — lock sweep runs once, batched at 200 #638

Closed

Description

@mforce

Discovered while loading the simulation fixture into a local debug database at Simulation__HistoryDays=730.

What happens

seed --profile simulation exits 1 with a count mismatch that is only ever about the daily-entry lifecycle split:

Simulation seed completion check failed — the seed is short/partial and must NOT be marked complete:
… dailyEntries.submitted: expected 10, got 1070; dailyEntries.locked: expected 1446, got 633 …

Why

  • SimulationDataSeeder.SeedAsync calls lockSweep.RunAsync(ct) once (SimulationDataSeeder.cs:397).
  • DailyEntryLockSweep.LockDueEntriesAsync takes .Take(BatchSize) with BatchSize = 200 (DailyEntryLockSweep.cs:21) — one call locks at most 200 entries and returns; the sweep is designed to be re-polled by DurableJobWorker, which never starts in a run-then-exit verb.
  • ExpectedLockedEntryCount mirrors the sweep’s cutoff rule and therefore expects every eligible entry locked: FlockTopologyCount × (HistoryDays − LockAfterDays).

Those agree only while 2 × (HistoryDays − 7) ≤ 200, i.e. HistoryDays ≤ 107. Above that the assertion can never be satisfied, on any database, however clean.

The class header at SimulationDataSeeder.cs:1890 claims the expectations are "depth-robust by construction". They are, arithmetically — but the seeding is not, so the claim reads as broader than it is.

Blast radius today

Low, and that is why it went unnoticed: the harness default is HistoryDays=90 and the test fixture is 12, both under the ceiling. It only bites someone deliberately asking for a deep fixture.

Note the failure is also post-commit: the check runs in EmitManifestAsync after every row is written, in no transaction, so an over-ceiling run still leaves its data behind — just unlocked past the first 200, with no manifest and a red exit.

Suggested fix

Drain the sweep in the seeder rather than assuming one pass is enough — re-invoke until a pass locks nothing, with a bound so a genuine bug cannot spin forever. Do not raise BatchSize: that is a serving-path constant sized for the background worker, and changing it to satisfy a seeder would move the ceiling rather than remove it.

Then add a test at a depth above the current ceiling (e.g. HistoryDays=120) that asserts a green seed — one that fails today.

Docs

docs/runbooks/simulation-fixture-on-a-dev-database.md documents the ceiling and points here. Remove that section when this closes.

Activity

  1. mforce commented on Sep 1, 2026

    @mforce
    OwnerAuthor

    Two follow-ups found while drafting the runbook — both lower the severity, neither removes the defect.

    The ceiling is per run, not per database. Each invocation locks up to 200, so a re-run at a deeper HistoryDays only needs to lock what the increase newly made eligible, 2 × (new − old). Ramping in steps of ≤100 days (90 → 190 → 290 → …) converges green at any depth on the same durable anchor.

    And an overshot run is recoverable by repetition. The validation throws in EmitManifestAsync, after every row is committed and outside any transaction, so the rows survive the red exit; re-running the identical command locks the next 200. About 2 × (HistoryDays − 7) ÷ 200 runs walks it green.

    Both are read off the code, not yet observed on a real run — the runbook says so and its drill exercises them.

    So the user-visible cost is a confusing red exit and a manual ramp, not an unreachable depth. The fix (drain the sweep in SeedAsync) is still worth doing: nothing about the failure message points at the ramp, and the "depth-robust by construction" comment at SimulationDataSeeder.cs:1890 actively points away from it.

  2. added a commit that references this issue on Sep 2, 2026
    730fa23
  3. added a commit that references this issue on Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions