Skip to content

fix(persistence): reconcile a file's real columns against the schema on open - #396

Merged
DeepDiver1975 merged 1 commit into
mainfrom
fix/schema-reconciliation
Sep 18, 2026
Merged

DeepDiver1975 merged 1 commit into
mainfrom
fix/schema-reconciliation

Conversation

@DeepDiver1975

Copy link
Copy Markdown
Contributor

Opening Einsatz-27-07-2026.fwincident failed with SQLite Error 1: 'no such column: previous_zugfuehrer_count', even though the migration that adds that column exists and is correct.

Root cause

The file carries schema_version = 21 — Migrations.CurrentVersion — but is missing both columns ApplyV18 adds (force_units.zugfuehrer_count, force_unit_edits.previous_zugfuehrer_count). It does have V19's underground_floor_count and V20's apartment_counts. V18 was jumped over, and the marker claims otherwise.

The file also contains a table wass_leitungen, which exists nowhere in main. git log --all -S wass_leitungen places it solely on the unmerged fix/wasserfoerderung-karte-zoom branch (#150/#164). A build from that branch was run against real Einsatzdateien; it had its own CurrentVersion = 19 with its own meaning for V18/V19, and stamped 19 onto them. When main later opened one, if (version < 18) was already false, so ApplyV18 never ran — and never would.

Migrate gates every step behind version < N. That is only sound if every build that ever stamped version N applied exactly this lineage's steps 1..N, which parallel feature branches break. The repair was already trivial — SchemaHelpers.AddColumnIfMissing is idempotent and no-ops on a healthy file. Only the version gate stood in front of it.

Six of the 33 files in one real Lagebuch folder are affected:

File version zugfuehrer_count underground_floor_count
Einsatz-27-07-2026 21 ✗ ✓
20260829-1948 19 ✗ ✗
20260831-1238 / -1300 / -1519 19 ✗ ✗
20260906-1344 19 ✗ ✓

The fix

SchemaGuard declares every column of the current schema that ALTER TABLE ... ADD COLUMN can legally restore — not a primary key, and either nullable or carrying a default — and Reconcile adds the missing ones on every open, regardless of the marker. The version marker becomes an optimisation rather than the sole source of truth about what is on disk. This is the same "no version marker, re-check on every open" shape MasterDataStore.EnsureSchema already uses, and the general form of ApplyV17's existing if (!ColumnExists(...)) return; guard.

Placement matters and is asserted by tests:

  • After the versioned steps, because V3/V6/V17 drop-and-rebuild scba_trupps — only the final shape is worth widening.
  • Before SetVersion, inside the existing transaction, so a failure rolls back with the marker un-advanced. No second transaction, no new commit point.
  • After the version > CurrentVersion guard, which is untouched: a genuinely newer file is still refused with UnsupportedSchemaVersionException, now provably with its schema unmodified too.

The pass is deliberately one-directional. It never drops a table or column it does not recognise, so a file a branch widened keeps that branch's data and still opens on both builds — wass_leitungen survives.

Re-adding a column a newer build legitimately dropped cannot cause harm, because a file ahead of main never reaches Reconcile. For version <= CurrentVersion, this build's Load will select that column, so restoring it is mandatory, not optional.

Why the declared list, and why it can't silently rot

The alternative scopes both fail: listing only the columns migrations add via AddColumnIfMissing cannot be drift-tested in the direction that matters (nothing fails if a future V22 forgets a line — this exact bug recurring), and listing every column forces entries that can never be repaired. The ALTER-addable set has a mechanical membership rule (pk = 0 AND (notnull = 0 OR dflt_value IS NOT NULL)) that a test evaluates against a freshly migrated database, giving exact bidirectional equality — and it is the largest set the reconciler can actually repair.

Declared_schema_matches_a_freshly_migrated_database asserts that equality in both directions and names the offending table.column when it breaks. A migration that adds a column without declaring it fails CI instead of a user's Einsatz.

Tests

SchemaReconciliationTests (45 tests):

  1. Stamped_current_but_missing_an_additive_column_is_repaired_on_load — the reported file reproduced. Without the fix this fails with exactly no such column: previous_zugfuehrer_count.
  2. A_foreign_branch_lineage_opens_and_keeps_its_own_tables — same damage plus wass_leitungen, stamped 19; the foreign table and its rows survive.
  3. Every_repairable_column_is_restored_after_being_dropped — a [Theory] over all 41 declared columns, so the next migration's column is covered the moment it is declared.
  4. Declared_schema_matches_a_freshly_migrated_database — the anti-drift guard.
  5. Reconciliation_does_not_modify_a_healthy_file — sqlite_master is byte-identical after a second Migrate.

Plus an assertion in MigrationForwardCompatTests that a refused newer-version file's schema is untouched, pinning the ordering invariant.

Verification

  • Red first for the right reason: with the Reconcile call commented out, the two reproduction tests fail with the reported error and 41 of 43 theory cases fail.
  • Full solution after: 1382 passed, 0 failed, 1 skipped, no analyzer warnings.
  • Against copies of the real affected files (originals untouched): all three load, all four missing columns come back, schema_version = 21, wass_leitungen preserved.

Scope

No UI change, so no screenshots. docs/master-data.example.json is unaffected (different store, MasterDataSet unchanged), and make samples is not needed — CurrentVersion and the fresh-file schema are unchanged.

AGENTS.md gains a "Schema migrations" section: a branch owns version N only until main takes it, branch builds must never be run against real Einsatzdateien, and a migration that adds a column must declare it in SchemaGuard.

Known adjacent gap, not fixed here

MasterDataStore.EnsureSchema has the analogous hole in a different shape: no version marker (good), but its reconciliation is three hand-written AddColumnIfMissing lines. A future column added to an existing md_* CREATE TABLE without a matching line silently breaks pre-existing stores — and an anti-drift test cannot catch it there, because a fresh store always matches its own CREATE TABLE. Tracked separately.

Two further notes for reviewers:

  • IncidentRepository.Save calls Migrate, so it is covered by the same call. Its INSERTs name columns explicitly, so extra foreign columns are harmless — unless a branch ever adds a NOT NULL-without-default column to a shared table, which no reconciler could repair. The affected files added only a table, so this is latent, not live.
  • IncidentRepository.TryReadState opens read-only and selects incident_meta.state without migrating. Safe only because that column has existed since V1 — the one read path the reconciler cannot protect.

🤖 Generated with Claude Code

…on open

An incident file records a schema version and `Migrate` runs only the steps
numbered above it. That is sound only as long as every build that ever stamped
version N applied exactly this lineage's steps 1..N — and a parallel feature
branch numbers its own migrations too. A branch build (the unmerged
Wasserförderung work, recognisable by the `wass_leitungen` table it leaves
behind) stamped `schema_version = 19` onto real Einsatzdateien. Main's
`ApplyV18` was then permanently skipped, and every read of those files died on
`no such column: previous_zugfuehrer_count` with no way forward — the repair
itself was already idempotent, only the `version < N` gate stood in front of it.

`SchemaGuard` declares every column `ALTER TABLE ... ADD COLUMN` can restore —
not a primary key, and either nullable or carrying a default — and adds the
missing ones on every open, regardless of the marker, so an affected file
repairs itself. The pass is deliberately one-directional: it drops nothing it
does not recognise, so a file a branch widened keeps that branch's data and
still opens on both builds. It also runs after the versioned steps and after
the `version > CurrentVersion` guard, so a file from ahead of us is still
refused untouched.

The declared set is compared against a freshly migrated database in both
directions by `SchemaReconciliationTests`, so a future migration that adds a
column without declaring it fails CI rather than a user's Einsatz. Verified
against copies of the six affected files: all four missing columns come back,
`wass_leitungen` survives, and nothing changes in a healthy file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant