Repository navigation
fix(persistence): reconcile a file's real columns against the schema on open - #396
Merged
Merged
Conversation
…on open An incident file records a schema version and `Migrate` runs only the steps numbered above it. That is sound only as long as every build that ever stamped version N applied exactly this lineage's steps 1..N — and a parallel feature branch numbers its own migrations too. A branch build (the unmerged Wasserförderung work, recognisable by the `wass_leitungen` table it leaves behind) stamped `schema_version = 19` onto real Einsatzdateien. Main's `ApplyV18` was then permanently skipped, and every read of those files died on `no such column: previous_zugfuehrer_count` with no way forward — the repair itself was already idempotent, only the `version < N` gate stood in front of it. `SchemaGuard` declares every column `ALTER TABLE ... ADD COLUMN` can restore — not a primary key, and either nullable or carrying a default — and adds the missing ones on every open, regardless of the marker, so an affected file repairs itself. The pass is deliberately one-directional: it drops nothing it does not recognise, so a file a branch widened keeps that branch's data and still opens on both builds. It also runs after the versioned steps and after the `version > CurrentVersion` guard, so a file from ahead of us is still refused untouched. The declared set is compared against a freshly migrated database in both directions by `SchemaReconciliationTests`, so a future migration that adds a column without declaring it fails CI rather than a user's Einsatz. Verified against copies of the six affected files: all four missing columns come back, `wass_leitungen` survives, and nothing changes in a healthy file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Thomas Müller <1005065+DeepDiver1975@users.noreply.github.com>
This was referenced Sep 18, 2026
This was referenced Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opening
Einsatz-27-07-2026.fwincidentfailed withSQLite Error 1: 'no such column: previous_zugfuehrer_count', even though the migration that adds that column exists and is correct.Root cause
The file carries
schema_version = 21—Migrations.CurrentVersion— but is missing both columnsApplyV18adds (force_units.zugfuehrer_count,force_unit_edits.previous_zugfuehrer_count). It does have V19'sunderground_floor_countand V20'sapartment_counts. V18 was jumped over, and the marker claims otherwise.The file also contains a table
wass_leitungen, which exists nowhere inmain.git log --all -S wass_leitungenplaces it solely on the unmergedfix/wasserfoerderung-karte-zoombranch (#150/#164). A build from that branch was run against real Einsatzdateien; it had its ownCurrentVersion = 19with its own meaning for V18/V19, and stamped19onto them. Whenmainlater opened one,if (version < 18)was already false, soApplyV18never ran — and never would.Migrategates every step behindversion < N. That is only sound if every build that ever stamped version N applied exactly this lineage's steps 1..N, which parallel feature branches break. The repair was already trivial —SchemaHelpers.AddColumnIfMissingis idempotent and no-ops on a healthy file. Only the version gate stood in front of it.Six of the 33 files in one real Lagebuch folder are affected:
zugfuehrer_countunderground_floor_countEinsatz-27-07-202620260829-194820260831-1238/-1300/-151920260906-1344The fix
SchemaGuarddeclares every column of the current schema thatALTER TABLE ... ADD COLUMNcan legally restore — not a primary key, and either nullable or carrying a default — andReconcileadds the missing ones on every open, regardless of the marker. The version marker becomes an optimisation rather than the sole source of truth about what is on disk. This is the same "no version marker, re-check on every open" shapeMasterDataStore.EnsureSchemaalready uses, and the general form ofApplyV17's existingif (!ColumnExists(...)) return;guard.Placement matters and is asserted by tests:
scba_trupps— only the final shape is worth widening.SetVersion, inside the existing transaction, so a failure rolls back with the marker un-advanced. No second transaction, no new commit point.version > CurrentVersionguard, which is untouched: a genuinely newer file is still refused withUnsupportedSchemaVersionException, now provably with its schema unmodified too.The pass is deliberately one-directional. It never drops a table or column it does not recognise, so a file a branch widened keeps that branch's data and still opens on both builds —
wass_leitungensurvives.Re-adding a column a newer build legitimately dropped cannot cause harm, because a file ahead of
mainnever reachesReconcile. Forversion <= CurrentVersion, this build'sLoadwill select that column, so restoring it is mandatory, not optional.Why the declared list, and why it can't silently rot
The alternative scopes both fail: listing only the columns migrations add via
AddColumnIfMissingcannot be drift-tested in the direction that matters (nothing fails if a future V22 forgets a line — this exact bug recurring), and listing every column forces entries that can never be repaired. The ALTER-addable set has a mechanical membership rule (pk = 0 AND (notnull = 0 OR dflt_value IS NOT NULL)) that a test evaluates against a freshly migrated database, giving exact bidirectional equality — and it is the largest set the reconciler can actually repair.Declared_schema_matches_a_freshly_migrated_databaseasserts that equality in both directions and names the offendingtable.columnwhen it breaks. A migration that adds a column without declaring it fails CI instead of a user's Einsatz.Tests
SchemaReconciliationTests(45 tests):Stamped_current_but_missing_an_additive_column_is_repaired_on_load— the reported file reproduced. Without the fix this fails with exactlyno such column: previous_zugfuehrer_count.A_foreign_branch_lineage_opens_and_keeps_its_own_tables— same damage pluswass_leitungen, stamped 19; the foreign table and its rows survive.Every_repairable_column_is_restored_after_being_dropped— a[Theory]over all 41 declared columns, so the next migration's column is covered the moment it is declared.Declared_schema_matches_a_freshly_migrated_database— the anti-drift guard.Reconciliation_does_not_modify_a_healthy_file—sqlite_masteris byte-identical after a secondMigrate.Plus an assertion in
MigrationForwardCompatTeststhat a refused newer-version file's schema is untouched, pinning the ordering invariant.Verification
Reconcilecall commented out, the two reproduction tests fail with the reported error and 41 of 43 theory cases fail.schema_version = 21,wass_leitungenpreserved.Scope
No UI change, so no screenshots.
docs/master-data.example.jsonis unaffected (different store,MasterDataSetunchanged), andmake samplesis not needed —CurrentVersionand the fresh-file schema are unchanged.AGENTS.mdgains a "Schema migrations" section: a branch owns version N only untilmaintakes it, branch builds must never be run against real Einsatzdateien, and a migration that adds a column must declare it inSchemaGuard.Known adjacent gap, not fixed here
MasterDataStore.EnsureSchemahas the analogous hole in a different shape: no version marker (good), but its reconciliation is three hand-writtenAddColumnIfMissinglines. A future column added to an existingmd_*CREATE TABLEwithout a matching line silently breaks pre-existing stores — and an anti-drift test cannot catch it there, because a fresh store always matches its ownCREATE TABLE. Tracked separately.Two further notes for reviewers:
IncidentRepository.SavecallsMigrate, so it is covered by the same call. Its INSERTs name columns explicitly, so extra foreign columns are harmless — unless a branch ever adds aNOT NULL-without-default column to a shared table, which no reconciler could repair. The affected files added only a table, so this is latent, not live.IncidentRepository.TryReadStateopens read-only and selectsincident_meta.statewithout migrating. Safe only because that column has existed since V1 — the one read path the reconciler cannot protect.🤖 Generated with Claude Code