You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(db): make directories migrations idempotent — journal-vs-schema drift boot-crashes the hub #270
Problem — Booting a current-fork server binary against a long-lived production database can crash-loop on duplicate column name: directories (observed on the erlich hub, 2026-08-30: a rebuilt binary exited ~1s after start, 29 systemd restarts, hub down). The database has both the directories columns AND complete migration-journal records — yet the runner sometimes replays an ADD COLUMN migration anyway, and the migration SQL itself is not idempotent, so any journal-vs-schema drift (e.g. a column added by a binary lineage that didn't journal it) becomes an unrecoverable boot crash instead of a self-heal.
Approach — Make schema-adding migrations tolerate reality: guard each ADD COLUMN with a column-existence check (skip + let the journal record it — the journal then reconciles to the actual schema on next boot). Apply to the two directories migrations first (both implicated in the incident); the same guard pattern is available for the other ADD COLUMN migrations as follow-up.
Scope — in: guarded directories migrations (session + project), boot verified against a production-DB copy · out: rewriting the migration runner's concurrency model (a separate suspected race — two subsystems migrating one file — filed as follow-up investigation with strace artifacts).
Acceptance Criteria
Booting against a copy of a drifted production DB (column present, journal row present) succeeds; before the fix it crashes with duplicate column name.
Booting against a copy with the journal row removed but the column present also succeeds and re-records the row (journal self-heal).
A genuinely fresh database still bootstraps and records all migrations (no behavior change on the happy path).
Important
Problem — Booting a current-fork server binary against a long-lived production database can crash-loop on
duplicate column name: directories(observed on the erlich hub, 2026-08-30: a rebuilt binary exited ~1s after start, 29 systemd restarts, hub down). The database has both thedirectoriescolumns AND complete migration-journal records — yet the runner sometimes replays an ADD COLUMN migration anyway, and the migration SQL itself is not idempotent, so any journal-vs-schema drift (e.g. a column added by a binary lineage that didn't journal it) becomes an unrecoverable boot crash instead of a self-heal.Approach — Make schema-adding migrations tolerate reality: guard each ADD COLUMN with a column-existence check (skip + let the journal record it — the journal then reconciles to the actual schema on next boot). Apply to the two
directoriesmigrations first (both implicated in the incident); the same guard pattern is available for the other ADD COLUMN migrations as follow-up.Scope — in: guarded directories migrations (session + project), boot verified against a production-DB copy · out: rewriting the migration runner's concurrency model (a separate suspected race — two subsystems migrating one file — filed as follow-up investigation with strace artifacts).
Acceptance Criteria
duplicate column name.