Repository navigation
fix(deploy): a production deploy fails at its storage check whenever the nightly backup is copying a volume #4784
Description
🤖 Generated by the Agentic Engineer
Expected Behavior
A production deploy that starts while the nightly full backup is running should either wait for the backup's short-lived volumes to go away or ignore them, and then continue. It should fail only when a volume the last storage maintenance recorded is missing or changed.
Actual Behavior
The deploy's storage-maintenance step (go run ./scripts/maintain-longhorn-clients before-publish) compares the live number of Longhorn volumes with the number in the record left by the last maintenance, and stops on any difference:
Longhorn client maintenance HOLD: backing volume inventory changed
The nightly backup creates temporary volumes for its snapshot-based copies. While one exists the count is higher than recorded, so every deploy that reaches this step in that window fails before publishing and its pull request is removed from the merge queue.
Observed on 2026-10-12:
- 02:17:58–02:39:15 UTC: nightly full backup. Its three snapshot-based copies ran 02:18:36–02:24:36 and 02:37:00–02:38:52 UTC.
- 02:22 UTC: the deploy for fix(homepage): remove the two legacy site headings #4770 (run 38194496846) stopped at this step. The change in fix(homepage): remove the two legacy site headings #4770 touches no storage.
- 01:58 UTC, before the backup: the same step passed in the deploy for fix(github-config): accept the next reviewed publisher revision beside the current one #4767.
- 02:43 UTC, after the backup: the record lists 16 volumes and production has the same 16, by name.
Nothing was published and the recovery job restored main, so the cost is one evicted pull request and about 35 minutes of queue time per occurrence. With the queue as full as it was tonight (eight entries), one entry lands in this window every night.
Steps to Replicate
- Have a maintenance record present (
longhorn-client-maintenance, phaserestored). - Start a production deploy so that its storage-maintenance step runs while a snapshot-based backup copy is in progress.
- The step exits 1 with the message above.
Acceptance Criteria
- A volume that exists only because a backup copy is in progress does not fail the check. Either the check compares the recorded volumes by identity and ignores additional ones it can attribute to a running backup, or the step waits a bounded time for them to disappear.
- A recorded volume that is missing, replaced or changed in size, engine or source still fails the check as it does today.
- A test covers both cases with a recorded inventory plus one extra temporary volume.
- Verified on production: a deploy whose storage step runs during a backup copy completes.
Notes
The comparison is backingFieldsPreserved in scripts/maintain-longhorn-clients/plan.go (len(current) != len(before)). Related: #4723 (why tonight's backup was also incomplete), #4719 and #4775 (where this step was added and repaired).
Activity
devantler commentedon Oct 12, 2026
Not building a fix yet: #4744 removes the path that fails
I traced the failure in run 38194496846 before changing anything.
main.goenters that guard only whilemainstill declares a non-emptysystemManagedComponentsNodeSelectorfor Longhorn. In that guard, a journal in phaserestoredis compared with the live cluster bybackingFieldsPreserved(interruption.go), which requires the same number of volumes.main, ordinary deploys return from the guard before any inventory comparison, so a backup's temporary volumes can no longer fail them.So the next step depends on #4744:
restoredbranch of the guard, require every recorded volume to still exist with the same identity and format, and stop requiring the counts to match. The strict comparison stays for an active or interrupted maintenance.One more thing the count comparison would do, which I have not seen happen: while the guard is active, any new permanent volume added after the last maintenance would fail every later deploy in the same way, not only during a backup.
I released my claim on this issue; no branch or pull request exists for it.
devantler commentedon Oct 12, 2026
Resolved by #4744
#4744 merged and its production deploy succeeded (run 38200854113), including the real Longhorn client maintenance.
Read back after it:
mainnow declares an emptysystemManagedComponentsNodeSelectorfor Longhorn, and the live Longhorn setting is empty too.main, so a deploy can no longer stop there during the nightly backup.No code change was needed here. If a later change sets the selector again, the count comparison returns with it; the note above describes the small fix for that case.
devantler commentedon Oct 12, 2026
Follow-up evidence: the next deploy after #4744 (run 38204707153, #4768, merged 05:02 UTC) passed all four phases of the storage step, and the live maintenance record is in phase
complete. Not yet observed: a deploy whose storage step runs during a backup copy (next chance about 02:18 UTC tonight); reopen if that evicts a deploy again.