Repository navigation
Restore volume attachment on compute-only workers #4718
Description
🤖 Generated by the Agentic Engineer
Delivered on: 2026-10-12
Problem
A protected node rollout exposed an integration mismatch: newly provisioned compute workers correctly omit the replica-storage label, but the platform uses that same label to select Longhorn volume-client components. Database standby Pods can schedule there but cannot attach their existing volumes. The primaries remain available and archiving continues; reduced redundancy and blocked rollout are not a clean state.
Intended outcome
Separate volume attachment from replica placement. Eligible compute workers need the volume-client components, without enabling replica storage there. Preserve the existing baseline-only replica policies, default-disk gating, data, disruption budgets, and graceful shutdown. No forced detachment, Pod deletion, storage-claim replacement, or direct production mutation.
Acceptance criteria
- A regression fails on the shipping selector mismatch, then passes after the narrow declarative repair.
- Render the pinned chart to prove actual client placement, while replica placement and unrelated security/resource settings stay unchanged.
- Pass current-head review and CI and deliver through protected deployment.
- Verify driver registration on eligible workers, all database replicas ready, volume health, and the init-image rollout in separated live reads.
This blocks the deployment of #4197. It is distinct from #4649's scheduling-priority problem.
Longhorn documents the distinction between restricting replica disks and restricting all components: https://longhorn.io/kb/tip-only-use-storage-on-a-set-of-nodes/
Part of #2043.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
chore: agent-claim #4718 nonce=93b30d4a1b90f8a218723ae33a0918c8
chore: agent-claim #4718 nonce=378c94b887f7d207850cb7cb4cd1fcd0
chore: agent-claim #4718 nonce=9a8a92473eb9984e5193917da8839d03
devantler commentedon Oct 11, 2026
The protected rollout for #4744 was evicted for failed checks at 17:06 UTC. Deployment run 38157208826, at merge-group commit
88af2883ece50e048a2a16283e7a8e9903b073de, stopped in the reviewed Longhorn maintenance preflight before publishing the candidate manifests. The database health and ownership guard returned HOLD; its failure-path maintenance restore passed.The restore-main job completed successfully at 17:31 UTC. Both the production redeploy and the check for objects left by the failed deployment passed. This closes the pending terminal restore evidence.
A fresh complete read-only database inventory still fails the healthy baseline. The retained-database incident remains tracked in #4266; #4759 is its repair and has successful native CI at
e164e0ce76bde108e9e5a255e4becc76d6ada4b7, but current-head review and delivery remain pending.Resume only after #4759 is reviewed, delivered and its database recovery is verified. Rebind the current source, head, checks and review before any queue admission. The original driver-registration, volume-health and database-redundancy acceptance checks remain required; #4197 stays dependent on this issue.
chore: agent-claim #4718 nonce=436c9bcc4213b3338d470b3799eb327c
chore: agent-claim #4718 nonce=a0dc8cb90c0b789b98c4347ee49893f3
devantler commentedon Oct 11, 2026
The latest protected attempt for #4744 failed at a different boundary from the earlier database preflight. Native logs from deployment and recovery both show that an oversized command argument prevented the maintenance journal update. The queue removed the unchanged source head
56ada44d7e5600153b377e3e53040d9f87970d7cat 21:59 UTC, and the PR returned to draft.The effect now reaches another queue entry: the publisher deployment for #4767 refused legacy publication after maintenance had begun. It was evicted at 22:15 UTC. Retrying either unchanged head cannot establish recovery.
#4775 is the newly published draft repair for the argument transport. Its delivery, maintenance-journal recovery, protected storage maintenance and independent retained-data checks remain separate acceptance gates. This run performed no production mutation or requeue and does not claim those gates passed. #4197 remains dependent on this issue.
2 remaining items
chore: agent-claim #4718 nonce=4015a4c382a060936b6451140f7f79e8
chore: agent-claim #4718 nonce=38bd3284c6d0269d166c1880410ba68d
chore: agent-claim #4718 nonce=9880cf0cb50793765409520ecb998b2c
devantler commentedon Oct 12, 2026
The volume-client repair in #4744 is merged; GitHub records its delivery on 2026-10-12. This issue remains open because the final acceptance check also requires the protected init-image rollout tracked by #4194. Its delivery PR #4197 is still a draft outside the merge queue, so that final check cannot run yet.
The prerequisite is now a native blocked-by relationship. The card returns from Verifying to Ready as a continuation of started work; its delivery date, acceptance criteria and earlier evidence are retained. Resume verification when #4194 has delivered its protected rollout and the evidence is available, with space in Verifying. Recheck driver registration, database readiness, volume health and the deployed init image separately before deciding completion.
No new runtime acceptance, rollout or production change is claimed.
chore: agent-claim #4718 nonce=240827d25aaaaf349a08d8b77451a8b1
devantler commentedon Oct 13, 2026
The pending rollout verification now passes. #4197 merged on 2026-10-13, and the protected main deployment passed the actual Talos image readback at 10:35 UTC. Independent OIDC reads this afternoon also match the corrected image with healthy, schedulable nodes.
The remaining storage checks pass in separated live reads: the volume-client driver is registered on every eligible worker, both client DaemonSets are fully ready, all database replicas are ready with settled primaries and healthy continuous archiving, and every attached Longhorn volume is healthy. The compute-only worker has no replica disks and remains ineligible for replica scheduling. The database claim identities and bindings are unchanged between the separated reads, and none is being deleted.
Together with the reviewed regression, rendered-chart proof, #4744 merge and protected maintenance/restoration evidence, this completes this issue's acceptance criteria. Closing it as completed without changing the existing delivery date or erasing the earlier failed attempts.
#4194 remains open for the distinct kernel audit-message clearance checks. This closure confirms volume attachment, preserved replica-placement restrictions and the deployed init image; it does not claim that every init error or Coroot counter is clear. No direct production mutation was performed during this verification.