Repository navigation
fix(substrate): restart atelet when the RustFS keys change - #5938
Merged
Merged
Conversation
atelet reads the RustFS keys from rustfs-credentials into its environment at startup and transfers snapshots with them; on S3 there is no per-actor credential. When the keys were regenerated, RustFS restarted with the new ones but the running atelet kept the old ones, so every snapshot upload was refused (PermissionDenied) and kagent could not build the Agentic Engineer's golden actor, which failed #5905's merge-group deploy. A post-renderer annotates the DaemonSet for Stakater Reloader, which restarts it when that Secret changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
✅
|
Contributor
✅
|
The previous commit adds a postRenderStrategy and one Kustomize post-renderer to the substrate-atelet HelmRelease. Its rendered authorization fingerprint therefore moves, and the validator refused the head with exactly this one-line ledger delta (CI run 42557748805). The only change to the release is a DaemonSet metadata annotation, secret.reloader.stakater.com/reload: rustfs-credentials. No RBAC, service account, volume, security context or other grant-bearing field changes, so the new fingerprint is approved as is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
Authorization ledger re-approval (
Parked ( Unpark when: a fresh run of the failed jobs on Update 2026-11-09 ~07:20Z: the attempt-3 rerun on |
devantler
commented
Nov 9, 2026
devantler
left a comment
devantler
left a comment
Contributor
Author
There was a problem hiding this comment.
🤖 Generated by the Agentic Engineer
Self-review (fallback — CodeRabbit, Codex and Cursor Bugbot unavailable)
Reviewed commit: 7f038ce1be1d473b50c5733916517eff250bd67c
- CodeRabbit: rate limited at 2026-11-09T06:48Z.
- Codex: usage limit since 2026-11-09T06:06Z.
- Cursor Bugbot: usage limit since 2026-11-09T06:09Z.
What I checked:
- The post-renderer patch targets
DaemonSet/atelet. The live cluster runs a DaemonSet with exactly that name inate-system(6 ready), so the patch matches the chart's rendered object and is not a silent no-op. secret.reloader.stakater.com/reload: rustfs-credentialssits on the DaemonSet's own metadata, which is where Reloader reads it. The platform Reloader runs withreloadStrategy: annotationsand no namespace restriction, andreloader-reloaderis running.rustfs-credentialsis the ExternalSecret target the atelet release already reads, so a key regeneration now restarts atelet with the new keys.postRenderStrategy: nohookswith a kustomize post-renderer matches the pattern used by a dozen other releases onmain(actual-budget, umami, fleetdm, cilium, flagger, …).- The ledger line moves only the
substrate-ateletfingerprint, to the value the authorization check printed for this rendered release. Every required check is green at this head, including🔐 Validate Production Authorizationand the Go regressions. - What this means for a user: after a RustFS key rotation, agents can be built again without someone restarting atelet by hand.
Verdict: no P0/P1 findings
devantler
marked this pull request as ready for review
November 9, 2026 07:15
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
atelet, Substrate's node agent, reads the RustFS keys from
rustfs-credentialsinto its environment at startup and uploads actor snapshots with them; on the S3 backend there is no per-actor credential. When those keys were regenerated during #5903's rollout, RustFS restarted with the new keys while the running atelet pods kept the old ones. Every snapshot upload has since been refused (PermissionDenied ... storage refused the request with the actor's credential), so kagent cannot build the Agentic Engineer's golden actor, and #5905's merge-group deploy timed out waiting for the Agent to become ready.What
A post-renderer on the
substrate-ateletrelease annotates its DaemonSet withsecret.reloader.stakater.com/reload: rustfs-credentials, so Stakater Reloader restarts atelet whenever the keys change. The chart renders no hooks, so the release post-renders only its manifest stream (nohooks), as the post-renderer guard expects.The annotation does not restart pods that are already running, so the pods holding the stale key on prod need one manual restart; this PR prevents the next occurrence.
🤖 Generated with Claude Code
Part of #5893