Repository navigation
fix: don't bump a deployment's updated_on when a reconcile leaves its status unchanged - #10049
Merged
Merged
Conversation
…ts status unchanged Since #9653 the ValidateDeploymentsWorker schedules ReconcileDeployment for every prod deployment twice an hour. For a running deployment the job always called UpdateDeploymentStatus(Running, ""), which sets updated_on = now() even when nothing changed. The runtime's configReloader uses updated_on as its "deployment changed" signal, so every reload forced a repo pull and a full re-parse of the project (or a controller restart for archive-backed projects), cancelling in-flight reconciles. The job now re-fetches the deployment and only writes the status when it actually differs. The re-fetch matters because UpdateDeploymentInner may have moved the status to updating in the DB without updating the job's copy. Side effect: the orphaned prod deployment teardown in ValidateDeploymentsWorker (updated_on older than 3h) can fire again, as it did before v0.89.
k-anshul
requested changes
Oct 21, 2026
k-anshul
left a comment
Member
There was a problem hiding this comment.
This doesn't seem safe on a high level.
A lot of validation/garbage collection in admin depends on updated_on so bumping it on reconcile seems correct.
May be we should revisit the approach commented here :
rill/runtime/config_reloader.go
Lines 28 to 32 in de14a79
updated_on when a reconcile leaves its status unchangedconfig_updated_on instead of updated_on
k-anshul
force-pushed
the
nishant/skip-unchanged-deployment-status
branch
from
October 22, 2026 06:50
a343b46 to
9fbdab8
Compare
config_updated_on instead of updated_onupdated_on when a reconcile leaves its status unchanged
k-anshul
approved these changes
Oct 22, 2026
k-anshul
pushed a commit
that referenced
this pull request
Oct 23, 2026
…ts status unchanged (#10049) Since #9653 the ValidateDeploymentsWorker schedules ReconcileDeployment for every prod deployment twice an hour. For a running deployment the job always called UpdateDeploymentStatus(Running, ""), which sets updated_on = now() even when nothing changed. The runtime's configReloader uses updated_on as its "deployment changed" signal, so every reload forced a repo pull and a full re-parse of the project (or a controller restart for archive-backed projects), cancelling in-flight reconciles. The job now re-fetches the deployment and only writes the status when it actually differs. The re-fetch matters because UpdateDeploymentInner may have moved the status to updating in the DB without updating the job's copy. Side effect: the orphaned prod deployment teardown in ValidateDeploymentsWorker (updated_on older than 3h) can fire again, as it did before v0.89.
nishantmonu51
added a commit
that referenced
this pull request
Oct 29, 2026
…ts status unchanged (#10049) Since #9653 the ValidateDeploymentsWorker schedules ReconcileDeployment for every prod deployment twice an hour. For a running deployment the job always called UpdateDeploymentStatus(Running, ""), which sets updated_on = now() even when nothing changed. The runtime's configReloader uses updated_on as its "deployment changed" signal, so every reload forced a repo pull and a full re-parse of the project (or a controller restart for archive-backed projects), cancelling in-flight reconciles. The job now re-fetches the deployment and only writes the status when it actually differs. The re-fetch matters because UpdateDeploymentInner may have moved the status to updating in the DB without updating the job's copy. Side effect: the orphaned prod deployment teardown in ValidateDeploymentsWorker (updated_on older than 3h) can fire again, as it did before v0.89. (cherry picked from commit b778dce)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ValidateDeploymentsWorkerschedulesReconcileDeploymentfor every prod deployment twice an hour. For a running deployment the job always calledUpdateDeploymentStatus(Running, ""), which setsupdated_on = now()even when nothing changed.configReloaderusesupdated_onas its "deployment changed" signal, so every reload forced a repo pull and a full re-parse of the project (a controller restart for archive-backed projects), cancelling in-flight model reconciles.UpdateDeploymentStatuswhen the status or status message actually differs. The re-fetch is needed becauseUpdateDeploymentInnermay have moved the status toUpdatingin the DB without updating the job's in-memory copy.UpdateDeployment,UpdateDeploymentDesiredStatusandUpdateDeploymentSafebumpupdated_onbefore the job runs, and the runtime compares variables directly.ValidateDeploymentsWorker(non-primary prod deployment, not stopped,updated_onolder than 3h) can fire again, as it did before v0.89. It could never trigger for a running deployment whileupdated_onwas refreshed every 30 minutes.TestRuntimeDeploymentsnow asserts that two no-op reconciles leaveupdated_onand the project parser's state version unchanged, and that a reconcile after a--prod-slotschange leaves the deploymentRunning.Steps to reproduce:
RUNNING.deployments.updated_onfor that deployment and the runtime's parser logs for its instance.updated_onstays constant and the project is not re-parsed.Actual:
updated_onmoves at every*/30run ofValidateDeploymentsWorker, and the runtime re-parses the whole project on each of its following config reloads.Checklist:
Developed in collaboration with Claude Code