Skip to content

fix: don't bump a deployment's updated_on when a reconcile leaves its status unchanged - #10049

Merged
k-anshul merged 1 commit into
mainfrom
nishant/skip-unchanged-deployment-status
Oct 22, 2026
Merged

k-anshul merged 1 commit into
mainfrom
nishant/skip-unchanged-deployment-status

Conversation

@nishantmonu51

@nishantmonu51 nishantmonu51 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator
  • Since Move resource checks to deployment reconciliation #9653, ValidateDeploymentsWorker schedules ReconcileDeployment for every prod deployment twice an hour. For a running deployment the job always called UpdateDeploymentStatus(Running, ""), which sets updated_on = now() even when nothing changed.
  • The runtime's configReloader uses updated_on as its "deployment changed" signal, so every reload forced a repo pull and a full re-parse of the project (a controller restart for archive-backed projects), cancelling in-flight model reconciles.
  • The job now re-fetches the deployment and only calls UpdateDeploymentStatus when the status or status message actually differs. The re-fetch is needed because UpdateDeploymentInner may have moved the status to Updating in the DB without updating the job's in-memory copy.
  • Real changes still propagate: UpdateDeployment, UpdateDeploymentDesiredStatus and UpdateDeploymentSafe bump updated_on before the job runs, and the runtime compares variables directly.
  • Behavior change: the orphaned prod deployment teardown in ValidateDeploymentsWorker (non-primary prod deployment, not stopped, updated_on older than 3h) can fire again, as it did before v0.89. It could never trigger for a running deployment while updated_on was refreshed every 30 minutes.
  • TestRuntimeDeployments now asserts that two no-op reconciles leave updated_on and the project parser's state version unchanged, and that a reconcile after a --prod-slots change leaves the deployment Running.

Steps to reproduce:

  1. Deploy a project to Rill Cloud and wait for its prod deployment to reach RUNNING.
  2. Watch deployments.updated_on for that deployment and the runtime's parser logs for its instance.
  3. Make no changes to the project, its environment variables or its deployment for an hour.
  4. Expected: updated_on stays constant and the project is not re-parsed.
    Actual: updated_on moves at every */30 run of ValidateDeploymentsWorker, and the runtime re-parses the whole project on each of its following config reloads.

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

Developed in collaboration with Claude Code

…ts status unchanged

Since #9653 the ValidateDeploymentsWorker schedules ReconcileDeployment for every prod deployment twice an hour.
For a running deployment the job always called UpdateDeploymentStatus(Running, ""), which sets updated_on = now() even when nothing changed.
The runtime's configReloader uses updated_on as its "deployment changed" signal, so every reload forced a repo pull and a full re-parse of the project (or a controller restart for archive-backed projects), cancelling in-flight reconciles.

The job now re-fetches the deployment and only writes the status when it actually differs.
The re-fetch matters because UpdateDeploymentInner may have moved the status to updating in the DB without updating the job's copy.

Side effect: the orphaned prod deployment teardown in ValidateDeploymentsWorker (updated_on older than 3h) can fire again, as it did before v0.89.

@k-anshul k-anshul left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem safe on a high level.
A lot of validation/garbage collection in admin depends on updated_on so bumping it on reconcile seems correct.
May be we should revisit the approach commented here :

// to avoid repo handshake refresh on each reload we track last updatedon of each deployment
// if the deployment has not changed skip the repo pull
//
// this can further be optimised to only check for properties that affect repo like url, branch etc but keeping it simple for now
updatedOn map[string]time.Time

@k-anshul k-anshul changed the title fix: don't bump a deployment's updated_on when a reconcile leaves its status unchanged fix: key the runtime's repo pull on config_updated_on instead of updated_on Oct 21, 2026
@k-anshul
k-anshul force-pushed the nishant/skip-unchanged-deployment-status branch from a343b46 to 9fbdab8 Compare October 22, 2026 06:50
@k-anshul k-anshul changed the title fix: key the runtime's repo pull on config_updated_on instead of updated_on fix: don't bump a deployment's updated_on when a reconcile leaves its status unchanged Oct 22, 2026
@k-anshul
k-anshul merged commit b778dce into main Oct 22, 2026
30 checks passed
@k-anshul
k-anshul deleted the nishant/skip-unchanged-deployment-status branch October 22, 2026 07:11
k-anshul pushed a commit that referenced this pull request Oct 23, 2026
…ts status unchanged (#10049)

Since #9653 the ValidateDeploymentsWorker schedules ReconcileDeployment for every prod deployment twice an hour.
For a running deployment the job always called UpdateDeploymentStatus(Running, ""), which sets updated_on = now() even when nothing changed.
The runtime's configReloader uses updated_on as its "deployment changed" signal, so every reload forced a repo pull and a full re-parse of the project (or a controller restart for archive-backed projects), cancelling in-flight reconciles.

The job now re-fetches the deployment and only writes the status when it actually differs.
The re-fetch matters because UpdateDeploymentInner may have moved the status to updating in the DB without updating the job's copy.

Side effect: the orphaned prod deployment teardown in ValidateDeploymentsWorker (updated_on older than 3h) can fire again, as it did before v0.89.
nishantmonu51 added a commit that referenced this pull request Oct 29, 2026
…ts status unchanged (#10049)

Since #9653 the ValidateDeploymentsWorker schedules ReconcileDeployment for every prod deployment twice an hour.
For a running deployment the job always called UpdateDeploymentStatus(Running, ""), which sets updated_on = now() even when nothing changed.
The runtime's configReloader uses updated_on as its "deployment changed" signal, so every reload forced a repo pull and a full re-parse of the project (or a controller restart for archive-backed projects), cancelling in-flight reconciles.

The job now re-fetches the deployment and only writes the status when it actually differs.
The re-fetch matters because UpdateDeploymentInner may have moved the status to updating in the DB without updating the job's copy.

Side effect: the orphaned prod deployment teardown in ValidateDeploymentsWorker (updated_on older than 3h) can fire again, as it did before v0.89.

(cherry picked from commit b778dce)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants