Skip to content

fix(retirement): sample each layer until it settles before judging it - #5430

Merged
devantler merged 1 commit into
mainfrom
claude/retirement-wait-for-settled-layers
Oct 27, 2026
Merged

devantler merged 1 commit into
mainfrom
claude/retirement-wait-for-settled-layers

Conversation

@devantler

Copy link
Copy Markdown
Contributor

Why

The second run of "Retire Unused App Leftovers" again stopped before deleting anything. Because the Email workload is still failing (#5412), Flux re-runs that part of the platform's health checks every two minutes and reports "progressing" while each attempt runs. The workflow checked at one of those moments and read it as "not deployed".

What

The workflow now re-checks each part of the platform for up to ten minutes and goes ahead once it has settled. What counts as settled is unchanged: healthy, or failing only an unrelated health check. If it never settles, the workflow still stops without deleting anything.

Part of #2496

👉 After merge/promotion: run the "Retire Unused App Leftovers" workflow from main again.

🤖 Generated with Claude Code

The second Retire Unused App Leftovers run refused again, before deleting
anything. With the unrelated private-mail workload failing (#5412), the
apps Kustomization retries its health checks every two minutes
(retryInterval 2m) and reports Ready=Unknown/Progressing during each
attempt, which can last minutes. The run read it in that window.

Read each layer up to 40 times, 15 seconds apart, and proceed once it
reports Ready or an unrelated HealthCheckFailed at its current
generation. The acceptance rule is unchanged, and a layer that never
settles is still refused.

Part of #2496

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@devantler

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 08fe36a and 1a44dc8.

Files selected for processing: 2
  • scripts/retire-unused-apps.sh
  • scripts/tests/test-retire-unused-apps.sh

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Recent review details
Additional context used
Code guidelines: 1
AGENTS.md — auto-discovered
Path-based instructions: 1
Source excerpt: **Directories are kebab-case**, named after the **application/component** *or* a **CR Kind in plural**.

CodeRabbit inference engine (AGENTS.md)

Files:

  • scripts/tests/test-retire-unused-apps.sh
  • scripts/retire-unused-apps.sh
Walkthrough

The retirement script adds configurable settling intervals and attempt limits. It retries Kustomization validation until the current generation has a non-empty inventory and is Ready or has a HealthCheckFailed condition, or until it reaches the attempt limit. The tests add support for a one-time response fixture and verify that a later HealthCheckFailed response allows retirement after an initial Progressing response.

Possibly related issues

  • devantler-tech/platform#5391 — The issue depends on the retirement workflow completing before Crossview seed cleanup. This change updates the workflow's health-check decision while retaining its attempt-limit failure behavior.

Priority: Normal

Merge Risk: Minimal · up to 1a44d

The change makes the retirement workflow wait for each layer to settle before judging it, so a transient progressing state no longer causes a false refusal. Deletion safety is unchanged, because the production lock serializes publication against the longer run. No merge-blocking risk remains.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within survi… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly identifies the main change: retrying each retirement layer until it settles before evaluation.
Description check Passed The description explains the temporary progressing state, the retry behavior, the ten-minute limit, and the expected workflow outcome. It directly relates to the changeset.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 2 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devantler

Copy link
Copy Markdown
Contributor Author

Readiness evaluation at 1a44dc8213d59a4a8c55b23eaf376b7b8d13c91b

  • The real case: run 40459698606 refused before deleting anything. A read-only check minutes later shows apps at its current generation (24), retryInterval: 2m, and moving between Ready=Unknown/Progressing during each retry and Ready=False/HealthCheckFailed (latest transition 06:16:23Z) because of the unrelated private-mail workload (Restore Email availability after repeated startup failures #5412).
  • Offline fixture: a new case returns Progressing on the first read and HealthCheckFailed afterwards. It proceeds and is shown to read the layer again. Failing on the first unsettled read is killed by that case. The failed-apply, stale-generation and owned-inventory refusals are unchanged.
  • Review: CodeRabbit is green at this head.

Unreachable gate: the deletion needs the prod environment credential. The next workflow run is the observation, verified read-only afterwards.

🤖 Generated with Claude Code

@devantler
devantler marked this pull request as ready for review October 27, 2026 06:42
@devantler
devantler added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit 44d1d8c Oct 27, 2026
41 checks passed
@devantler
devantler deleted the claude/retirement-wait-for-settled-layers branch October 27, 2026 06:53
@github-project-automation github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

1 participant