Skip to content

fix(platform): retire the unused Backstage portal - #4639

Merged
devantler merged 5 commits into
mainfrom
codex/backstage-retirement-4266-01a11089
Oct 11, 2026
Merged

devantler merged 5 commits into
mainfrom
codex/backstage-retirement-4266-01a11089

Conversation

@devantler

Copy link
Copy Markdown
Contributor

🤖 Generated by the Agentic Engineer

Why

Backstage remains as an unused demo after the portfolio chose GitHub as its developer platform. Retiring it removes unnecessary maintenance and hosting overhead.

What

Retire the portal and its dedicated resources, remove obsolete access and monitoring entries, and document GitHub's catalog, templates, scorecard badges and self-service workflows. Complete the final live cleanup after merge, keeping the old database contents offline.

Part of #4266

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Validated source head 9036e6295d15ba5eab0e369ff3231b2b92a554ed against deployed base 770216803a35a8cb3f6579c6e1670edb4f0ef5d5.

  • All four independently reconciled production roots render with no retired portal references, including embedded authentication and monitoring configuration. Negative controls reject the old scaffold and residual shared authentication wiring even with a large rendered payload.
  • Remaining SSO callback coverage, dashboard group policy, CNPG networking, Flux pull-auth behavior and affected Coroot reconciler suites pass. Two-stage persistence retirement checks pass against the deployed base.
  • The generated Headlamp exception mirror was regenerated through its owner. The authorization evaluator was regenerated and verified with the approved kubectl v1.36.2/Kustomize v5.8.1 renderer: two retired objects disappear, and only the Dex callback-subtraction hash changes.
  • ShellCheck, actionlint, affected Go tests and whitespace checks pass. Local and production KSail schema validation pass for 127 Kustomizations. Local OCI Helm chart rendering remains limited by the unavailable credential helper; required CI is the authority for that coverage.
  • Self-review completed correctness, simplification and security passes. Changes to shared authentication and permissions only remove the retired application's entries; no access or enforcement boundary is widened.

Runtime cleanup remains pending until this PR actually merges and its source revision has deployed. Existing protected orphans will be explicitly removed, database contents retained offline, the orphan-proof Job run successfully, and unrelated live services checked before closing #4266. No live mutation has been made for this change yet.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@codex review for the retired portal removal, remaining shared authentication and monitoring, and the render regression guard

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-11T11:23:56.875690Z bb42ced Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 9036e6295d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@codex review for the complete portal retirement including the CI executable-mode correction

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 5eae306b2e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Ready at signed head 5eae306b2e080ccc3e4b4a5592ed46b9e60b241d.

Current-head CI passes, including manifest, rendered production authorization, chart post-rendering, disruption-budget, signature and required-aggregate checks. Code scanning, code quality, dependency review and workflow scanning also pass. The executable-mode failure at the earlier head was fixed and both executable guards pass.

Codex review is clean at this head. Fresh full review-surface reads show zero unresolved threads, inline comments or body findings. GitHub reports CLEAN and MERGEABLE; the latest fetched base changes do not overlap the removal files.

Exercised the desired configuration by rendering all four production roots and their embedded configuration; no retired portal wiring remains. Negative controls reject the deployed scaffold and residual shared authentication with a large render. Remaining SSO callbacks, dashboard groups and affected monitoring behavior pass their tests. The changed Dex HelmRelease also renders successfully in targeted production validation. Broader unchanged local OCI chart coverage is limited by the unavailable local credential helper; CI passed the required coverage.

This PR changes desired GitOps configuration. Its actual production retirement remains a post-merge operation: confirm the deployed revision, retain the former database volumes offline, explicitly remove the protected runtime orphans, run the canonical orphan proof and verify unrelated services before closing #4266.

@devantler
devantler marked this pull request as ready for review October 7, 2026 21:13
@devantler
devantler added this pull request to the merge queue Oct 7, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 8, 2026
@devantler
devantler marked this pull request as draft October 8, 2026 01:25
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 36 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository YAML (base), Organization UI (inherited)
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 4a256b84-0f0f-43d0-86ba-2fd60e71ed3f

📥 Commits

Reviewing files that changed from the base of the PR and between bf3fe77 and f6a6d8b.


📒 Files selected for processing (49)
  • .github/renovate.json
  • .github/workflows/ci.yaml
  • AGENTS.md
  • README.md
  • docs/backstage.md
  • docs/developer-platform.md
  • docs/dr/cnpg-storageclass-migration.md
  • k8s/bases/apps/backstage/cilium-network-policy.yaml
  • k8s/bases/apps/backstage/cluster.yaml
  • k8s/bases/apps/backstage/external-secret.yaml
  • k8s/bases/apps/backstage/helm-release.yaml
  • k8s/bases/apps/backstage/helm-repository.yaml
  • k8s/bases/apps/backstage/http-route.yaml
  • k8s/bases/apps/backstage/kustomization.yaml
  • k8s/bases/apps/backstage/namespace.yaml
  • k8s/bases/apps/backstage/password.yaml
  • k8s/bases/apps/homepage/config-map.yaml
  • k8s/bases/apps/kustomization.yaml
  • k8s/bases/infrastructure/cluster-policies/best-practices/restrict-homepage-service-groups.yaml
  • k8s/bases/infrastructure/cluster-security-exceptions/secret-reader-rbac.yaml
  • k8s/bases/infrastructure/controllers/auth-proxy/cilium-network-policy.yaml
  • k8s/bases/infrastructure/controllers/auth-proxy/config-map.yaml
  • k8s/bases/infrastructure/controllers/coroot/cluster-role-cnpg-degraded-alert.yaml
  • k8s/bases/infrastructure/controllers/dex/helm-release.yaml
  • k8s/bases/infrastructure/controllers/kubescape/config-map-headlamp-exceptions.yaml
  • k8s/bases/infrastructure/controllers/kubescape/helm-release.yaml
  • k8s/bases/infrastructure/controllers/oauth2-proxy/reference-grant.yaml
  • k8s/providers/hetzner/apps/backstage/cilium-network-policy.yaml
  • k8s/providers/hetzner/apps/backstage/external-secret-coroot-monitor.yaml
  • k8s/providers/hetzner/apps/backstage/password-coroot-monitor.yaml
  • k8s/providers/hetzner/apps/backstage/patches/enable-coroot-monitoring.yaml
  • k8s/providers/hetzner/apps/backstage/patches/enable-user-namespaces.yaml
  • k8s/providers/hetzner/apps/kustomization.yaml
  • k8s/providers/hetzner/infrastructure/coroot/cluster.yaml
  • k8s/providers/hetzner/infrastructure/coroot/config-map-risk-acceptances.yaml
  • k8s/providers/hetzner/infrastructure/coroot/cron-job-alert-autosuppressor.yaml
  • k8s/providers/hetzner/infrastructure/coroot/cron-job-application-health-reconciler.yaml
  • k8s/providers/hetzner/infrastructure/flux-notifications/alert.yaml
  • scripts/tests/cnpg-network-policy/policy_test.go
  • scripts/tests/refresh-flux-ghcr-auth/fake_kubectl_test.go
  • scripts/tests/refresh-flux-ghcr-auth/rollout_core_test.go
  • scripts/tests/test-backstage-retirement.sh
  • scripts/tests/test-coroot-alert-autosuppressor.sh
  • scripts/tests/test-coroot-application-health-reconciler.sh
  • scripts/tests/test-coroot-availability-remediation.sh
  • scripts/tests/test-coroot-postgres-scrape-policy.sh
  • scripts/tests/test-coroot-risk-acceptance-reconciler.sh
  • scripts/validate-eks-ci-role-policy/approved-surface-history.md
  • scripts/validate-eks-ci-role-policy/approved-surface.txt

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository YAML (base), Organization UI (inherited)
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: c8b97fec-6619-4e27-a11a-3353f4454cc4

📥 Commits

Reviewing files that changed from the base of the PR and between 7db7fd8 and 415aa0c.


📒 Files selected for processing (49)
  • .github/renovate.json
  • .github/workflows/ci.yaml
  • AGENTS.md
  • README.md
  • docs/backstage.md
  • docs/developer-platform.md
  • docs/dr/cnpg-storageclass-migration.md
  • k8s/bases/apps/backstage/cilium-network-policy.yaml
  • k8s/bases/apps/backstage/cluster.yaml
  • k8s/bases/apps/backstage/external-secret.yaml
  • k8s/bases/apps/backstage/helm-release.yaml
  • k8s/bases/apps/backstage/helm-repository.yaml
  • k8s/bases/apps/backstage/http-route.yaml
  • k8s/bases/apps/backstage/kustomization.yaml
  • k8s/bases/apps/backstage/namespace.yaml
  • k8s/bases/apps/backstage/password.yaml
  • k8s/bases/apps/homepage/config-map.yaml
  • k8s/bases/apps/kustomization.yaml
  • k8s/bases/infrastructure/cluster-policies/best-practices/restrict-homepage-service-groups.yaml
  • k8s/bases/infrastructure/cluster-security-exceptions/secret-reader-rbac.yaml
  • k8s/bases/infrastructure/controllers/auth-proxy/cilium-network-policy.yaml
  • k8s/bases/infrastructure/controllers/auth-proxy/config-map.yaml
  • k8s/bases/infrastructure/controllers/coroot/cluster-role-cnpg-degraded-alert.yaml
  • k8s/bases/infrastructure/controllers/dex/helm-release.yaml
  • k8s/bases/infrastructure/controllers/kubescape/config-map-headlamp-exceptions.yaml
  • k8s/bases/infrastructure/controllers/kubescape/helm-release.yaml
  • k8s/bases/infrastructure/controllers/oauth2-proxy/reference-grant.yaml
  • k8s/providers/hetzner/apps/backstage/cilium-network-policy.yaml
  • k8s/providers/hetzner/apps/backstage/external-secret-coroot-monitor.yaml
  • k8s/providers/hetzner/apps/backstage/password-coroot-monitor.yaml
  • k8s/providers/hetzner/apps/backstage/patches/enable-coroot-monitoring.yaml
  • k8s/providers/hetzner/apps/backstage/patches/enable-user-namespaces.yaml
  • k8s/providers/hetzner/apps/kustomization.yaml
  • k8s/providers/hetzner/infrastructure/coroot/cluster.yaml
  • k8s/providers/hetzner/infrastructure/coroot/config-map-risk-acceptances.yaml
  • k8s/providers/hetzner/infrastructure/coroot/cron-job-alert-autosuppressor.yaml
  • k8s/providers/hetzner/infrastructure/coroot/cron-job-application-health-reconciler.yaml
  • k8s/providers/hetzner/infrastructure/flux-notifications/alert.yaml
  • scripts/tests/cnpg-network-policy/policy_test.go
  • scripts/tests/refresh-flux-ghcr-auth/fake_kubectl_test.go
  • scripts/tests/refresh-flux-ghcr-auth/rollout_core_test.go
  • scripts/tests/test-backstage-retirement.sh
  • scripts/tests/test-coroot-alert-autosuppressor.sh
  • scripts/tests/test-coroot-application-health-reconciler.sh
  • scripts/tests/test-coroot-availability-remediation.sh
  • scripts/tests/test-coroot-postgres-scrape-policy.sh
  • scripts/tests/test-coroot-risk-acceptance-reconciler.sh
  • scripts/validate-eks-ci-role-policy/approved-surface-history.md
  • scripts/validate-eks-ci-role-policy/approved-surface.txt

💤 Files with no reviewable changes (28)
  • k8s/providers/hetzner/infrastructure/coroot/config-map-risk-acceptances.yaml
  • k8s/bases/apps/backstage/helm-repository.yaml
  • k8s/bases/apps/backstage/external-secret.yaml
  • k8s/bases/apps/kustomization.yaml
  • k8s/providers/hetzner/apps/backstage/external-secret-coroot-monitor.yaml
  • k8s/providers/hetzner/apps/backstage/patches/enable-coroot-monitoring.yaml
  • k8s/providers/hetzner/infrastructure/flux-notifications/alert.yaml
  • k8s/bases/infrastructure/controllers/auth-proxy/config-map.yaml
  • k8s/bases/apps/backstage/cluster.yaml
  • k8s/providers/hetzner/apps/backstage/cilium-network-policy.yaml
  • k8s/bases/apps/backstage/cilium-network-policy.yaml
  • k8s/bases/apps/backstage/kustomization.yaml
  • k8s/bases/apps/backstage/password.yaml
  • k8s/bases/infrastructure/controllers/auth-proxy/cilium-network-policy.yaml
  • k8s/bases/apps/homepage/config-map.yaml
  • k8s/bases/infrastructure/cluster-policies/best-practices/restrict-homepage-service-groups.yaml
  • k8s/bases/apps/backstage/helm-release.yaml
  • k8s/bases/infrastructure/controllers/kubescape/config-map-headlamp-exceptions.yaml
  • k8s/bases/infrastructure/controllers/oauth2-proxy/reference-grant.yaml
  • k8s/bases/infrastructure/controllers/dex/helm-release.yaml
  • docs/backstage.md
  • k8s/providers/hetzner/apps/backstage/password-coroot-monitor.yaml
  • k8s/bases/apps/backstage/namespace.yaml
  • k8s/bases/infrastructure/cluster-security-exceptions/secret-reader-rbac.yaml
  • scripts/tests/test-coroot-availability-remediation.sh
  • k8s/providers/hetzner/apps/backstage/patches/enable-user-namespaces.yaml
  • k8s/providers/hetzner/infrastructure/coroot/cron-job-application-health-reconciler.yaml
  • k8s/bases/apps/backstage/http-route.yaml

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


📜 Recent review details
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2026-08-11T12:41:28.242Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 3082
File: k8s/bases/infrastructure/controllers/coroot/cron-job-cnpg-degraded-alert.yaml:113-120
Timestamp: 2026-08-11T12:41:28.242Z
Learning: When changing behavior in Kubernetes manifests or related documentation, review comments and documentation in YAML/YML and Markdown files for statements describing the previous behavior. Update every stale statement in the same change so the repository’s explanatory text remains consistent with the implementation.

Applied to files:

  • k8s/bases/infrastructure/controllers/kubescape/helm-release.yaml

📚 Learning: 2026-08-10T13:01:12.782Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 3057
File: .github/workflows/ci.yaml:622-659
Timestamp: 2026-08-10T13:01:12.782Z
Learning: Repository shell tests and scripts must remain compatible with macOS Bash 3.2. Do not use Bash 4+ features such as `mapfile`; use portable constructs, such as a `while IFS= read -r` loop, instead.

Applied to files:

  • scripts/tests/test-backstage-retirement.sh

🔇 Additional comments (10)
k8s/providers/hetzner/apps/kustomization.yaml (1)

69-70: LGTM!

Also applies to: 95-95


.github/renovate.json (1)

142-142: LGTM!


docs/dr/cnpg-storageclass-migration.md (1)

31-31: LGTM!


k8s/bases/infrastructure/controllers/coroot/cluster-role-cnpg-degraded-alert.yaml (1)

4-4: LGTM!


k8s/bases/infrastructure/controllers/kubescape/helm-release.yaml (1)

296-296: LGTM!


k8s/providers/hetzner/infrastructure/coroot/cluster.yaml (1)

57-57: LGTM!


k8s/providers/hetzner/infrastructure/coroot/cron-job-alert-autosuppressor.yaml (1)

173-173: LGTM!

Also applies to: 175-176


AGENTS.md (1)

11-13: LGTM!


README.md (1)

303-303: LGTM!


docs/developer-platform.md (1)

11-13: 🔒 Security & Privacy

The concern is refuted. The prod environment has protection rules and requires protected branches. Direct production deploy and recovery jobs use that environment. Workflows without it either dispatch the protected CD path or explicitly perform no production operation. The documentation accurately describes the controls.



📝 Walkthrough

Walkthrough

The change removes the Backstage deployment and its related routing, identity, policy, monitoring, and database-maintenance configuration. It updates related tests and authorization records, and adds a test that checks Backstage paths and rendered production layers. The documentation now describes GitHub as the developer platform and links to the new developer-platform guide.

Priority: ➖ Normal

Priority: ➖ Normal

Estimated code review effort:


Merge Risk: ⚪ Minimal · up to 415aa

No actionable merge-blocking risk is identified; remaining production cleanup is an operational follow-up after deployment.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 8 files. (13 skipped: 1… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check Passed The title clearly and concisely identifies the main change: retiring the unused Backstage portal.
Description check Passed The description directly explains why the unused Backstage portal is being retired and summarizes the related cleanup and GitHub developer-platform documentation.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 8 files. (13 skipped: 13 unsupported.)


  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Integrated reviewed main bf3fe77f0e144763baca5840a11de5cbea5b17f1 without rewriting history. The signed repair head is f6a6d8b3e803884264e8811798b28cc86878c490; both approval-history entries were preserved, and the automatically merged ledger passes the canonical authorization evaluator with checksum-verified kubectl v1.36.2 / Kustomize v5.8.1. No generated approval entries were hand-edited.

Both local and production schema validations pass. All four production roots contain no retired portal wiring. The affected monitoring, database network-policy and two-stage persistence guards pass, as does the affected credential-rollout drain case. The broader local credential-rollout suite exceeded its five-minute timeout and is not reported as passed. New-head hosted CI and substantive review remain required; this PR stays draft.

Actual production retirement and retained-storage/orphan proof remain post-deployment work on #4266. No live deletion was performed.

@devantler devantler removed the blocked label Oct 10, 2026
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor
⚠️ Action not completed

Review rate limited.


Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 36 minutes.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: f6a6d8b3e8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Ready for the normal protected merge queue at f6a6d8b against freshly fetched main bf3fe77. All 33 checks are settled (25 passed, 8 intentional skips), complete comment/review/thread reads show no remaining finding or human blocker, all review threads are resolved, and the forge reports CLEAN. Codex’s current-head review is green. All readable required gates pass; the separately reported code-quality analysis has no readable verdict and is not claimed as verified.

As an operator, I rendered the actual local and production roots and exercised the retirement guard against all four production layers. The current-head manifest CI confirms those layers contain no retired portal or access wiring; its complete credential-helper suite passed with the configured 20-minute bound. Local and production workload validation and the targeted changed-case standard/race checks passed. This proves the declared retirement. Production reconciliation and final retained-resource cleanup remain post-merge checks; no live database or retained storage was deleted in this run.

The preceding deployment’s restore-main job succeeded, and the fresh platform health check exited 0. Its separate drain failure remains tracked in #3388; that failed rollout is not being retried here.

@devantler
devantler marked this pull request as ready for review October 11, 2026 00:31
@devantler
devantler added this pull request to the merge queue Oct 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 11, 2026
@devantler

devantler commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Removed from the merge queue at 05:37Z: the production deploy failed. Re-queue only after #4733 has merged

What failed. Merge-queue run 38113692499, job 🚀 Deploy to Prod, step 🚦 Verify the gateway applies every HTTPRoute:

The gateway is not serving these HTTPRoutes as declared after 300s:
  rejected backstage/backstage parent=kube-system/platform/https conditions=ResolvedRefs/RefNotPermitted

Root cause (found, tracked in #4732). This change removes the Internal Developer Platform group from the Homepage group policy and the only route that uses it. The infrastructure layer applied the shorter list at 05:23:15Z. Five seconds later Flux tried to delete the route, and the same policy refused the delete because the route's group was no longer declared. The Flux event on the apps layer:

PruneFailed  HTTPRoute/backstage/backstage delete failed: admission webhook "validate.kyverno.svc-fail" denied the request ... restrict-homepage-service-groups: declared-service-group

The route stayed, its access grant was already gone, and the gateway rejected it. Nothing is wrong with this pull request's own content: the policy should never judge a delete.

Fix. #4733 limits the policy to create and update. Its test reproduces the refusal offline and passes after the change.

State now. Prod is back on main (the route reads Accepted and ResolvedRefs true since 05:47:44Z, platform-live-health.sh OK). Backstage was unreachable for about 25 minutes; nothing else was affected, and the queue behind this PR kept moving.

Next. Once #4733 is merged and deployed, re-queue this head. A re-queue before that fails the same way.

@devantler

devantler commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

No longer parked (2026-10-11 11:15Z). #4733 merged at 11:08Z and its production deploy passed. Read back on the live cluster: restrict-homepage-service-groups is Ready and now matches create and update only, so deleting the Backstage route is no longer refused.

What is left before this can be queued again: the branch conflicts with main (head f6a6d8b3e, unchanged since 2026-10-10). That is for the Codex lane, which owns this branch: bring main in, let CI and a review run on the new head, then queue it. The earlier eviction was merge-queue run 38113692499 (05:05Z), caused by the policy that #4733 fixed.

@devantler devantler added blocked and removed blocked labels Oct 11, 2026
@devantler
devantler marked this pull request as draft October 11, 2026 11:12
# Conflicts:
#	scripts/validate-eks-ci-role-policy/approved-surface-history.md
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

The deletion-policy prerequisite #4733 is merged and deployed. Its live rule still checks create/update requests and permits the retirement to prune its resources. I integrated that reviewed main state into this PR without changing the retirement scope; the sole conflict was two independent authorization-history entries, both retained.

Current head: bb42cedee3d0909c76fc4d22e160744659d538df.

Validation passed: both local and production manifest renders, the production authorization evaluator with the approved renderer, the four-layer retirement regression, Homepage policy regression, Coroot alert/application/risk/scrape regressions, all 218 database-network tests under the race detector, the changed PDB-blocked-drain regression under the race detector, ShellCheck, actionlint, and diff whitespace checks. The drain test preserves both the refusal diagnostic and restoration of only the owned cordon. The updated branch remains a draft for fresh current-head review and CI.

Self-review found no additional correctness, security or simplification changes to make: shared access is removed only for the retired portal, remaining services and admission checks stay intact, and the new regression examines every independently reconciled production layer. Actual resource pruning and remaining-service health still require protected deployment readback after merge; local renders are not that proof.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: bb42cedee3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Readiness rechecked at bb42cedee3d0909c76fc4d22e160744659d538df. Native CI 38135259561 succeeded at this head. The actual local and production renders, four independent production retirement-layer assertions, authorization validation, policy regression, database network tests and drain regression are recorded in the current-head evaluation.

Codex's clean review names this head. Complete conversation, review-body, inline-comment and thread reads found no open findings or maintainer directions; the thread count is zero and no live worktree holder was found. All 33 current checks are settled; GitHub reports CLEAN and MERGEABLE. The repository's unreadable per-head code-quality verdict remains UNVERIFIED rather than being reported as green; the normal protected queue remains authoritative.

Self-promoting and submitting this evaluated head to the normal merge queue. #4266 remains open: deployment must remove the obsolete route/access wiring while retaining the protected runtime resources, then a separately reviewed protected cleanup must retain the database backing volumes before retiring runtime ownership. Source merge is not completed live retirement and is not authority for manual storage deletion.

@devantler
devantler marked this pull request as ready for review October 11, 2026 11:53
@devantler
devantler added this pull request to the merge queue Oct 11, 2026
Merged via the queue into main with commit 9537b5c Oct 11, 2026
33 checks passed
@devantler
devantler deleted the codex/backstage-retirement-4266-01a11089 branch October 11, 2026 15:50
@github-project-automation github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

1 participant