Skip to content

bug(ci): EKS authorization validator fails on main, reddening every PRΒ #2801

Closed

Description

@devantler

πŸ€– Generated by the Agentic Engineer

Evidence

scripts/validate-eks-ci-role-policy fails on main, and every open PR inherits the failure the
moment its CI re-runs. It is invisible on main itself because the job is PR-only β€” main's own
check-runs are all green.

Bisected to a single commit, with the test genuinely executing in both directions:

commit result unresolved-substitution lines
6b67c945 (fix(aws): allow EKS version discovery (#2753)) --- PASS: TestValidateAuthorizationAcceptsCommittedPolicy / ok 0
f89efff4 (fix(secret): update alertmanager webhook URL and metadata timestamps, current main) --- FAIL 35

Run as go test -count=1 ./scripts/validate-eks-ci-role-policy/... on a clean worktree of each
commit. -count=1 matters: the first attempt returned a cached ok and looked green.

The failure reports unresolved Flux substitution in authorization resource across the whole
authorization surface β€” HelmReleases in kube-system, longhorn-system, openbao, kyverno,
kubescape, velero, … plus RoleBinding/ClusterRoleBinding entries such as oidc-view and
oidc-cluster-reader. It is the entire surface, not one resource, which points at the substitution
source rather than at any individual manifest.

f89efff4 changed exactly one file: k8s/clusters/prod/bootstrap/secret.enc.yaml, 4 insertions /
4 deletions.

How it was found

platform#2776 went red on four checks β€” πŸ§ͺ Test, πŸ“Š Code Coverage, πŸ” Validate EKS Authorization and CI - Required Checks β€” after a documentation-only push. All four are the
same root cause: they all build on that one Go package.

The discriminator was that the branch passes alone and fails merged: CI checks out the merge
result, so a PR inherits main's state even when its own diff is inert.

tree result
claude/lint-gate-green-2775 alone ok
main alone FAIL (35)
branch merged with main FAIL

So this is not that PR's defect. Any PR whose CI re-runs will now go red the same way.

Affected audience and impact

Every contributor and every open PR on this repository. The gate is a security control over the
EKS authorization surface, so a persistently red result also trains reviewers to discount it β€” and
because main never runs it, nothing signals that main is the thing that is broken.

Expected behaviour

  • TestValidateAuthorizationAcceptsCommittedPolicy passes on main.
  • The authorization surface's Flux substitutions resolve, or the validator is taught the new
    substitution source if that source legitimately moved in f89efff4.
  • main runs this validator, so a regression is caught where it is introduced instead of
    surfacing as an unexplained red on unrelated PRs. That gap is what let this sit unnoticed.

Acceptance criteria

  • go test -count=1 ./scripts/validate-eks-ci-role-policy/... passes on main.
  • A previously-red open PR goes green on these four checks with no change to its own diff.
  • The validator runs on pushes to main, not only on pull requests.
  • A regression test pins whichever invariant f89efff4 broke, so the next edit to the
    substitution source fails loudly and locally.

Size: small–medium once the substitution source is identified; the main-coverage gap is a
separate, small change and can ship first.

I have deliberately not reproduced any secret material here β€” only the commit, the changed path, and
the validator's own resource identifiers, all of which are already public in this repository.

Activity

  1. added theissue type on Jul 25, 2026
  2. coderabbitai commented on Jul 25, 2026

    @coderabbitai
    Contributor
    πŸ”— Related PRs

    #2507 - fix(apps): make actual-budget openid schema offline-resolvable [merged]
    #2565 - feat(aws): activate EKS CI identity resources [merged]
    #2625 - fix(deploy): synchronize production GHCR pull auth [merged]
    #2633 - fix(deploy): verify incoming image on Talos nodes [merged]
    #2753 - fix(aws): allow EKS version discovery [merged]


    πŸ“ Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    πŸ§ͺ Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    πŸ’¬ Have feedback or questions? Drop into our discord!

  3. moved this to πŸ“₯ Backlog in 🌊 Project Boardon Jul 25, 2026
  4. self-assigned this
    on Jul 25, 2026
  5. moved this from πŸ“₯ Backlog to πŸ‘€ In Review in 🌊 Project Boardon Jul 25, 2026
  6. moved this from πŸ‘€ In Review to πŸ“₯ Backlog in 🌊 Project Boardon Jul 25, 2026
  7. moved this from πŸ“₯ Backlog to βœ… Done in 🌊 Project Boardon Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Effort

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions