Skip to content

fix(kyverno): roll the reports controller through GitOps after the RBAC fix - #3779

Merged
devantler merged 2 commits into
mainfrom
claude/kyverno-reports-restart-3760
Sep 13, 2026
Merged

devantler merged 2 commits into
mainfrom
claude/kyverno-reports-restart-3760

Conversation

@devantler

Copy link
Copy Markdown
Contributor

🤖 Generated by the Agentic Engineer

Why

Kyverno has produced no new policy reports since 2026-08-27, so the policy dashboard shows 17-day-old results and new workloads get no report at all. The missing permission behind it was fixed in #3762, but the reports controller stayed idle afterwards. The issue's next step is one clean restart, to learn whether the controller was simply stuck.

What

Rolls the reports controller once through Git, using a pod annotation that can be bumped again whenever a clean restart is needed. No chart or version change, so the result tells a stuck controller apart from a bug in this Kyverno version.

After the merge-queue deploy, the check on #3760 is: new pods get reports within minutes and results refresh within the hourly background scan. If the controller stays idle after a clean start, the next step is the Kyverno 3.9.1 bump.

Part of #3760

…AC fix

The reports controller stayed up but idle after #3762 restored its read
permission, and no PolicyReport has refreshed since 2026-08-27. Add a
restart-generation pod annotation so a clean restart is a recorded Git
change, and use it once to tell a stuck controller apart from a bug in
this Kyverno version.

Part of #3760

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t annotation

The restart-generation pod annotation changes the kyverno HelmRelease
values, which the production authorization check fingerprints verbatim.
Record the re-approval against main f83314c (previous aggregate
0270e54e): only the kyverno HelmRelease changes, the identity set and all
46 grant-bearing documents are identical, and a synthetic ClusterRole
control proves the comparison detects a grant. The new aggregate is the
value CI printed on the approved renderer.

Part of #3760

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

@devantler I will review the changes in #3779.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 78e3333a-9c2c-47f0-82ef-dfb6043b4eab

📥 Commits

Reviewing files that changed from the base of the PR and between f83314c and a3d72cb.

📒 Files selected for processing (2)
  • k8s/bases/infrastructure/controllers/kyverno/helm-release.yaml
  • scripts/validate-eks-ci-role-policy/main.go

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-13T20:04:45.335728Z a3d72cb Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: a3d72cb5e9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Readiness at a3d72cb5

  • Tested: all required checks green at this head (17 success, 7 skipped), including the production authorization fingerprint check against the re-approved aggregate.
  • Reviewed: CodeRabbit was requested first and refused (Review rate limited, 20:00:50Z). Codex then reviewed this head: Didn't find any major issues, reviewed commit a3d72cb5 (20:04:43Z), with no finding comments, no review objects and zero review threads.
  • Evaluated as a user (read-only, prod, 2026-09-13 ~20:10Z):
    • All 480 PolicyReports in the cluster were created on or before 2026-08-27T00:13Z; the newest managed-field write is 2026-08-27T20:11Z. Nothing has been reported for 17 days, so the dashboard problem is live.
    • Both kyverno-reports-controller pods are Running and have been since 2026-08-27T11:46Z, i.e. they have been up the whole time without producing reports. That matches the "alive but stuck" hypothesis this PR tests with a clean restart.
    • The change is a pod annotation, so its effect is a rollout of exactly those pods through Flux. Post-merge check (recorded on fix(kyverno): background reporting has been frozen cluster-wide since the 3.9.0 upgrade #3760): new pods, fresh PolicyReports within minutes for new workloads, and refreshed results within the hourly background scan. If reports stay stale after a clean start, the next step is the Kyverno 3.9.1 bump.
  • ⚠️ Merge order: fix(openbao): require the static workers by hostname #3780 also re-approves the authorization fingerprint from the same base; whichever merges second must re-approve against the new base.

@devantler
devantler marked this pull request as ready for review September 13, 2026 20:05
@devantler
devantler added this pull request to the merge queue Sep 13, 2026
Merged via the queue into main with commit 4cd330c Sep 13, 2026
30 checks passed
@devantler
devantler deleted the claude/kyverno-reports-restart-3760 branch September 13, 2026 20:44
@github-project-automation github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

1 participant