Expose Crossplane managed-resource sync state as metrics Coroot can query

🤖 Generated by the Agentic Engineer

Evidence: 17 of 20 Repository managed resources sat Synced: False for weeks with no signal
(parent #2820). Today kubectl get managed -A spans 109 resources across 7 API groups, of which
7 are at reason=ReconcileError and 9 unifi resources are at reason=ReconcilePaused by design.
None of that state is queryable as a metric: the cluster has no ServiceMonitor/PodMonitor CRDs
(prometheus-operator went with the kube-prometheus-stack), and Coroot's bundled cluster-agent
exposes only image under clusterAgent.kubeStateMetrics — no CustomResourceState config — so
nothing currently emits Crossplane condition state.

Audience/problem: nothing can observe, chart, or alert on a managed resource that stops
reconciling, because the underlying signal does not exist in the metrics store. This blocks every
option weighed on the parent, not just one of them.

Expected behaviour: each Crossplane managed resource's Synced condition is exposed as a
Prometheus metric in Coroot's bundled Prometheus, carrying at minimum the resource's
kind/name/namespace and its condition reason — the reason label is what makes the deliberate
ReconcilePaused state separable from a genuine ReconcileError, and the parent's measurements show
a filter without it selects exactly the wrong set.

Proposed: upstream kube-state-metrics with a CustomResourceState ConfigMap, scraped by the
cluster-agent via the pod annotations it supports (coroot.com/scrape-metrics: "true" +
coroot.com/metrics-port; nothing in k8s/ uses these yet). A pinned upstream image keeps this
declarative and Renovate-trackable and avoids standing up an image build — platform's Go module is
yaml.v3 only, so a bespoke exporter would be a project rather than an increment. Bind its
ServiceAccount to the existing aggregated crossplane-view ClusterRole, verified on the parent to
grant no core secrets; authoring a new ClusterRole is unnecessary and would trip the EKS
authorization-surface fingerprint gate.

Known cost to weigh during implementation: CustomResourceState is configured per-GVK, so covering
7 API groups means 7 entries, and a newly-installed provider needs one added. Note that trade-off in
the PR against the alternative of a small first-party exporter.

Acceptance criteria:

Synced condition state for Crossplane managed resources is present in Coroot's bundled
Prometheus and queryable in the UI.
The metric carries the condition reason, and a query excluding ReconcilePaused returns
the genuinely-failing set and not the paused one.
up{job="crossplane-sync-exporter"} is delivered too, so "no condition series" is not
ambiguous between a healthy fleet and a dead exporter.
RED-proved against live data: the Repository resources at reason=ReconcileError appear.
No new ClusterRole — amended 2026-08-10; decided, not pending. What feat(coroot): export Crossplane managed-resource conditions as metrics #2994 shipped is a
purpose-built crossplane-sync-exporter ClusterRole granting get/list/watch on exactly
repo.github.m.upbound.io/repositories. The alternative this criterion originally asked for —
binding the aggregated crossplane-view — carries 49 rules, so the deviation is a
narrowing, not a widening. The 🔐 Validate EKS Authorization gate that the criterion
predicted would trip passed on the merged head. Do not "fix" this by rebinding to
crossplane-view: that would widen read access from one kind to the whole aggregate.
No access to core secrets — held; the role above grants none.
~~Covers managed resources generically across API groups, not only Repository.~~
Scope amended 2026-08-10 → Extend the Crossplane sync-state exporter past Repository — 90 of 110 managed resources are unobserved #3052. CustomResourceStateMetrics takes an exact GVK with no
wildcard, so generic coverage is a separate piece of work: 110 managed resources across 7
groups and 14 kinds exist, of which this exporter observes 20.

Two criteria above are decided rather than outstanding, and are ticked for that reason. The
record and the artifact disagreed until this amendment, and on the ClusterRole item that
disagreement pointed at the less safe option — see the 2026-08-06 comment below for the full
measurement.

Rough size: M. Delivers the signal only — alerting and Slack routing are #2987.

Part of #2820

⚠️ Possible Duplicate Issue(s)

Export Crossplane managed-resource conditions as metrics (first increment of #2820) #2993

🔗 Related PRs

#2445 - feat(kubescape): route runtime-detection alerts to Headlamp, Slack, and Coroot [merged]
#2459 - feat: deploy Kyverno Policy Reporter (UI + kyverno plugin) [merged]
#2794 - fix(opencost): restore usage metrics [merged]
#2994 - feat(coroot): export Crossplane managed-resource conditions as metrics [merged]
#3017 - fix(security): disposition the six unreachable-probe checkov findings [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer

Reopened. #2994 merged at 06:19:36Z and this closed 1 second later via Fixes #2986 in the
PR body — but the acceptance criteria are not met, and I said so in the PR's own review before
merging it.

The exporter ships as an unreferenced Kustomize component, so nothing is running and the
metric is not in coroot-prometheus. What merged is the capability, not the outcome.

Remaining, and it is the whole point of this issue:

Activate the component (a separate, independently reversible change).
Confirm crossplane_managed_resource_condition is queryable and agrees with a live read —
at the time of writing, 3 Synced=False / 17 Synced=True across 20 Repository
resources, with 3 showing Ready=True while Synced=False.
Confirm up{job="crossplane-sync-exporter"} arrives too. It is exempt from the
keep filter, and it is what stops "no series" being ambiguous between a healthy fleet
and a dead exporter.

Moving to 📊 Verifying.

My error, and it is worth naming precisely. I have recorded this trap three times as "keep
the closing keyword out of the commit message" — that framing is incomplete and is why I walked
into it anyway. Fixes #N in the PR body closes the issue just as surely. The actual rule:
use Fixes only when the issue is genuinely done at merge, and Part of whenever its
definition-of-done extends past it. This issue's did.

🤖 Generated by the Agentic Engineer

Two acceptance criteria are deviations, not pending verifications — and one is a live trap

The reopen above covers what is unverified. These two are different in kind: what merged
deliberately departs from the written criteria, so they will never be satisfied by activation
alone and must be decided rather than waited on.

1. "No new ClusterRole" — knowingly not followed, and the reason is sound

This AC said to bind to the existing aggregated crossplane-view, warning a new ClusterRole
"would trip the EKS authorization-surface fingerprint gate". What merged authors a new
ClusterRole (crossplane-sync-exporter). Measured just now:

crossplane-view exists, is aggregated, carries 49 rules, and 2 of them already cover
repo.github* — so the AC's route was genuinely viable, not a mistake.
The 🔐 Validate EKS Authorization gate passed (success) on the merged head
1b748f28ce. The predicted gate trip did not happen.

The merged choice grants get/list/watch on one kind; binding crossplane-view would grant the
full 49-rule aggregated read surface. So the deviation is a narrowing, and defensible.

🔴 The trap: left as an unticked box, a future run draining this issue reads "No new ClusterRole"
as unmet and "fixes" it by rebinding to crossplane-view — widening read access from one kind to
the whole aggregate. That would be a security regression produced by faithfully following a stale
criterion. Please amend this AC to record the decision (narrow purpose-built role, gate verified
passing) rather than leaving it to be re-litigated.

2. "Covers managed resources generically across API groups, not only `Repository`" — genuinely unmet

What merged covers Repository alone. This is not a verification gap; it is scope.
CustomResourceStateMetrics takes an exact GVK with no wildcard and no categories: managed
selector, so each of the remaining kinds needs its own entry — against
109 managed resources across 7 API groups (16 MR CRDs). The PR states this limit explicitly and
calls the coverage "explicitly partial".

This needs either a child issue for the remaining groups or an explicit scope amendment here. Until
one exists, the issue cannot close on activation alone, and the parent #2820's original evidence
(which spanned all groups) is only partially addressed.

Not a criticism of the merge

Both departures are defensible and both are documented in the PR. The problem is only that the
criteria still read as originally written, so the record and the artifact disagree — and on #1 that
disagreement points a future run at the less safe option.

test(coroot): move the exporter contract to its activated state

chore: agent-claim #2986 nonce=50d513d2940959a375996f385ad62fc2

🤖 Generated by the Agentic Engineer

Live verification now satisfies the remaining acceptance criteria:

The crossplane-sync-exporter Deployment is observed at generation 1 with 1/1 available, ready, and updated replicas. Its pod is Running, both containers are ready, and both have zero restarts.
Coroot Prometheus returns up{job="crossplane-sync-exporter"} = 1.
The current crossplane_managed_resource_condition{condition="Synced"} query returns all 20 Repository resources, with kind, name, namespace, and reason labels. Every series is 1 with reason ReconcileSuccess.
The Kubernetes API independently reports the same 20/20 Repository resources as Synced=True / ReconcileSuccess. The query for Synced=0 excluding ReconcilePaused is empty, matching the live resource truth rather than masking a failure.
Thirty-day Prometheus history retains ReconcileError-labelled series whose minimum value is 0 for all 20 resources, so the failing state is observably represented rather than only the healthy state.

The generic multi-kind expansion remains separately tracked in #3052, and alert delivery remains #2987. This issue now owns no residual work, so I am closing it as completed.

Navigation Menu

Repository navigation

Expose Crossplane managed-resource sync state as metrics Coroot can query #2986

Two acceptance criteria are deviations, not pending verifications — and one is a live trap

1. "No new ClusterRole" — knowingly not followed, and the reason is sound

2. "Covers managed resources generically across API groups, not only `Repository`" — genuinely unmet

Not a criticism of the merge

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Expose Crossplane managed-resource sync state as metrics Coroot can query #2986

Description

Activity

coderabbitai commented on Aug 6, 2026

devantler commented on Aug 6, 2026

devantler commented on Aug 6, 2026

Two acceptance criteria are deviations, not pending verifications — and one is a live trap

1. "No new ClusterRole" — knowingly not followed, and the reason is sound

2. "Covers managed resources generically across API groups, not only Repository" — genuinely unmet

Not a criticism of the merge

devantler commented on Aug 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

2. "Covers managed resources generically across API groups, not only `Repository`" — genuinely unmet