A single unready Coroot workload blocks every production deploy, including recovery #4658

🤖 Generated by the Agentic Engineer

Problem

Every production deploy runs a pre-publish check that requires all six Coroot workloads to be fully ready. When one Coroot server is unready for a reason that has nothing to do with the change being deployed, no deploy can pass — and neither can the automatic recovery deploy, which runs the same check.

On 8 October this stopped all delivery for more than 13 hours: 27 consecutive merge-group deploys failed at this check from 01:52 UTC onward, including the repair for an unrelated degraded component (#4653). The cause of the unready server is tracked separately in #4652 (database connection exhaustion); this issue is about the blast radius.

Why it matters

• A fault in one observability application becomes a full stop for the platform, including security and incident fixes.
• The pipeline cannot heal itself: the recovery job is blocked by the condition it would need to pass.
• The only way out is a manual production action by the maintainer.

What the check is for

The check exists to prove that the Coroot-scoped admission change stays converged: the six operator-generated templates keep the two admitted fields, are owned by the operator, and are not being rewritten in a loop. Application-level readiness is used as part of that proof, which is what couples delivery to Coroot's runtime health.

Hypothesis

The convergence proof (population, ownership, admitted fields present, generations observed and stable) can be kept in full while a workload that is unready for an unrelated runtime reason no longer refuses the publish step — for example by distinguishing "rollout complete but a replica is unhealthy" from "rollout not converged", and reporting the former loudly without blocking.

This narrows a safety check, so the design needs the maintainer's agreement before it is built.

Acceptance criteria

• A deploy that does not touch Coroot can publish while one Coroot replica is unhealthy and its templates are otherwise converged.
• The check still refuses when a template is missing, extra, not operator-owned, lacks an admitted field, or is being rewritten.
• The recovery deploy can run in the state that blocked it on 8 October.
• A regression covers both the blocked and the allowed case.

Related: #4652, #4653, #4016.