Repository navigation
Diagnose and prevent the correlated control-plane interruption #4592
Description
🤖 Generated by the Agentic Engineer
Problem
A correlated control-plane interruption at approximately 17:54–17:55 UTC on October 6 caused consensus read failures and several controller readiness, liveness and admission failures. The platform recovered before the subsequent scoped-reader health checks, but recovery alone does not explain or prevent recurrence.
Verified evidence
- Finite, private consensus-service log captures show a leader transition around 17:54:56–17:54:57 UTC, read-index retries and slow apply requests. The longest slow apply request observed in these bounded tails was 18.68 seconds.
- The captured tails contain 600 lines, of which 594 parse as structured records. They have different coverage windows, and one begins after the leadership transition; they cannot establish the initiating cause or whole-event coverage.
- Current API readiness passes, including its consensus checks. The existing platform health command returns
PLATFORM-HEALTH=OK. The affected named controllers are available at their current generations, and current node conditions show no pressure. - Current consensus readback reports a common leader and term, applied indexes caught up and no reported errors. Current resource samples do not reconstruct usage during the interruption.
Raw logs, endpoints, identities, request contents and machine provenance stay in private diagnostic receipts. No machine or workload was restarted, no maintenance operation was run, and no check or alert was disabled.
Outcome and acceptance
Identify the initiating dependency and deliver the durable repair in its owning declaration or supported operational path. Join bounded pre-event leadership, storage/CPU/network timings and controller failures; distinguish a scheduling pause, storage stall, transport failure and workload-induced request load with actual evidence. Preserve consensus, admission, persistence and authentication protections.
Validate the identified repair before delivery, drive its PR through protected merge and deployment, and verify healthy consensus plus affected controller and admission behavior afterward. Record recurrence evidence over a declared observation window. Do not close this issue merely because an instantaneous readiness check passes, and do not conflate this event with the separately unproven request-attribution gate in #4329.
Part of #2043.
Activity
devantler commentedon Oct 6, 2026
The extended, bounded reader captures show two leadership transitions: a lost leader at 17:54:49.934 UTC, election of the next term at approximately 17:54:51, and another lost leader at 17:54:56.273 followed by a common leader at approximately 17:54:57. Peers subsequently report timeout/EOF transport failures and further election attempts through approximately 17:55:12. The extended captures contain a longest slow request of 21.21 seconds; the smaller initial tails observed 18.68 seconds.
The three extended captures contain 3,000 lines, with 2,948 structured records and 52 unsupported lines. Two reach earlier history, but the affected peer's retained tail starts near the first leadership loss. These records establish consensus disruption and transport symptoms; they do not yet identify whether a scheduling pause, storage stall, transport problem or request load initiated it.
A separate finite kernel/service capture on the affected machine records an existing consensus service health check timing out at 17:55:10.253 and succeeding at 17:55:16.729. Only two records fall in the inspected ten-minute event window, so the absence of a kernel fault in that window is not proof that none occurred. No restart or maintenance action was performed.
The after-capture inventory retains all seven node identities and addresses, all nodes are ready, and no memory, disk or PID pressure condition is asserted. The existing reader role was used throughout; no credential, machine configuration, workload, alert or protection was changed. Raw request contents and machine provenance remain private.
The cause and durable repair remain unproven. This issue stays open while historical CPU, storage and network metrics are joined to this timeline and while recurrence evidence is gathered.
Further timing joins place the first retained slow apply at 17:54:41.813 and the first retained read-index retry at 17:54:42.175, before the first database liveness warning at 17:54:46. The event inventory contains 15 unhealthy probe records in the two-minute burst, spanning database, controller and admission-related workloads. This supports a correlated dependency disruption; it does not identify the initiating failure.
All 425 slow-apply warnings in the inspected 17:53:30–17:56:30 window identify read-only operations. Their largest reported response contains 256 results. The warning duration therefore does not itself demonstrate a slow disk write or an expensive full-keyspace read. The read-index and peer-transport joins remain necessary.
The three event-window node charts show no sustained aggregate CPU saturation. The affected node's visible CPU and bandwidth rise follows the start of the consensus delays. Its displayed mean disk latency remains below one millisecond, but this is not the consensus WAL or backend latency distribution. The upstream metrics documentation identifies WAL fsync, backend commit and peer RTT histograms as the distinguishing measurements. Those finer historical measurements have not been retrieved, so neither slow storage nor a transport/scheduling pause is excluded by these averages.
A later platform health check still returns
PLATFORM-HEALTH=OK. Coroot, Wedding and Umami each reportCluster in healthy statewith all three desired instances ready. These readbacks verify current recovery; they do not close the cause or recurrence acceptance.At 18:46:59 UTC, another finite readback using the existing reader reports three valid member rows, one common leader, unchanged term 199, equal raft and applied indexes, no learners and no reported errors. Node identity/address inventories before and after this read are unchanged and all nodes remain ready. This is another verified recovery checkpoint, not continuous proof that no transient disruption occurred between samples.
devantler commentedon Oct 15, 2026
Fresh recurrence evidence, 15 October: native OIDC reads joined controller lease timeouts at 22:00:01 UTC to an API-server lease failure at 22:00:03 reporting an etcd request timeout, possibly due to previous leader failure. This supports a correlated API/consensus interruption; it does not establish the initiating cause or prove a leader transition. Bounded API log tails have incomplete event coverage and are not a whole-event census.
Current read-only API readiness, including etcd and etcd-readiness, passes. Affected controller deployments are available at their current generations, the API-server containers have no restarts, and database clusters retain their desired ready instances. These are recovery observations, not recurrence prevention or closure evidence.
Keep separate observations separate: a 22:12:58 CNPG plugin request was cancelled; a 22:15:32 scaling attempt met a resource-version conflict; and a 22:29:05 Homepage release-feed request timed out connecting to GitHub. Their common initiating cause is unproven. No restart, threshold change, suppression, authentication change or maintenance action was performed. Raw endpoints, identities and log payloads remain private.
devantler commentedon Oct 25, 2026
Refinement, 25 October: the deciding measurements are not collected
The earlier comment named the measurements that tell a slow disk, a transport problem and a scheduling pause apart: the consensus store's WAL fsync, backend commit and peer round-trip histograms. A source read of
mainfinds no configuration that exposes or scrapes them. There is no metrics-listener setting for the consensus service and no scrape of it. So when the interruption recurred on 15 October, these histograms could not be read, and they will not be there next time either.Proposed first slice (small): expose the consensus service's metrics endpoint on the control-plane nodes, restricted to the in-cluster collector, and scrape those three histogram families into the existing observability stack. Keep the endpoint unauthenticated only on a node-local or policy-restricted address.
This is refinement only. No machine configuration, policy or workload was changed.
chore: agent-claim #4592 nonce=6a7061e7420c00500619065c79b9be49
devantler commentedon Oct 28, 2026
New bounded observation on 28 October, 03:20–03:23 UTC: infrastructure reconciliation encountered three policy-admission request deadlines, delaying a dependent application reconciliation. The API-server log tails independently contain admission timeout records. Both admission replicas and service endpoints were ready, with no observed container restarts; the later resource samples do not reconstruct conditions during the timeout.
At 03:26–03:27 UTC, both affected reconciliation resources reported Ready and Healthy at their current generations, with applied and attempted revisions equal. The native platform health check now passes. Current API readiness, including its consensus checks, also passes. Recovery occurred through the existing reconciliation path; no restart, manual reconcile, policy change or production mutation was performed.
These observations establish an admission interruption and subsequent recovery. They do not establish a consensus fault, its initiating cause, or recurrence prevention. The bounded API tails contain no matching consensus-timeout record, which is not proof that none occurred outside their coverage. Keep this issue open and retain the missing-measurement first slice. Raw logs, request contents and operational identities remain in private diagnostic receipts.
devantler commentedon Nov 4, 2026
Status, 4 November: another recurrence on 2 November, cause still unproven
New recurrence on 2 November, around 01:12 UTC. The evidence is read-only: the API-server logs from 01:08 to 01:16, plus the previous-container logs of the two control-plane components that restarted.
etcdserver: leader changed, so a leadership transition marks the start.etcdserver: request timed outand 18context deadline exceededconsensus errors through 01:12:23. The other two logged two or three each, so the disruption centred on one control-plane node.context deadline exceeded). They exited with code 1 (leaderelection lost) at 01:12:20 and 01:12:23, then restarted. These are their only restarts since 29 October.context canceled) to a single consensus endpoint. Nothing further was logged after that. All three control-plane nodes have stayedReadysince 29 October.Pattern so far. On 6 October (17:54), 15 October (22:00) and 2 November (01:12), a leadership transition came with 10–20 seconds of consensus timeouts and lost controller leases. Separately, 28 October (03:20) had an admission-only interruption.
What would settle it is still missing. As of
mainat c199bf6, the platform neither exposes nor scrapes the consensus metrics endpoint: there is no metrics listener and no scrape, the same as on 25 October. So for every recurrence, the WAL fsync, backend-commit and peer round-trip histograms that would tell a storage stall, a transport fault and a scheduling pause apart do not exist. I filed the first slice from 25 October as #5880, so the next recurrence can be joined to those series.Not established: the initiating cause, which consensus member stalled, and whether the node-local API proxy contributed. This issue stays open. No restart, configuration change or maintenance action was performed, and no endpoint, address or identity is published here.
🤖 Generated with Claude Code