Problem
As each host finishes reconciling, the operator writes the entire CHI .status back with a read-modify-write (pkg/controller/chi/kube/cr.go, statusUpdateProcess): Get the current CHI → EnsureStatus().CopyFrom(...) → full Update, wrapped in statusUpdateRetry (up to maxRetryAttempts = 50, with a sleep between attempts).
When many hosts reconcile concurrently, they all update the same single CHI object. Kubernetes allows only one update to an object at a time (optimistic concurrency), so the others get:
Operation cannot be fulfilled on clickhouseinstallations.clickhouse.altinity.com "<chi>":
the object has been modified; please apply your changes to the latest version and try again
…and then re-Get and retry. Under concurrency this leads to repeated write conflicts: a single host's status write can take several attempts (observed attempt 2, 3, 4, 6, …) with backoff between each.
Impact
At higher reconcile concurrency (>50) the per-host status write can become a significant per-host cost (observed in the tens of seconds), most of it spent on conflict retries and backoff rather than the update itself. It also results in repeated full-object writes to the apiserver. Because the contention comes from concurrency, the cost tends to increase as reconcile concurrency is raised.
Fix ideas
- Use a status subresource
Patch (JSON/strategic-merge patch of just the changed fields) instead of a read-modify-write of the whole .status.
- Or coalesce/batch host-completion status updates rather than one full-object write per completing host (for example, a single writer that flushes accumulated per-host transitions).
Problem
As each host finishes reconciling, the operator writes the entire CHI
.statusback with a read-modify-write (pkg/controller/chi/kube/cr.go,statusUpdateProcess):Getthe current CHI →EnsureStatus().CopyFrom(...)→ fullUpdate, wrapped instatusUpdateRetry(up tomaxRetryAttempts = 50, with a sleep between attempts).When many hosts reconcile concurrently, they all update the same single CHI object. Kubernetes allows only one update to an object at a time (optimistic concurrency), so the others get:
…and then re-
Getand retry. Under concurrency this leads to repeated write conflicts: a single host's status write can take several attempts (observed attempt 2, 3, 4, 6, …) with backoff between each.Impact
At higher reconcile concurrency (>50) the per-host status write can become a significant per-host cost (observed in the tens of seconds), most of it spent on conflict retries and backoff rather than the update itself. It also results in repeated full-object writes to the apiserver. Because the contention comes from concurrency, the cost tends to increase as reconcile concurrency is raised.
Fix ideas
Patch(JSON/strategic-merge patch of just the changed fields) instead of a read-modify-write of the whole.status.