fix(homepage): Flagger canary wedged in Failed since 06-27 — new revisions (#2322, #2336) never restart analysis #2387

🤖 Generated by the Daily AI Assistant

Summary

The homepage Flagger canary has been wedged in Failed since 2026-06-27T23:05:46Z, and Flagger has not restarted analysis for either of the two real revisions shipped since — #2322 (Enable Banking bookmark, 06-28) and #2336 (hostUsers: false user namespaces, 06-30). The apex dashboard's homepage-primary is pinned to the pre-06-27 template; both merged changes are inert on prod. Any future homepage rollout will silently join the queue.

Evidence (live, 2026-07-02 ~11:30Z)

• Canary homepage/homepage: phase=Failed, condition Promoted=False "Canary analysis failed, Deployment scaled to zero." — lastTransitionTime/lastUpdateTime 2026-06-27T23:05:46Z, untouched since (status lastTransitionTime ticks every ~1-2m, so Flagger IS reconciling it).
• iterations=0, failedChecks=0, lastAppliedSpec=9bc949574 ≠ lastPromotedSpec=78cdc7b5f8.
• Source Deployment/homepage: replicas=0 (scaled down by the failure), template DOES carry hostUsers: false (generation 45, observed 45) — the HelmRelease applied feat(homepage): run inside a user namespace (hostUsers: false) #2336 fine; only the Flagger promotion is missing.
• ScaledObject/homepage: autoscaling.keda.sh/paused-replicas: "0" (Flagger's between-rollouts pause).
• homepage-primary pods (2× Running): hostUsers unset — old template.
• Flagger leader log (pod up since 06-30 ~18:25Z, i.e. covering feat(homepage): run inside a user namespace (hostUsers: false) #2336's merge 17:44Z): the ONLY homepage line is "Synced homepage/homepage" at 06-30T18:27 — no "New revision detected" for feat(homepage): add Enable Banking bookmark link #2322 or feat(homepage): run inside a user namespace (hostUsers: false) #2336, while umami-umami promoted normally 07-01T18:27 (Flagger 1.43.0 itself is healthy).

Root-cause hypotheses (pick during fix)

1. Restart-from-Failed never fires when the source is scaled to 0 / scaler paused at 0 — Flagger's readiness gate on the target deployment blocks shouldAdvance before the revision check (KEDA autoscalerRef + blue/green provider: kubernetes deadlock: Failed → scale-to-0 + paused-replicas: 0 → target never Ready → revision change never evaluated). The silent logs fit a debug-level "not ready" path.
2. lastAppliedSpec is being refreshed in the Failed phase (recording new spec hashes without running analysis), defeating change detection.

Also unknown: what failed at 06-27T23:05Z in the first place (events expired; merges that evening: #2313 VPA 22:35Z, #2315 naming refactor 23:07Z; trackedConfigs shows configmap/homepage tracked). Worth checking the loadtester + Coroot metric templates when reproducing.

Immediate unwedge (operator kick — safe, blue/green means user traffic stays on primary throughout)

kubectl --context admin@prod -n homepage annotate scaledobject homepage autoscaling.keda.sh/paused-replicas=1 --overwrite

lets the source deployment become Ready so Flagger can (a) evaluate the new revision and (b) run the pre-rollout/load-test webhooks against homepage-canary. Flagger re-asserts the pause after promotion. If the analysis then fails again, the failure output is the missing 06-27 root cause. (A bare kubectl rollout restart deploy/homepage is NOT sufficient if hypothesis 1 holds — the target can never become Ready at 0 replicas.)

Durable fix / follow-ups

• Root-cause against Flagger 1.43.0 source; if hypothesis 1 confirms, this is an upstream Flagger bug (restart-from-Failed with autoscalerRef paused at 0) — upstream issue/PR is maintainer-gated per contract (ask-tool first).
• Prevention: a wedged canary was invisible for 5 days. Add an alert on Flagger's flagger_canary_status metric (phase Failed > threshold) via the existing Coroot/Flux alerting path so a Failed canary pages instead of rotting.
• Validation once promoted: homepage-primary pods show hostUsers: false → closes the "extend to homepage" checkbox chain of k8s 1.36 readiness: clean on removals; pilot user namespaces, watch VolumeGroupSnapshot #1807 (whoami pilot already validated live: Running, 0 restarts, no warnings; id-in-container check still open — needs an exec/debug the autonomous agent is not permitted).

Related: #1807 (userns epic), #2336 (inert userns change), #2322 (inert bookmark), #1808 (whoami pilot, promoted fine — whoami has no Canary).