Test isolation: reset reachability circuit-breaker registry between tests (kills monolithic pytest flakiness)

[jgruberf5]

Problem

Running the backend suite as one process (pytest tests/ / make test-backend) produces order-dependent failures that do NOT occur in CI. CI runs segmented buckets in separate processes (make test-backend-unit = tests/unit/, test-backend-component = tests/component/, test-backend-legacy = flat tests/test_*.py, integration -m full), so cross-process global-state pollution never happens there.

Observed during WO-1 (PR #364): a monolithic pytest tests/ run shows 13 failures; the 3 tests/test_helm_service.py::TestListReleases ones fail with:

core.errors.BreakerOpenError: Cluster 'cluster:1' is unreachable
  services/reachability/breaker.py:254

Root cause

The reachability circuit breaker uses a process-global singleton registry (services/reachability/registry.registry) keyed by cluster:{id}. An earlier integration test trips the breaker OPEN for cluster:1 and nothing resets it between tests (no conftest fixture). Later tests whose fixtures reuse cluster id 1 (e.g. helm's make_k8s_cluster) then short-circuit with BreakerOpenError before their mocked logic runs. The breaker has a time-based cooldown, so the failure is sensitive to suite ordering/timing.

Fix

Add an autouse fixture (in backend/tests/conftest.py) that resets/clears the reachability breaker registry before (and/or after) each test, so breaker state never leaks across tests. Confirm pytest tests/ (monolithic) then runs clean, matching the per-bucket CI result.

Why it matters

• Local pytest tests/ is a common dev workflow; the false failures erode trust and waste debugging time (cost a full WO-1 verification cycle to diagnose).
• A reset fixture is a small, low-risk test-infra change benefiting everyone.

Notes

• Not a product bug — the breaker behaves correctly at runtime (separate process per request lifecycle). Purely test isolation.
• The 10 other monolithic-only failures (benchmarks-token, blueprint_catalog, module_sources, kube_context_threading) are pre-existing and also order-dependent; the breaker reset may help some, others may need their own isolation fixes — investigate while here.

Follow-up from WO-1 / PR #364.

---

_{Migrated from sp-prod-field/bnk-forge #366 (opened 2026-06-24; original labels: needs-triage). That repository is archived and read-only.
Bare #NNN references in the text above refer to issues and PRs in the original repository, not to numbering here.}

[jgruberf5]

Analysis: this is worth prioritising above its apparent size.

The failure is invisible in CI by construction — CI runs segmented buckets in separate processes, so the shared circuit-breaker registry never leaks between them. It only bites developers running the suite as one process locally, which is exactly the workflow the contributing docs recommend before pushing.

That mismatch (green in CI, red locally) tends to erode trust in the suite, and it makes any real order-dependent bug much harder to spot. Related: #272 is the same class on a different registry.

[jgruberf5]

Fix in the PR above. One finding worth recording here separately: with the breaker reset in place, the monolithic pytest tests/ run (9717 tests, one process) goes from the 13 failures described to 2 — and those 2 are a different leak, not this one:

• tests/unit/test_kube_context_threading.py::test_get_api_passes_context_to_kr8s
• tests/unit/test_kube_context_threading.py::test_get_api_omits_context_when_unset

Both pass in isolation, both fail only in the monolithic run, and the file never touches the reachability registry. So it's some other process-global state (a patched kr8s client or a cached kube context, most likely). Out of scope for this issue; flagging so it isn't assumed covered, and it probably deserves its own ticket once someone bisects which earlier test poisons it.

[github-actions]

Auto-closed by PR #168 ("Reset the reachability circuit-breaker registry between tests") merged to staging. (Native Closes-#N only fires on the default branch; this Action covers staging.)