test: add shuffle-fuzzer to find test-order state leaks (#5079) - #5096
Merged
Merged
Conversation
chalfontchubby
added
unit_test
INFRA
3 tasks
Tests share one PredBat/HA fixture, so a test that leaves shared state mutated can make an unrelated later test fail depending on run order - a pattern found repeatedly by manual review but never caught by the suite itself (#5079). unit_test.py gains --shuffle/--shuffle-seed for deterministic random test ordering. tools/shuffle_fuzz.py drives it: --bisect narrows a failing shuffle down to the minimal (culprit, victim) test pair (binary search when a single prior test is the cause, which is the common case in practice), --campaign samples many rounds to separate common leaks from one-off orderings, and --hunt (the normal way to run it, via coverage/run_shuffle --hunt --quick) is a one-liner that finds the next new pair not already recorded in tools/shuffle_fuzz_known_pairs.txt. That file is where --hunt records pairs it has found so each run looks for a different one - it's gitignored rather than committed, so contributors keep their own running list locally instead of it being seeded, fought over on pulls/rebases, or growing unbounded in the repo. Pairs found so far are tracked on issue #5079 instead, where fixed ones can be checked off (load_forecast_history and rate_text_scan were the most repeated culprits and are fixed in a follow-up PR). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
chalfontchubby
force-pushed
the
feat/shuffle-fuzz-test-order-leaks
branch
from
September 14, 2026 19:54
1d84c61 to
5068eae
Compare
This was referenced Sep 14, 2026
bisect_order() assumes the failure it is narrowing is deterministic and caused by something that ran earlier. A test that fails for another reason breaks that assumption: the search still returns a minimal subset, but the "culprit" it lands on is whichever test happened to precede the victim rather than a cause. An overnight hunt produced 12 such pairs - seven naming teslemetry and five naming plan_persistence, with culprits as unrelated as solax, carbon and web_annual. None reproduced. Both victims were in fact failing on the wall clock (plan_persistence builds an "already expired" timestamp from the fixture clock, which storage.load() checks against the real one), so they failed by time of day rather than by ordering. confirm_pair() now checks the victim passes on its own before a pair is recorded, since a victim that fails alone needs no culprit. Failures that cannot be reduced to a reproducing pair are written to --log-dir rather than discarded - a flaky or clock-dependent test is a real bug of a different kind, and previously left no trace once --hunt moved on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 15, 2026
philicibine
pushed a commit
to philicibine/batpred
that referenced
this pull request
Sep 16, 2026
…hecks against (springfall2008#5079) save_plan() computed its 8-hour storage expiry from self.now_utc - Predbat's own simulated/plan clock, which is deliberately not real wall-clock time during a debug-file replay or a test, and can drift from it in production too (e.g. a paused/resumed HA instance). storage.load() checks that expiry against real datetime.now(timezone.utc) though, not self.now_utc - the only expiry-bearing storage.save() call in the codebase using the wrong clock (every other one - github.py, octopus.py, enphase.py, fox.py, kraken.py, solax.py - already computes its expiry from datetime.now(timezone.utc)). Once now_utc drifted from real time by more than 8 hours, a freshly- saved plan looked already-expired the instant it was written, and load_plan() silently no-op'd on its next read. Found via the shuffle-fuzzer (springfall2008#5096): test_plan_persistence was misdiagnosed as order-dependent (three different "culprit" pairs on springfall2008#5079 - octopus_url, octopus_slots_change, clip_export_slots - each failing depending on shuffle order), but every "culprit" turned out to share nothing in common; the actual trigger was time-of-day-dependent - confirmed by pairing plan_persistence with an arbitrary unrelated test (basic_rates) and reproducing the identical failure. All three pairs retired by this fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(Written by Claude, on behalf of Rik.)
Test infrastructure only - no changes to any user-facing code, inverter/component logic, or defaults. Everything here lives in
apps/predbat/unit_test.py(new CLI flags only),tools/, andcoverage/.What and why
All unit tests share one
PredBat/Home Assistant fixture (create_predbat()), so a test that mutates shared state without fully restoring it can make an unrelated later test fail - but only when that specific ordering happens to occur, so a clean./run_alldoesn't prove there isn't one lurking. This is the pattern tracked in #5079: several instances have already been found and fixed by manual review, one at a time, with no way to look for the next one systematically.What's added
unit_test.py:--shuffle/--shuffle-seed N- run the selected tests (whatever--test/-k/the full registry would pick) in a deterministic random order instead of registry order.tools/shuffle_fuzz.py, run viacoverage/run_shuffle:--bisect SHUFFLE_SEED- given a shuffle that's known to fail, narrow down which earlier test caused it, down to a minimal--test A --test Brepro. Uses a binary search when a single prior test is the culprit (the common case observed in practice), falling back to full delta-debugging for multi-test interactions.--campaign N- repeat shuffle-then-bisect for N independent rounds and print a frequency table of (culprit, victim) pairs, to separate a common leak from a one-off ordering artefact.--hunt- the normal way to run it day to day: a one-liner that samples shuffled orderings until it finds a new pair (not already intools/shuffle_fuzz_known_pairs.txt), prints its minimal repro, and records it so the next run finds a different one.docs/developing.md, a pointer fromCLAUDE.md, and an entry intools/debug-journal.md.tools/shuffle_fuzz_known_pairs.txtships empty (just the format comment) - it's meant to be worked down to zero as pairs get fixed, not committed to as a growing list. The pairs found running this locally are tracked on #5079 instead, where they can be checked off.Not proposed as a CI gate - a full shuffled run plus bisection is too slow for per-commit, but it's cheap to run periodically or before a release.
Test plan
./run_pre_commitpasses (ruff, black, cspell, interrogate 100% docstring coverage on the new file)--shuffle/--bisect/--campaign/--hunteach smoke-tested against the real suite and found genuine ordering-dependent failures (see Test infrastructure keeps producing fixture state-leak bugs #5079)