Repository navigation
test: three modules pass alone but fail together, and pytest reports it green #981
Description
Activity
wip or blocked?
Neither, by the taxonomy — and that is the honest answer rather than a dodge.
wipmeans an issue
has a covering pull request;blockedmeans something checkable is in the way. This has neither:covering PR for #981 none (the #719 excepthook change touches tests/utilities/ but does not fix this; #457 is an unrelated merged one) blocker none — nothing is in the way still reproduces on main yes — Ran 73 tests ... FAILED (failures=1, errors=3)It is idle, and the reason it is idle is me: I filed it with three candidate approaches and no
recommendation, which left it waiting on you rather than on work. That was the wrong shape for a defect
report. So I am picking one and dispatching, which makes the labelwiptruthfully rather than
decoratively.The approach I am taking, and why it changed since I filed it. The defect found a second instance
today:tests/utilities/test_decorators.pyleakssys.tracebacklimitonmain— it builds a loud
StructErrorwith noquiet=Trueand no teardown, and 22 tests pass while leaving the global set. I
confirmed it still leaks onmainright now, with notearDownoraddCleanupin the file.Two independent instances makes this systemic rather than incidental, so the per-test fix is not
enough on its own. Both leaks are invisible to pytest, which reports a parent node aspassedwhen
only its subTests fail — so CI cannot see either one. I am therefore doing both halves: fix the leaking
teardowns, and add a plain-unittestleg to CI so the class stops being invisible. Fixing the two
known leaks without that leaves the next one undetectable.Labelled
wip. The#719traceback work already carries thetest_decorators.pyteardown as a
by-product, so I will scope this to the registry-isolation half and the CI leg to avoid a conflict, and
say so on the pull request.- added 10 commits that reference this issue
on Oct 2, 2026 2 remaining items
Closing — the leftover landed. #988 merged at 15:57:55Z as
efc3b08c2, and I verified onmainthat
tests/vendoris now in theunittest-orderingmatrix: 10 legs,vendorpresent.The whole of this issue is now done, by three merges rather than one:
- ci(tests): add a plain-unittest leg to catch conftest-masked ordering defects #984 (
32bcfba15) fixed the defect itself —RegistrationGateTestsre-resolves its gate classes in
setUp, so a sibling module'spurge_modulescannot leave them bound to a stale generation — and added the
unittest-orderingjob, which runs eachtests/subdirectory under plainunittestwhere
tests/conftest.py's autouserestore_module_tableis not there to mask the skew. - test(vendor): re-resolve VendorRuntimeWarning fresh per generation (#985) #986 (
6f41995b8) fixed the second instance this issue exposed, filed as test(vendor): a stale warning-class generation makes the snapshot-restore assertion order-dependent #985: a stale
VendorRuntimeWarningmakingassertWarnsRegexfail intests/vendor. - ci(tests): run tests/vendor in the unittest-ordering job (#981, #985) #988 (
efc3b08c2) re-enabled thetests/vendorleg, whose exclusion existed only for test(vendor): a stale warning-class generation makes the snapshot-restore assertion order-dependent #985.
Two measurements worth keeping, because they are what justified closing rather than guessing. On current
main,python -m unittest discover -s tests/vendorgivesRan 118 tests in 83.236s ... OK; as a leg, with the
five root modules, it is 297 tests in ~104s. And the defect class is structurally absent from that
directory, not merely unobserved — an AST sweep of all 14 modules, including statements nested in module-level
try/if/withand in class bodies, found zero module-levelpcapkit-reaching imports, so there is no
import-time binding left for a sibling purge to desync. The only two generation-sensitive assertions resolve
their class per test.Two things deliberately out of scope and still true, so the next reader does not mistake them for
regressions:tests/corekitstays excluded from the leg — its 5 identity failures are a pre-existing, documented
limitation recorded intests/_support.py'spurge_modulesdocstring; andtests/integrationstays excluded
becauseis_unit_tierrejects all 12 of its modules, so the leg would collect nothing from it.One correction recorded against my own work here: I reported mid-investigation that
tests/vendorstill failed
onmainafter #986. It did not — I had measured from a checkout four merges stale, sincegit fetchupdates
the remote ref and leaves the working tree alone.- ci(tests): add a plain-unittest leg to catch conftest-masked ordering defects #984 (
- added a commit that references this issue
on Oct 2, 2026 - added 7 commits that reference this issue
on Oct 6, 2026
Three test modules pass individually and fail when run in one process. pytest does not report the
failure at all, which is what makes it worth filing rather than just fixing.
Reproduced on
main(c49500438), repo venv,PYTHONPATHat the repo root:The failing assertion is
RegistrationGateTests.test_user_style_subclass_registers_when_it_opts_in,at
suite='dumpers':So a dumper the test registers under its opt-in keyword is absent from the registry by the time the
assertion runs, once the other two modules have imported first. Four subTests fail in total —
engines,reassembly,traceflow,dumpers.Two things make this more than a flaky ordering nuisance.
The file's own
.. note::predicts cross-module registry interference, so the mechanism is known; whatis not recorded is that it currently fires. And pytest reports the run fully green — a parent node
reads
passedwhen only its subTests failed, so CI cannot see this. Any suite-wide ordering regressionof this shape is invisible to the checks we run.
Not caused by any open change: I reproduced it at the merge base before the three prose PRs in flight.
Worth deciding separately whether the fix is isolating the registry per module, making the assertion
order-independent, or adding a plain-
unittestleg to CI so the class of defect stops being invisible.