Repository navigation
ci(unit-tests): the Python 3.15 job's p90 is past half its cap (CI timing HARD H1) #2751
Description
Describe the bug
The weekly CI timing run 41705381887 (re-baselined after 21ec0a6f1) failed on HARD H1. The Python 3.15 job in unit-tests.yml has a p90 of 24.8 min. That is 55% of its 45-minute cap, and the trend is +0.63 min/day. Its Run unit tests step takes 3.71× its paired legs (W3), because 3.15 is the LATEST tier and runs the sweeps the other versions skip.
The run's other HARD finding, H4, is the ordering cell protocols (rest) reaching its step cap. 21ec0a6f1 already fixed that by halving the cell; both halves now take about 9–10 min.
Reproduction
The ci-timing-report artifact is on run 41705381887. The slowest 3.15 modules (median of 5 push runs, seconds):
| Module | s |
|---|---|
tests.project.test_sweeps_on_latest_only_1732 |
273.4 |
tests.test_tier_guard |
180.1 |
tests.protocols.internet.test_mh_option_mutation_2684_unit |
128.2 |
tests.protocols.test_frame_layer_limit_1610_unit |
118.5 |
tests.protocols.internet.test_ext_chain_limit_rebuild_1619_unit |
108.7 |
tests.foundation.test_extraction_non_seekable_stream_1586_unit |
105.8 |
Expected behavior
The p90 of the 3.15 job falls back under half its cap. That needs either faster heavy modules (the class-level costs above) or a split of the LATEST-only work into its own job. The latter is how project was split in #2254.
Additional context
#2750, the slowdown caused by heap growth in one process, may be part of this.
Activity
JarryShaw commentedon Nov 3, 2026
Held until #2752 lands. It releases the purged pcapkit generations that slowed long test processes (#2750), and the 3.15 job may share that cause. Its CI time on #2752, against this run's 22.1 min median, decides whether a split is still needed.
JarryShaw commentedon Nov 3, 2026
#2752 ran the 3.15 step in 27.9 min. Other recent PR runs took 19.7, 22.7 and 28.8 min, so the heap fix did not change it, and the split is still needed. A fix is in progress on a branch off
05e113757. It finds the extra LATEST-only work and either gives it its own job, as #2254 did forproject, or makes the slowest modules cheaper.JarryShaw commentedon Nov 3, 2026
The PR is #2754.
ci(unit-tests): run the LATEST tier in a job of its own under coverage (
JarryShaw commentedon Nov 3, 2026
Closing: #2754 merged as
4f62f52ad.Python 3.15now runs only the ALL tier, at 14m50s on the PR (it was 23–31 min). The LATEST tier runs in its own halved job, and the merged coverage matched main at 94.44%. CI Timing will be re-baselined once main's first run on the new shape finishes.JarryShaw commentedon Nov 4, 2026
The re-baseline CI Timing run 41742562185 still fails, on H1 (
Python 3.15p90 25.3 min) and on H2 and H4 (protocols (rest)). Its window is 10-27 → 11-03 23:52Z, which is almost entirely runs from before the fixes.protocols (rest)no longer exists:21ec0a6f1halved it.Python 3.15takes 9m19s on main4f62f52ad, and the LATEST halves take 10m32s and 5m47s. The fixes are already in, so I have not filed a new issue. Next Monday's run measures only the new shape.docs(changelog): fold #2752, #2753, #2754 and the ordering split