Skip to content

ci: cut queue pressure and diagnose slow legs #1052

Description

@JarryShaw

CI is queue-bound. Peak concurrency sits at exactly 20 jobs (5 on macOS), which matches GitHub Free's per-account cap. One PR push starts about 69 jobs, about 61 of them in unit-tests.yml. Every timeout in the last two weeks was a throughput problem, not a hang:

  • the plain-unittest protocols leg slowing 10–15× in the tail of test_mh_unit.py (purge_modules without a module-table restore);
  • apt-get update stalls on 2026-10-01;
  • the old 30-minute cap.

A merge queue was probed and rejected for this user-owned repo (422 Invalid rule 'merge_queue').

Approved changes:

  • A. Run the unittest leg with --verbose, and add faulthandler.dump_traceback_later to it.
  • B. util/run_unittest_leg.py snapshots and restores sys.modules around each test.
  • C. A step timeout of about 20 min on the unittest run step; apt capped at about 5 min, with one retry.
  • D. pytest-timeout for the pytest jobs.
  • E. Fold the 36 engine jobs into about 6, with no loss of coverage.
  • F. Create Release skips its full-suite gate when nothing is publishable.
  • G. Docs-only PRs run only tests/project, the changelog check and the Pages build. required-checks accepts skipped legs only in that case.
  • H. unittest-ordering runs on main pushes and on schedule only.

Kept on purpose: the full run on main pushes. Merges land trees that no PR run tested, so that run is the only test of what lands.

Activity

  1. added
    ciPull requests that change CI or workflow configuration (ci: subject prefix)
    wipWork in flight - a covering PR is open or an agent is actively on it
    on Oct 5, 2026
  2. added this to the 1.5 milestone on Oct 6, 2026
  3. JarryShaw commented on Oct 6, 2026

    @JarryShaw
    OwnerAuthor

    Closed by #1055 (merged 7958b6425): engine jobs folded 36 → 6, a docs-only changes classifier, and required-checks accepting skipped legs only on a docs-only classification. The slow-leg diagnosis landed alongside it in #1059, #1064 and #1070.

  4. removed
    wipWork in flight - a covering PR is open or an agent is actively on it
    on Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciPull requests that change CI or workflow configuration (ci: subject prefix)

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions