Skip to content

[None][infra] select Python tests by recorded coverage symbols - #22072

Draft
crazydemo wants to merge 1 commit into
NVIDIA:mainfrom
crazydemo:codex/cbts-agent-pr-selection
Draft

crazydemo wants to merge 1 commit into
NVIDIA:mainfrom
crazydemo:codex/cbts-agent-pr-selection

Conversation

@crazydemo

Copy link
Copy Markdown
Collaborator

Description

CBTS currently tries to infer runtime consumers through callers, decorators, cached values, reflection and shared pytest state. These bounds can force full CI even when the changed code has directly recorded coverage. Full CI also does not establish whether newly introduced paths have adequate assertions.

Map verified Python patches to the union of directly changed symbols in the old and new source images, then query the coverage DB by (file, qualname). Missing file/function records contribute no inferred tests. Deleted functions use their old symbols. Module/class/definition-time changes additionally require a registered import-sanity inventory. For example, a cached method body change selects its recorded tests without traversing cache-hit callers; changing a module binding queries <module> and retains import sanity without inferring every reader.

Preserve PR-added/modified tests, explicit rule contributions, unknown test entries, incomplete collection records, shared-block protections, CPU obligations and the existing multi-GPU gate. Missing or unusable source/patch data, unsupported residual files and incompatible coverage baselines still prevent narrowing. The legacy --no-data-policy values remain accepted; all now use the lookup-only behavior and the default becomes ignore.

Add cold subprocess imports and a unified 28-selector sanity inventory, with CPU/package-sanity registration and the missing H100 registration. Retain registered test consumers for explicitly mapped test infrastructure. Update selection diagnostics, documentation and regression coverage.

Test Coverage

  • Complete CBTS unit suite: 1027 passed, 6 skipped, 2 subtests passed. Includes old/new diff attribution, missing coverage rows, deletions, import obligations, test retention and infrastructure rules.
  • Repository pre-commit and DCO hooks passed. The committed branch merges cleanly with upstream main 5f66e8a10e in a local merge-tree check.
  • Fixed historical replay: the same 24 PR snapshots and DB3063 (2d5b1fb68fee16b49b43f501d9165af23fb83c33), pre-merge mode, multi-GPU gate closed. Both policies use the same complete sanity registrations with matching CPU/non-CPU marker eligibility. DB SHA256: b37654fac2fe0fe102ba1cda82f52b6208508258590c59f75b0344ea1591a698.
Fixed-cohort metric Previous policy Lookup-only policy
Selector hits 14/24 22/24
Hits after baseline compatibility 14/24 (58.3%) 21/24 (87.5%)
Remaining fallback 10 3
Selected YAML entries after fallback 26,186 / 40,777 25,204 / 40,777
Effective weighted skip rate 35.78% 38.19%

Seven additional compatible hits: #20903, #21204, #21220, #21463, #21192, #21466 and #21121. No previously compatible hit was lost. Remaining fallback: #21280 has C++ residual changes; #20104 has CUDA residual changes; #21225 has a patch conflict with the coverage baseline in mega_moe_cute_dsl.py.

The replay checked 1,008 required sanity-retention obligations and 8,274 direct-coverage or untrusted-entry obligations; 9 replay-alignment helper tests passed, with no replay errors. Counts are weighted YAML entries, not expanded pytest cases or measured GPU savings. The replay executes the selector, not historical product builds. The real native/CUDA cold-import inventory has not been executed at this PR head; passing retained or full CI does not prove coverage of new paths.

Fixed replay cohort and per-PR outcomes
PR Frozen head Previous scope Lookup-only scope Baseline compatible Selected / total entries
#21777 d9a177c57a coverage coverage yes 1597 / 1767
#20903 042776e170 fallback coverage yes 1647 / 1767
#21704 3b4bbce3d1 testsonly testsonly yes 128 / 1768
#21631 17b03c69d7 coverage coverage yes 1430 / 1775
#21204 3b74af0c2a fallback coverage yes 1404 / 1769
#21874 31903dc655 noop noop yes 0 / 1775
#21692 4f9a5bda8e testdefonly testdefonly yes 44 / 1768
#21220 9ae1819125 fallback coverage yes 1520 / 1611
#21280 5208f1284d fallback fallback no 1635 / 1635
#21463 004f6cc4ec fallback coverage yes 1628 / 1724
#20104 b7dae02645 fallback fallback no 1281 / 1281
#21144 21f0dbd18b coverage coverage yes 1661 / 1764
#21192 448febe2a6 fallback coverage yes 1710 / 1715
#21249 7017065e01 testsonly testsonly yes 118 / 1720
#20918 d246bdaa61 coverage coverage yes 1386 / 1722
#20089 fc65496d9f coverage coverage yes 1392 / 1617
#21558 4760c852f0 testsonly testsonly yes 259 / 1718
#21466 63a7f7d13a fallback coverage yes 1557 / 1712
#21199 62902cec2a coverage coverage yes 1551 / 1716
#21121 8a28b0216d fallback coverage yes 1541 / 1700
#21465 bc77820e71 testsonly testsonly yes 2 / 1712
#21302 4861cb1059 noop noop yes 0 / 1635
#21225 90e95a4c0d fallback coverage no 1559 / 1706
#21387 e9799f0706 waiveonly waiveonly yes 7 / 1700

The table shows raw selector counts; effective totals above restore the full entry count for baseline-incompatible selections.

PR Checklist

  • Reviewed coding guidelines and updated selection documentation.
  • Added regression coverage and CI registration for the new sanity test.
  • No public product API, dependency, ownership or model-architecture changes.

Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant