Repository navigation
scripts: test the mutants merge as #952 changed it - #1024
Closed
Pedro Henrique Penna (ppenna) wants to merge 1 commit into
Closed
Pedro Henrique Penna (ppenna) wants to merge 1 commit into
Pedro Henrique Penna (ppenna) wants to merge 1 commit into
Conversation
#952 changed the summarize job of nvx-rs-mutants.yml: each shard now stages its cargo-mutants output under its index before the upload, so that one shard's artifact, like several, downloads to shards/<index>/; the merge counts a shard as reported when its outcomes.json is there; and the merge step first fails unless the mutants job succeeded, which it reads from needs.mutants.result as MUTANTS_RESULT. Its tests weren't updated, and it merged without CI, which a fork's pull request runs only once approved. Since then, two of MutantsWorkflowStepTests fail on every Linux host, and so the NVX CLI job of every CI run: test_the_outcomes_of_every_shard_merge and test_shards_that_report_nothing_fail_the_merge run the step without MUTANTS_RESULT, which set -u rejects ("MUTANTS_RESULT: unbound variable"), and write each shard's lists under the old artifact name, which the step no longer reads. _merge now passes the mutants job's result, success unless a test says otherwise, and the tests write each shard's lists with its outcomes.json under its index, as the staging step uploads them. The test of shards that report nothing adds lists without an outcomes.json, which the merge must neither count nor merge, and a new test checks that a mutants job that failed or was skipped fails the merge before it merges or summarizes anything. The tests skip on Windows, which runs no bash steps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the mutation-workflow test updates.
No open issues 🎉
What changed in this PR
Updates the mutation-workflow tests to match the shard layout and merge checks introduced in #952, addressing the reported NVX CLI test failures.
Changes:
- Supplies
MUTANTS_RESULTand uses index-based shard fixtures containingoutcomes.json. - Verifies that lists without
outcomes.jsonare neither counted nor merged. - Adds coverage for failed and skipped mutants jobs, checking that merging and summarization do not occur.
| File | Description |
|---|---|
scripts/test_nvx_rs_mutants.py |
Aligns merge fixtures with the workflow and expands failure-path assertions. |
Review effort: Balanced
Contributor
Author
|
Superseded by #1023, which merged at 21:58Z with the same fix to \scripts/test_nvx_rs_mutants.py\ while this was open. On nvx-mshv-test, \dev\ now passes \MutantsWorkflowStepTests\ again, so this is no longer needed. |
Pedro Henrique Penna (ppenna)
deleted the
agents/fix-mutants-summarize-test
branch
October 29, 2026 22:00
Pedro Henrique Penna (ppenna)
restored the
agents/fix-mutants-summarize-test
branch
October 29, 2026 22:41
Pedro Henrique Penna (ppenna)
deleted the
agents/fix-mutants-summarize-test
branch
October 29, 2026 22:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#952 changed the summarize job of
.github/workflows/nvx-rs-mutants.ymlbut not its tests, and merged without CI, which a fork's pull request runs only once approved. Since then,devfails two Linux tests thatvalidate-nvxruns, so the NVX CLI job of every CI run fails:MutantsWorkflowStepTests.test_the_outcomes_of_every_shard_mergeMutantsWorkflowStepTests.test_shards_that_report_nothing_fail_the_mergeBoth run the merge step without
MUTANTS_RESULT, which itsset -urejects (MUTANTS_RESULT: unbound variable). They also write each shard's lists under the old artifact name, which the step no longer reads: since #952, a shard's output lands atshards/<index>/, and the merge counts a shard as reported when itsoutcomes.jsonis there.Changes
_mergepasses the mutants job's result asMUTANTS_RESULT,successunless a test says otherwise.outcomes.jsonunder its index, as the staging step uploads them.outcomes.json, which the merge must neither count nor merge.Validation
nvx-mshv-test(Azure Linux 3, Python 3.12),python3 -m unittest scripts/test_nvx_rs_mutants.pypasses all 11 tests. Atdev(d049844), the two tests above fail.ruff check,ruff format --check, andpyright(Linux and Windows) pass on the file.