Repository navigation
scripts: test the mutants merge against the shards' indexed outcomes - #1023
Merged
Pedro Henrique Penna (ppenna) merged 1 commit intoOct 29, 2026
Merged
Conversation
#952 changed how the summarize job of nvx-rs-mutants.yml finds the outcomes of the shards. Each shard now uploads its outcomes under its index, the download merges the artifacts of all the shards into shards/, and the merge counts a shard as reported when shards/INDEX/outcomes.json exists. The merge step also fails before it merges anything when the mutants job did not succeed, which it reads from MUTANTS_RESULT under set -u. MutantsWorkflowStepTests still ran the merge step with each shard's lists in a directory named after its artifact, and without MUTANTS_RESULT. On Linux, where these tests run, test_the_outcomes_of_every_shard_merge failed with "MUTANTS_RESULT: unbound variable", and test_shards_that_report_nothing_fail_the_merge did not find its "Only 1 of 3 shards reported" error. Both fail at b558d34, which merged #952, and pass at its parent, 29c6854; they failed NVX CLI / Linux in CI run 40958853538 of #1021. The tests now lay the outcomes out as the download does, under the index of each shard and with outcomes.json, and pass the mutants job's result. The test of missing shards also gives one shard lists without outcomes.json, which the merge must leave out of both the lists and the count. A new test requires that a mutants job that failed, was cancelled, or was skipped fail the merge before it merges or summarizes anything. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The mutation-workflow test updates have no active review issues.
No open issues 🎉
What changed in this PR
Updates the mutation-workflow tests to match the indexed shard layout introduced in #952, fixing stale test assumptions without changing the workflow itself.
Changes:
- Stages shard fixtures under their indexes with
outcomes.jsonand suppliesMUTANTS_RESULT. - Verifies that shards without
outcomes.jsonare excluded from merged lists and the reported count. - Adds coverage ensuring failed, cancelled, or skipped mutants jobs stop before merging or summarizing.
| File | Description |
|---|---|
scripts/test_nvx_rs_mutants.py |
Aligns merge fixtures with the workflow and strengthens missing-shard and unsuccessful-job checks. |
Review effort: Balanced
Pedro Henrique Penna (ppenna)
deleted the
agents/fix-952-mutants-merge-tests
branch
October 29, 2026 21:58
This was referenced Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#952 changed how the
summarizejob ofnvx-rs-mutants.ymlfinds the shards' outcomes:shards/.shards/<index>/outcomes.jsonexists.MUTANTS_RESULTunderset -u.MutantsWorkflowStepTestsinscripts/test_nvx_rs_mutants.pystill ran the merge step with the old layout, one directory named after each artifact, and withoutMUTANTS_RESULT. On Linux, where these tests run (they skip on Windows), two of them fail ondev:test_the_outcomes_of_every_shard_merge:/usr/bin/bash: line 2: MUTANTS_RESULT: unbound variabletest_shards_that_report_nothing_fail_the_merge:'::error::Only 1 of 3 shards reported' not found in ''So
NVX CLI / Linuxfails for every pull request based ondev, as in run 40958853538 of #1021.Changes
scripts: test the mutants merge against the shards' indexed outcomes: the tests lay the outcomes out as the download does, under each shard's index and withoutcomes.json, and pass the mutants job's result. The test of missing shards also gives one shard lists withoutoutcomes.json, which the merge must leave out of both the lists and the count. A new test requires that a mutants job that failed, was cancelled, or was skipped fail the merge before it merges or summarizes anything.The workflow itself is unchanged.
Validation
nvx-kvm-testLinux VM, the two tests fail atb558d34bd, which merged ci: fix nvx-rs mutants summarize miscounting shards #952, and pass at its parent,29c68547b.scripts/test_nvx_rs_mutants.pypass there, and so does the wholevalidate-nvxunittest list: 1,484 tests, 34 skipped.ruff check,ruff format --check, andpyright --pythonplatform LinuxandWindowsare clean, and the module's Windows tests pass.