Skip to content

scripts: test the mutants merge as #952 changed it - #1024

Closed
Pedro Henrique Penna (ppenna) wants to merge 1 commit into
devfrom
agents/fix-mutants-summarize-test
Closed

Pedro Henrique Penna (ppenna) wants to merge 1 commit into
devfrom
agents/fix-mutants-summarize-test

Conversation

@ppenna

Copy link
Copy Markdown
Contributor

Summary

#952 changed the summarize job of .github/workflows/nvx-rs-mutants.yml but not its tests, and merged without CI, which a fork's pull request runs only once approved. Since then, dev fails two Linux tests that validate-nvx runs, so the NVX CLI job of every CI run fails:

  • MutantsWorkflowStepTests.test_the_outcomes_of_every_shard_merge
  • MutantsWorkflowStepTests.test_shards_that_report_nothing_fail_the_merge

Both run the merge step without MUTANTS_RESULT, which its set -u rejects (MUTANTS_RESULT: unbound variable). They also write each shard's lists under the old artifact name, which the step no longer reads: since #952, a shard's output lands at shards/<index>/, and the merge counts a shard as reported when its outcomes.json is there.

Changes

  • _merge passes the mutants job's result as MUTANTS_RESULT, success unless a test says otherwise.
  • The tests write each shard's lists with its outcomes.json under its index, as the staging step uploads them.
  • The test of shards that report nothing adds lists without an outcomes.json, which the merge must neither count nor merge.
  • A new test checks that a mutants job that failed or was skipped fails the merge before it merges or summarizes anything.

Validation

  • On nvx-mshv-test (Azure Linux 3, Python 3.12), python3 -m unittest scripts/test_nvx_rs_mutants.py passes all 11 tests. At dev (d049844), the two tests above fail.
  • ruff check, ruff format --check, and pyright (Linux and Windows) pass on the file.
  • The tests skip on Windows, which runs no bash steps.
  • A change to the CLI tests alone runs only the GitHub-hosted CI jobs.

#952 changed the summarize job of nvx-rs-mutants.yml: each shard now
stages its cargo-mutants output under its index before the upload, so
that one shard's artifact, like several, downloads to shards/<index>/;
the merge counts a shard as reported when its outcomes.json is there;
and the merge step first fails unless the mutants job succeeded, which
it reads from needs.mutants.result as MUTANTS_RESULT. Its tests weren't
updated, and it merged without CI, which a fork's pull request runs
only once approved. Since then, two of MutantsWorkflowStepTests fail on
every Linux host, and so the NVX CLI job of every CI run:
test_the_outcomes_of_every_shard_merge and
test_shards_that_report_nothing_fail_the_merge run the step without
MUTANTS_RESULT, which set -u rejects ("MUTANTS_RESULT: unbound
variable"), and write each shard's lists under the old artifact name,
which the step no longer reads.

_merge now passes the mutants job's result, success unless a test says
otherwise, and the tests write each shard's lists with its outcomes.json
under its index, as the staging step uploads them. The test of shards
that report nothing adds lists without an outcomes.json, which the
merge must neither count nor merge, and a new test checks that a
mutants job that failed or was skipped fails the merge before it merges
or summarizes anything.

The tests skip on Windows, which runs no bash steps.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 29, 2026 21:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the mutation-workflow test updates.

No open issues 🎉

What changed in this PR

Updates the mutation-workflow tests to match the shard layout and merge checks introduced in #952, addressing the reported NVX CLI test failures.

Changes:

  • Supplies MUTANTS_RESULT and uses index-based shard fixtures containing outcomes.json.
  • Verifies that lists without outcomes.json are neither counted nor merged.
  • Adds coverage for failed and skipped mutants jobs, checking that merging and summarization do not occur.
File Description
scripts/​test_nvx_rs_mutants.py Aligns merge fixtures with the workflow and expands failure-path assertions.

Review effort: Balanced

@ppenna

Copy link
Copy Markdown
Contributor Author

Superseded by #1023, which merged at 21:58Z with the same fix to \scripts/test_nvx_rs_mutants.py\ while this was open. On nvx-mshv-test, \dev\ now passes \MutantsWorkflowStepTests\ again, so this is no longer needed.

@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/fix-mutants-summarize-test branch October 29, 2026 22:00
@ppenna
Pedro Henrique Penna (ppenna) restored the agents/fix-mutants-summarize-test branch October 29, 2026 22:41
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/fix-mutants-summarize-test branch October 29, 2026 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants