Skip to content

scripts: test the mutants merge against the shards' indexed outcomes - #1023

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
devfrom
agents/fix-952-mutants-merge-tests
Oct 29, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
devfrom
agents/fix-952-mutants-merge-tests

Conversation

@ppenna

Copy link
Copy Markdown
Contributor

#952 changed how the summarize job of nvx-rs-mutants.yml finds the shards' outcomes:

  • Each shard uploads its outcomes under its index.
  • The download merges every shard's artifact into shards/.
  • The merge counts a shard as reported when shards/<index>/outcomes.json exists.
  • The merge step fails before it merges anything when the mutants job did not succeed, which it reads from MUTANTS_RESULT under set -u.

MutantsWorkflowStepTests in scripts/test_nvx_rs_mutants.py still ran the merge step with the old layout, one directory named after each artifact, and without MUTANTS_RESULT. On Linux, where these tests run (they skip on Windows), two of them fail on dev:

  • test_the_outcomes_of_every_shard_merge: /usr/bin/bash: line 2: MUTANTS_RESULT: unbound variable
  • test_shards_that_report_nothing_fail_the_merge: '::error::Only 1 of 3 shards reported' not found in ''

So NVX CLI / Linux fails for every pull request based on dev, as in run 40958853538 of #1021.

Changes

  • scripts: test the mutants merge against the shards' indexed outcomes: the tests lay the outcomes out as the download does, under each shard's index and with outcomes.json, and pass the mutants job's result. The test of missing shards also gives one shard lists without outcomes.json, which the merge must leave out of both the lists and the count. A new test requires that a mutants job that failed, was cancelled, or was skipped fail the merge before it merges or summarizes anything.

The workflow itself is unchanged.

Validation

  • On the nvx-kvm-test Linux VM, the two tests fail at b558d34bd, which merged ci: fix nvx-rs mutants summarize miscounting shards #952, and pass at its parent, 29c68547b.
  • At this branch's head, all 11 tests of scripts/test_nvx_rs_mutants.py pass there, and so does the whole validate-nvx unittest list: 1,484 tests, 34 skipped.
  • Locally on Windows: ruff check, ruff format --check, and pyright --pythonplatform Linux and Windows are clean, and the module's Windows tests pass.

#952 changed how the summarize job of nvx-rs-mutants.yml finds the
outcomes of the shards. Each shard now uploads its outcomes under its
index, the download merges the artifacts of all the shards into
shards/, and the merge counts a shard as reported when
shards/INDEX/outcomes.json exists. The merge step also fails before it
merges anything when the mutants job did not succeed, which it reads
from MUTANTS_RESULT under set -u.

MutantsWorkflowStepTests still ran the merge step with each shard's
lists in a directory named after its artifact, and without
MUTANTS_RESULT. On Linux, where these tests run,
test_the_outcomes_of_every_shard_merge failed with "MUTANTS_RESULT:
unbound variable", and test_shards_that_report_nothing_fail_the_merge
did not find its "Only 1 of 3 shards reported" error. Both fail at
b558d34, which merged #952, and pass at its parent, 29c6854; they
failed NVX CLI / Linux in CI run 40958853538 of #1021.

The tests now lay the outcomes out as the download does, under the
index of each shard and with outcomes.json, and pass the mutants job's
result. The test of missing shards also gives one shard lists without
outcomes.json, which the merge must leave out of both the lists and
the count. A new test requires that a mutants job that failed, was
cancelled, or was skipped fail the merge before it merges or
summarizes anything.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 29, 2026 21:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The mutation-workflow test updates have no active review issues.

No open issues 🎉

What changed in this PR

Updates the mutation-workflow tests to match the indexed shard layout introduced in #952, fixing stale test assumptions without changing the workflow itself.

Changes:

  • Stages shard fixtures under their indexes with outcomes.json and supplies MUTANTS_RESULT.
  • Verifies that shards without outcomes.json are excluded from merged lists and the reported count.
  • Adds coverage ensuring failed, cancelled, or skipped mutants jobs stop before merging or summarizing.
File Description
scripts/​test_nvx_rs_mutants.py Aligns merge fixtures with the workflow and strengthens missing-shard and unsuccessful-job checks.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants