Skip to content

chore: add mutants-review agent skill - #4375

Open
jarhodes314 wants to merge 4 commits into
thin-edge:mainfrom
jarhodes314:jarhodes314/cargo-mutants-review-skill
Open

jarhodes314 wants to merge 4 commits into
thin-edge:mainfrom
jarhodes314:jarhodes314/cargo-mutants-review-skill

Conversation

@jarhodes314

@jarhodes314 jarhodes314 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Proposed changes

Adds /mutants-review [PR], a user-invoked agent skill in .agents/skills/, that runs cargo-mutants on a PR's diff and drafts tests for the mutants that survive.

/mutants-review <pr>
  check out the PR head in a temporary worktree
  cargo mutants --in-diff <pr diff> --test-tool nextest --all-features
  for each missed mutant
    gap            -> write the smallest test that catches it
    equivalent     -> one-line reason
    not worth it   -> one-line reason
  rerun cargo mutants on those mutants with the tests added
    the baseline proves each test passes on the real code
    caught.txt proves each test catches its mutant
  add [AI] comments to the user's pending review (after they approve)

The user's checkout is never touched, and nothing is visible to the PR author until the user submits the review.

Tried on #4373: cargo mutants --in-diff found 8 mutants (5 caught, 2 missed, 1 unviable). For the missed replace is_connection_closed -> bool with true, the skill wrote connection_refused_is_not_treated_as_a_closed_connection. The verify run reported that mutant caught and the unmutated baseline passed, and the test was added as a pending review comment.

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
  • Documentation Update (if none of the other choices apply)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue


Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
  • I ran just format as mentioned in CODING_GUIDELINES
  • I used just check as mentioned in CODING_GUIDELINES
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Further comments

The change is a single Markdown file that only runs when someone types /mutants-review; it touches no code, build or CI.

Every comment the skill posts is prefixed with [AI], so readers can tell which review comments were generated.

🤖 Generated with Claude Code

@jarhodes314
jarhodes314 marked this pull request as draft October 16, 2026 08:48
@jarhodes314
jarhodes314 force-pushed the jarhodes314/cargo-mutants-review-skill branch from a86104e to 6b7e092 Compare October 16, 2026 08:55
@jarhodes314
jarhodes314 marked this pull request as ready for review October 16, 2026 09:02
@codecov

codecov Bot commented Oct 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@reubenmiller reubenmiller added the theme:developer-exp Theme: improve developer experience label Oct 16, 2026
@jarhodes314 jarhodes314 changed the title chore: add mutants-review Claude Code skill chore: add mutants-review agent skill Oct 16, 2026
@jarhodes314
jarhodes314 force-pushed the jarhodes314/cargo-mutants-review-skill branch from 6b7e092 to 3b3b743 Compare October 16, 2026 09:52
@jarhodes314
jarhodes314 deployed to Test Pull Request October 16, 2026 09:52 — with GitHub Actions Active
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Robot Results

✅ Passed ❌ Failed ⏭️ Skipped Total Pass % ⏱️ Duration
1047 0 8 1047 100 2h47m15.278827s

jarhodes314 and others added 2 commits October 21, 2026 15:29
Runs cargo-mutants on a PR's diff, writes a test for each missed
mutant, verifies with cargo-mutants that the test catches it, and adds
the tests as pending GitHub review comments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jarhodes314
jarhodes314 force-pushed the jarhodes314/cargo-mutants-review-skill branch from ed9625f to fede877 Compare October 21, 2026 14:50

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI] question: Why not keep the skill in .agents/skills/ with a wrapper here, like pr-review?


from me: The PR description says:

Adds /mutants-review [PR], a user-invoked agent skill in .agents/skills/

So it seems this file is mislocated?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved back to .agents/skills/mutants-review/SKILL.md, with a wrapper in .claude/commands/ like pr-review.

Comment thread .claude/commands/mutants-review.md Outdated

Show the user a summary table (mutant, verdict, verified?) and the comment bodies. Under it, name the changed logic no mutant touched: cargo-mutants leaves macro arguments (e.g. the arms of a `matches!`) and most `if` conditions unmutated, so "all caught" says nothing about them. Posting is outward-facing: post only after the user approves.

Each untested behaviour becomes one comment, anchored on a mutated line (always inside the diff, since `--in-diff` produced it), `side: RIGHT`. Mutants killed by the same test share one comment.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI] issue: A mutant's line in missed.txt can be outside the diff, so posting the review can fail.

--in-diff keeps every mutant whose replaced code overlaps a changed line. But missed.txt gives the line where the replaced code starts. For a "replace function body" mutant, that is the first line of the body. If the PR changed only the end of a long function, that line is outside the diff hunks. GitHub rejects a review comment on such a line, so the REST call that creates the new review fails as a whole.

Suggested change
Each untested behaviour becomes one comment, anchored on a mutated line (always inside the diff, since `--in-diff` produced it), `side: RIGHT`. Mutants killed by the same test share one comment.
Each untested behaviour becomes one comment, anchored on a changed line of `pr.diff` that lies inside the mutated function, `side: RIGHT`. Mutants killed by the same test share one comment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied, thanks. The old wording wrongly claimed the mutated line is always inside the diff.

This branch was successfully deployed

1 active and 1 inactive deployments
Test Auto — c3d50b6c Deployed Oct 27, 2026 by jarhodes314 via Test x86_64 #8792
Test Pull Request — c3d50b6c Deployed Oct 27, 2026 by jarhodes314 via Approve #8792
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

theme:developer-exp Theme: improve developer experience

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants