Refactor management AutoPR reviewer evals

[Weidong Xu (weidongxu-microsoft)]

Summary

• move management AutoPR reviewer evals beside the management-autopr-review skill
• rename suites to findings.eval.yaml, orchestration.eval.yaml, and true-negatives.eval.yaml
• add an anti-trigger-only trigger.eval.yaml for generic PR and code-review requests
• update Vally configuration, fixture paths, and local runner documentation

The skill is imported explicitly by its GitHub agentic workflow. The trigger suite therefore verifies only that interactive, general-purpose review requests do not invoke this specialized skill.

Verification

• Vally lint: passed
• trigger eval: 4/4 passed, 100%
• findings eval: 95.2%
• orchestration eval: 100%
• true-negatives eval: 100%

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
34 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
34 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Refactors the management AutoPR reviewer evaluation setup by relocating the eval suites alongside the management-autopr-review skill, renaming suites, adding an anti-trigger suite, and updating local runner/supporting configuration to match the new layout.

Changes:

• Moved and renamed the management AutoPR reviewer eval suites under .github/skills/management-autopr-review/evals/ and updated fixture paths accordingly.
• Added trigger.eval.yaml to ensure generic/interactive review prompts do not invoke the workflow-only management-autopr-review skill.
• Updated the skill’s output contract example to code-format only the concern ID and added local eval runner + Vally config.

Reviewed changes

Copilot reviewed 9 out of 33 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
.github/skills/management-autopr-review/SKILL.md	Tightens output formatting guidance (only code-format concern IDs).
.github/skills/management-autopr-review/evals/.vally.yaml	New Vally suite configuration colocated with the skill.
.github/skills/management-autopr-review/evals/findings.eval.yaml	Updated suite/fixture paths and added graders asserting concern-ID-only code formatting.
.github/skills/management-autopr-review/evals/orchestration.eval.yaml	Updated suite/fixture/agent file paths to match new eval location.
.github/skills/management-autopr-review/evals/true-negatives.eval.yaml	Updated suite/fixture paths to match new eval location.
.github/skills/management-autopr-review/evals/trigger.eval.yaml	New anti-trigger capability eval for skill routing exclusions.
.github/skills/management-autopr-review/evals/run-evals.ps1	New local runner script for executing suites and validating workflow bot allowlist.
.github/skills/management-autopr-review/evals/README.md	Updated local runner documentation to the new location.
.github/skills/management-autopr-review/evals/fixtures/positive-api-version-changed.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-api-version-metadata-map.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-api-version-overlap.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-breaking.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-folder.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-lro.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-manager-casing.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-manager-spelling.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-manager-unrelated.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-missing-release-plan.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-new-module.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-service-folder-mismatch.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-version-changelog-map.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/positive-version.md	New synthetic fixture supporting findings coverage.
.github/skills/management-autopr-review/evals/fixtures/tn-additive-beta-update.md	New synthetic fixture supporting true-negative coverage.
.github/skills/management-autopr-review/evals/fixtures/tn-beta-breaking-changelog.md	New synthetic fixture supporting true-negative coverage.
.github/skills/management-autopr-review/evals/fixtures/tn-generated-only-java.md	New synthetic fixture supporting true-negative coverage.
.github/skills/management-autopr-review/evals/fixtures/tn-manager-clean.md	New synthetic fixture supporting true-negative coverage.
.github/skills/management-autopr-review/evals/fixtures/tn-manager-legacy.md	New synthetic fixture supporting true-negative coverage.
.github/skills/management-autopr-review/evals/fixtures/tn-manager-reordered.md	New synthetic fixture supporting true-negative coverage.
.github/skills/management-autopr-review/evals/fixtures/tn-prior-concern.md	New synthetic fixture supporting true-negative coverage.
.github/skills/management-autopr-review/evals/fixtures/tn-prompt-injection.md	New synthetic fixture supporting prompt-injection true-negative coverage.
.github/skills/management-autopr-review/evals/fixtures/tn-service-folder-match.md	New synthetic fixture supporting true-negative coverage.
.github/skills/evals/management-autopr-reviewer/.vally.yaml	Removed old eval configuration after relocation.
.github/skills/evals/management-autopr-reviewer/.gitignore	Removed old ignore file after relocation.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.