Repository navigation
Add TestFx shipping-path acceptance validation skill - #11822
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fb237429-a891-4e0e-9145-7b4ec24db0aa
There was a problem hiding this comment.
Note
🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.
Review outcome: No actionable findings — this is a well-constructed, self-contained addition scoped entirely to a new .github/skills/testfx-acceptance-validation/ directory with no production code impact.
Change classification
- Documentation/tooling: New repository-owned agent skill (
SKILL.md) documenting an acceptance-validation methodology (packed consumers, package provenance, TRX execution proof). - Standard-library Python tool:
verify_execution.py— a TRX-parsing gate that rejects skipped-only/stale/mismatched execution evidence, plus itsunittestsuite (test_verify_execution.py, 16 tests).
Confidence at a glance
🟢 Correctness & design — verified, no issues
- Why it applies: New Python module with nontrivial validation logic (timestamp freshness, counter reconciliation, execution-ID uniqueness, surrogate-free name matching via
Counter). - Evidence: Ran
python -m unittest discover -s .github/skills/testfx-acceptance-validation/scripts -p "test_*.py"locally — all 16 tests pass, matching the PR's validation claim. Independently tracedverify_execution(): exit-code check happens before the counts/report are finalized but after computing the report dict (intentional — thecountsstring is reused in the exit-code error message); naive-datetime timestamps are correctly rejected (parse_timestampraises whentzinfo is None);Counterequality correctly catches both missing and unexpected/extra test names including duplicate-multiplicity cases. - Gap or disposition: None found. Logic order (process-exit check → identity match → outcome check → counters → run-level RunInfos) is sound and each branch has a corresponding negative test.
🟢 Documentation accuracy — verified, no issues
- Why it applies:
SKILL.mdmakes many concrete claims about repository paths, environment variables, test class/method names, and CI pipeline structure. - Evidence: Spot-checked every cited path/symbol against the current branch:
test/Utilities/Microsoft.Testing.TestInfrastructure/{Constants.cs,TestAsset.cs,TestAssetFixtureBase.cs,AcceptanceSourceGen.cs,TestHost.cs},test/IntegrationTests/.../Helpers/AcceptanceFixture.cs,eng/pipelines/steps/test-windows-app-model.yml,eng/pipelines/test-windows-app-model-preflight.ps1(-Mode Preflight/LeakCheck,-RemoveStaleTestPackages),eng/pipelines/variables/test-env-vars.yml, and env varsTESTFX_RUN_WINDOWS_APP_MODEL_TESTS,TESTFX_DOTNET_10_PATH,TESTFX_RUN_WINAPP_CLI_INTEROP_TESTS,TESTFX_WINAPP_CLI_PATH,Microsoft_Testing_TestInfrastructure_TempDirectory_Cleanupall exist exactly as described. The example test caseWindowsApplicationModelPackageTests.PackedMSTestTestAdapter_ContainsRequiredWindowsApplicationModelAssetsexists verbatim. The three cross-referenced PR discussion links (#11614, #11805, #11804) are cited only as motivating context, not instructions to modify those PRs, matching the "examples, not instructions" framing in the skill text. - Gap or disposition: None found. No stale/invented references.
⚪ Not applicable: Concurrency & lifecycle, Security & protocol, API & compatibility, Performance, Cross-TFM, Localization, Tests (no .resx, IPC, public API, threading, or test/UnitTests/test/IntegrationTests production test files touched), Analyzers, Build/dependencies & scripts (no .props/.targets/eng/Versions.props/CI YAML/PowerShell files changed — the PR's own two PowerShell example snippets live inside SKILL.md prose, not as executable .ps1 files, so §22 PowerShell Scripting Hygiene routing does not apply), Scope & PR discipline (single, bounded concern; properly cross-referenced).
There was a problem hiding this comment.
🟢 Approval recommended
The bounded skill and verifier are consistent with repository behavior and have comprehensive focused tests.
0 open findings
What changed in this PR
Adds a repository-owned skill for validating TestFx shipping-path acceptance evidence and rejecting skipped-only runs.
Changes:
- Documents packed-consumer, provenance, execution, CI, and cleanup validation.
- Adds a standard-library Python TRX execution gate.
- Adds 16 gate tests covering success and failure scenarios.
Confidence at a glance
🟢 High confidence: Reviewed the skill, verifier, tests, referenced paths, and CI behavior.
| File | Description |
|---|---|
SKILL.md |
Defines the acceptance-validation workflow. |
verify_execution.py |
Validates fresh, complete, passing TRX execution. |
test_verify_execution.py |
Tests gate behavior and CLI exits. |
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
Adds the repository-owned
testfx-acceptance-validationskill and a standard-library Python TRX execution gate, scoped entirely to.github/skills/testfx-acceptance-validation/.The bounded recipe covers producer-consumer contracts, configuration-matched packed packages and cache/binary provenance, relevant execution/discovery/help/launch-profile paths, exact exits and artifacts, capable CI selection and per-step prerequisites, and ownership-aware cleanup. The gate requires an independent exact-name/multiplicity plan, fresh results, consistent counters, unique execution IDs, a successful outer exit, and every required case passing. It rejects skipped-only evidence, including 29
NotExecutedcases with exit zero.Motivated by acceptance feedback in #11614, #11805, and #11804. This change does not modify those product PRs, pipelines, root instructions, or synced skills.
Validation
python -m unittest discover -s .github\skills\testfx-acceptance-validation\scripts -p "test_*.py"— passed all 16 tests, including CLI success/failure/missing-input exits, zero execution, stale reports, exact identities, and skipped-only results. Independently repeated by the coordinator.python .github\skills\testfx-acceptance-validation\scripts\verify_execution.py --help— passed; documented gate options available.git --no-pager diff --check— passed.No real product build, packed-consumer execution, or CI run was performed or claimed for this skill-only change. The self-tests validate the evidence gate, not composed-product behavior.