Skip to content

Add TestFx shipping-path acceptance validation skill - #11822

Merged
Amaury Levé (Evangelink) merged 1 commit into
mainfrom
dev/amauryleve/acceptance-validation-skill
Oct 8, 2026
Merged

Amaury Levé (Evangelink) merged 1 commit into
mainfrom
dev/amauryleve/acceptance-validation-skill

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Adds the repository-owned testfx-acceptance-validation skill and a standard-library Python TRX execution gate, scoped entirely to .github/skills/testfx-acceptance-validation/.

The bounded recipe covers producer-consumer contracts, configuration-matched packed packages and cache/binary provenance, relevant execution/discovery/help/launch-profile paths, exact exits and artifacts, capable CI selection and per-step prerequisites, and ownership-aware cleanup. The gate requires an independent exact-name/multiplicity plan, fresh results, consistent counters, unique execution IDs, a successful outer exit, and every required case passing. It rejects skipped-only evidence, including 29 NotExecuted cases with exit zero.

Motivated by acceptance feedback in #11614, #11805, and #11804. This change does not modify those product PRs, pipelines, root instructions, or synced skills.

Validation

  • python -m unittest discover -s .github\skills\testfx-acceptance-validation\scripts -p "test_*.py" — passed all 16 tests, including CLI success/failure/missing-input exits, zero execution, stale reports, exact identities, and skipped-only results. Independently repeated by the coordinator.
  • python .github\skills\testfx-acceptance-validation\scripts\verify_execution.py --help — passed; documented gate options available.
  • git --no-pager diff --check — passed.
  • Both PowerShell examples parsed; actual single-case JSON serialization, 23 repository source paths, named regression/reporting properties, frontmatter, and whitespace checked successfully.

No real product build, packed-consumer execution, or CI run was performed or claimed for this skill-only change. The self-tests validate the evidence gate, not composed-product behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: fb237429-a891-4e0e-9145-7b4ec24db0aa
Copilot AI balanced review requested due to automatic review settings October 8, 2026 11:22
@github-actions github-actions Bot added the state/needs-review Awaiting review from the team. label Oct 8, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Review outcome: No actionable findings — this is a well-constructed, self-contained addition scoped entirely to a new .github/skills/testfx-acceptance-validation/ directory with no production code impact.

Change classification

  • Documentation/tooling: New repository-owned agent skill (SKILL.md) documenting an acceptance-validation methodology (packed consumers, package provenance, TRX execution proof).
  • Standard-library Python tool: verify_execution.py — a TRX-parsing gate that rejects skipped-only/stale/mismatched execution evidence, plus its unittest suite (test_verify_execution.py, 16 tests).

Confidence at a glance

🟢 Correctness & design — verified, no issues
  • Why it applies: New Python module with nontrivial validation logic (timestamp freshness, counter reconciliation, execution-ID uniqueness, surrogate-free name matching via Counter).
  • Evidence: Ran python -m unittest discover -s .github/skills/testfx-acceptance-validation/scripts -p "test_*.py" locally — all 16 tests pass, matching the PR's validation claim. Independently traced verify_execution(): exit-code check happens before the counts/report are finalized but after computing the report dict (intentional — the counts string is reused in the exit-code error message); naive-datetime timestamps are correctly rejected (parse_timestamp raises when tzinfo is None); Counter equality correctly catches both missing and unexpected/extra test names including duplicate-multiplicity cases.
  • Gap or disposition: None found. Logic order (process-exit check → identity match → outcome check → counters → run-level RunInfos) is sound and each branch has a corresponding negative test.
🟢 Documentation accuracy — verified, no issues
  • Why it applies: SKILL.md makes many concrete claims about repository paths, environment variables, test class/method names, and CI pipeline structure.
  • Evidence: Spot-checked every cited path/symbol against the current branch: test/Utilities/Microsoft.Testing.TestInfrastructure/{Constants.cs,TestAsset.cs,TestAssetFixtureBase.cs,AcceptanceSourceGen.cs,TestHost.cs}, test/IntegrationTests/.../Helpers/AcceptanceFixture.cs, eng/pipelines/steps/test-windows-app-model.yml, eng/pipelines/test-windows-app-model-preflight.ps1 (-Mode Preflight/LeakCheck, -RemoveStaleTestPackages), eng/pipelines/variables/test-env-vars.yml, and env vars TESTFX_RUN_WINDOWS_APP_MODEL_TESTS, TESTFX_DOTNET_10_PATH, TESTFX_RUN_WINAPP_CLI_INTEROP_TESTS, TESTFX_WINAPP_CLI_PATH, Microsoft_Testing_TestInfrastructure_TempDirectory_Cleanup all exist exactly as described. The example test case WindowsApplicationModelPackageTests.PackedMSTestTestAdapter_ContainsRequiredWindowsApplicationModelAssets exists verbatim. The three cross-referenced PR discussion links (#11614, #11805, #11804) are cited only as motivating context, not instructions to modify those PRs, matching the "examples, not instructions" framing in the skill text.
  • Gap or disposition: None found. No stale/invented references.

⚪ Not applicable: Concurrency & lifecycle, Security & protocol, API & compatibility, Performance, Cross-TFM, Localization, Tests (no .resx, IPC, public API, threading, or test/UnitTests/test/IntegrationTests production test files touched), Analyzers, Build/dependencies & scripts (no .props/.targets/eng/Versions.props/CI YAML/PowerShell files changed — the PR's own two PowerShell example snippets live inside SKILL.md prose, not as executable .ps1 files, so §22 PowerShell Scripting Hygiene routing does not apply), Scope & PR discipline (single, bounded concern; properly cross-referenced).

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 8, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The bounded skill and verifier are consistent with repository behavior and have comprehensive focused tests.

0 open findings

What changed in this PR

Adds a repository-owned skill for validating TestFx shipping-path acceptance evidence and rejecting skipped-only runs.

Changes:

  • Documents packed-consumer, provenance, execution, CI, and cleanup validation.
  • Adds a standard-library Python TRX execution gate.
  • Adds 16 gate tests covering success and failure scenarios.

Confidence at a glance

🟢 High confidence: Reviewed the skill, verifier, tests, referenced paths, and CI behavior.

File Description
SKILL.md Defines the acceptance-validation workflow.
verify_execution.py Validates fresh, complete, passing TRX execution.
test_verify_execution.py Tests gate behavior and CLI exits.

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 8, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. labels Oct 8, 2026
@Evangelink
Amaury Levé (Evangelink) merged commit ab56117 into main Oct 8, 2026
24 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the dev/amauryleve/acceptance-validation-skill branch October 8, 2026 11:38
@github-actions github-actions Bot removed the state/approved Proposal approved; ready for implementation. label Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants