Repository navigation
Fix MTP MSBuild reporting for controller-backed test runs - #12049
Conversation
Limit callback suppression to actual AppContainer children so ordinary TRX, HangDump, and CrashDump controller children report module information and test results to MSBuild. Add package-consuming regression cases for both MSBuild entry points and passing/failing test runs, and update hook registration coverage. Fixes #12032 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a559d41-a07c-4671-9391-a28b09697ffc
There was a problem hiding this comment.
🧪 Test quality review — PR #12049
Compared merge base c83be2a87a95cdd8db8ce526d1827c5d480875f7 with head
a46ec0d7576422c3bf0969490924b397fa4b101f. Evidence applies to this head, not
every intermediate commit.
Test-value evidence
This is a bug fix: AddExtensions previously suppressed MSBuild registration for any controller-backed child (matched only on --internal-testhostcontroller-pid), losing module-info/test-summary IPC for TRX/HangDump/CrashDump-triggered controllers. The fix narrows suppression to actual Windows AppContainer children via a new IsAppContainer() P/Invoke check.
| Contract / test | Counterfactual | Result |
|---|---|---|
AddExtensions on a controller PID in a non-sandboxed (ordinary CI) process → full MSBuild registration: AddExtensions_ControllerPidOptionAtAnyPosition_RegistersFullMSBuildIntegration |
Pre-fix behavior (if (_.Contains(...)) with no IsAppContainer() guard) would register only MSBuildCommandLineProvider, failing AssertFullRegistration's checks for host/orchestrator lifetimes and the data consumer. |
Static prediction: detected. The renamed test's assertion set (AssertFullRegistration) is the exact contract that the old code violated; on a non-AppContainer runner (the repo's Linux/macOS/Windows-non-sandboxed CI), this test fails against the pre-fix hook and passes against the fix. Not independently executed by this review. |
MSBuild-node controller end-to-end handshake/summary reporting across Test target, InvokeTestingPlatform target, TRX/HangDump/CrashDump options, and pass/fail outcomes: InvokeTestingPlatform_Target_With_AutoRegistered_Extensions_Should_Report_Test_Results |
Pre-fix hook (same new test, same packed consumer asset) | Static prediction: detected. The test asserts the exact pre-fix defect strings ("MSBuild extension failed to receive test application information" / "...test run summary information") are absent, plus exit code, summary-line, failure-identity, and TRX-outcome content — all of which the PR description reports as failing (12/16 cases) before the fix and passing after. This review did not execute the suite; it relies on the PR's reported red/green evidence, which is methodologically sound but unverified here. |
IsAppContainer() false-path / pre-Windows-8 / non-Windows guard |
No test exercises this branch directly | Not applicable / unverified. No unit test targets IsAppContainer() itself (it is private static and platform-gated); coverage is indirect via the two tests above, which only exercise the "not an AppContainer" branch on non-Windows-sandboxed CI. The genuine-AppContainer branch (isAppContainer != 0) and the GetTokenInformation failure/Win32Exception path are untested in this repository and were only validated manually per the PR description (WinUI consumer run, not part of this diff). |
Only two distinct contracts are introduced/changed by source; both are covered by the two tests above. The IsAppContainer() true-branch and native-call-failure branch remain an acknowledged gap (manual-only verification, per the PR's own validation section) — flagged as an evidence gap, not an inline defect, since testing real AppContainer token state from this repo's CI is impractical.
🧵 Parallel safety
STATUS: CLEAN (delegated to the parallel-safety specialist)
| Test assembly | Scope | Workers |
|---|---|---|
| Microsoft.Testing.Platform.Acceptance.IntegrationTests | MethodLevel (Workers = 0) |
CPU count |
| Microsoft.Testing.Platform.MSBuild.UnitTests | MethodLevel (Workers = 0) |
CPU count |
No cross-test hazards found. The new data-driven acceptance test calls TestAsset.GenerateAssetAsync per data-row instance, which roots each asset under a fresh randomly-named temp subdirectory, so the derived results directory is unique per instance; useMultithreadedMSBuild: false matches every sibling test in the class. RegistrationRecorder in the unit tests is a fresh per-test field instance (never static), so no mutable state crosses test methods. No [DoNotParallelize]/[ResourceLock] was needed, added, or removed.
Per-test assessment
Pseudo-mutation evidence below consists of static predictions, not executed mutation results.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | A (90–100) | new MSBuildTests_Test. |
Asserts absence of both pre-fix defect strings, exit code, pass/fail summary, failure identity, and TRX outcome across 16 packed-consumer data rows. | None |
| Pass | A (90–100) | mod TestingPlatformBuilderHookTests. |
Renamed to match new full-registration contract; AssertFullRegistration verifies provider, host/orchestrator lifetimes, and shared consumer. |
None |
Pseudo-mutation evidence
InvokeTestingPlatform_Target_With_AutoRegistered_Extensions_Should_Report_Test_Results: Mutating the fix back to the unconditionalif (_.Contains(TestHostControllerPidOption, ...))(noIsAppContainer()guard) would route every controller-backed invocation throughMSBuildCommandLineProvideronly, which — per the PR's own reported baseline run — reproduces the exact"MSBuild extension failed to receive test application information"output the test asserts is absent. Likely killed (inferred); not independently executed by this review.AddExtensions_ControllerPidOptionAtAnyPosition_RegistersFullMSBuildIntegration: Same mutation (dropping&& IsAppContainer()) would makeAssertFullRegistration'sAssert.HasCount(1, registrations.HostLifetimes)andAssert.AreSame(...DataConsumers..., ...SessionLifetimes...)checks fail, since only the command-line provider would be registered. Likely killed (inferred); not independently executed.IsAppContainer()true-branch (isAppContainer != 0) and theGetTokenInformationfailure →Win32Exceptionpath: N/A / unverified — no test in this diff exercises these branches from the repository; the PR description reports only a separate manual WinUI-consumer run outside this test suite.
This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety
specialist with /parallel-audit.
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · 214.1 AIC · ⌖ 9.79 AIC · ⊞ 22.3K · ◷
There was a problem hiding this comment.
🔵 Needs a closer look
The native sandbox-routing boundary requires human confirmation of runtime evidence and cross-platform validation.
No open issues 🎉
What changed in this PR
Fixes #12032 by restoring MTP’s MSBuild reporting for ordinary controller-backed runs while preserving AppContainer suppression.
Changes:
- Restricts suppression to Windows AppContainer children using process-token detection.
- Adds 16 packed acceptance cases covering both MSBuild entry points, controller options, and passing/failing tests.
- Updates unit expectations for ordinary controller children.
Test-value evidence: The PR reports 12 regression failures before the fix and 16 passing acceptance cases afterward (c83be2a → a46ec0d), plus AppContainer preservation checks. Execution artifacts were not independently verified.
🟡 Confidence: registration and regression coverage
Human confirmation remains needed for native AppContainer behavior and cross-platform validation.| File | Description |
|---|---|
| test/UnitTests/Microsoft.Testing.Platform.MSBuild.UnitTests/TestingPlatformBuilderHookTests.cs | Expects full registration for ordinary controller children. |
| test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/MSBuildTests.Test.cs | Adds packed controller-reporting regression cases. |
| src/Platform/Microsoft.Testing.Extensions.MSBuild/TestingPlatformBuilderHook.cs | Limits registration suppression to AppContainer children. |
Review effort: Balanced
This comment has been minimized.
This comment has been minimized.
Restored controller-child registration can report its default results directory before any reporter creates it. Initialize the module log directory in the first-module ownership block so MSBuild does not fail fast before cooperative cancellation completes. Add real-filesystem coverage for missing and existing directories, preserve first-module ownership, and keep the cancellation acceptance assertion unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a559d41-a07c-4671-9391-a28b09697ffc
There was a problem hiding this comment.
🧪 Test quality review — PR #12049
Compared merge base c83be2a87a95cdd8db8ce526d1827c5d480875f7 with head
780b5e31fa605ff8d04be0075abc95270d746026. Evidence applies to this head, not
every intermediate commit.
Test-value evidence
This diff contains three distinct behavioral contracts: the Windows-AppContainer-gated MSBuild suppression fix (also present at a prior reviewed commit a46ec0d) and a new follow-up fix — create the first module's results directory before opening its log — added between a46ec0d and 780b5e3.
| Contract / test | Counterfactual | Result |
|---|---|---|
AddExtensions on a controller PID in a non-sandboxed process → full MSBuild registration: AddExtensions_ControllerPidOptionAtAnyPosition_RegistersFullMSBuildIntegration |
Pre-fix if (_.Contains(...)) with no IsAppContainer() guard would register only MSBuildCommandLineProvider, failing AssertFullRegistration's host/orchestrator/consumer checks. |
Static prediction: detected. On the repo's non-AppContainer CI, this test fails against the pre-fix hook and passes against the fix. Not independently executed by this review. |
MSBuild-node controller end-to-end handshake/summary reporting across both entry points, TRX/HangDump/CrashDump options, and pass/fail outcomes: InvokeTestingPlatform_Target_With_AutoRegistered_Extensions_Should_Report_Test_Results |
Pre-fix hook, same packed-consumer assertions | Static prediction: detected, consistent with the PR's own reported 12-failed/16-passed before/after acceptance run. Not independently executed here. |
First module's results directory is created before the log stream opens: Execute_ModuleInfoCreatesResultsDirectoryBeforeOpeningLog (new) and the strengthened directory assertions in Execute_FirstModuleOwnsOutputFileAndCompleteHandshakeAllowsSuccess / Execute_FailedToolPersistsOutputInDefaultOrModuleOwnedLog |
Without _fileSystem.CreateDirectory(_moduleInfo.TestResultFolder); in HandleRequestAsync, a module announcing a not-yet-existing nested results directory (real Windows Release builds per the PR's own CI follow-up evidence, Build 1642134) throws DirectoryNotFoundException when StreamWriter/CreateNew tries to open the log file there. |
Static prediction: detected, corroborated by the author's own reported CI red/green (DirectoryNotFoundException before, pass after) — an author claim, not independently executed by this review. The new test uses a real FileSystem (not the no-op-recording StubFileSystem), which is the right choice: a stub's CreateDirectory wouldn't have reproduced the original bug. |
IsAppContainer() true-branch (isAppContainer != 0) and the GetTokenInformation failure → Win32Exception path |
No test in this diff exercises these branches | Not applicable / unverified (unchanged from prior review) — coverage is only indirect via the "not an AppContainer" branch on non-sandboxed CI; the true-AppContainer path was validated only by the author's separate manual WinUI-consumer run, outside this test suite. |
All three source-level contracts changed since the merge base are covered by at least one test in this diff; the acknowledged gap (genuine-AppContainer / native-call-failure branches) remains unchanged from the prior review and is impractical to cover from this repository's CI.
🧵 Parallel safety
STATUS: CLEAN (delegated to the parallel-safety specialist)
| Test assembly | Scope | Workers |
|---|---|---|
| Microsoft.Testing.Platform.Acceptance.IntegrationTests | MethodLevel ([assembly: Parallelize]) |
0 (CPU count) |
| Microsoft.Testing.Platform.MSBuild.UnitTests | MethodLevel ([assembly: Parallelize]) |
0 (CPU count) |
No cross-test hazards found. The new acceptance test's TestAsset.GenerateAssetAsync call allocates a uniquely-named temp directory per data-row instance, and workingDirectory is passed explicitly to the child dotnet process (no Directory.SetCurrentDirectory). The unit-test fixtures (AppHostTaskFixture, RegistrationRecorder, StubFileSystem, the new real FileSystem instance) are all freshly constructed per test with no static/shared state. No parallelization configuration changed between the merge base and head. No [DoNotParallelize]/[ResourceLock] was needed, added, or removed.
Per-test assessment
Pseudo-mutation evidence below consists of static predictions, not executed mutation results.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | B (80–89) | new InvokeTestingPlatformTaskTests. |
Uses a real FileSystem to prove the directory is created and the log content is exact, across both existing/missing-directory data rows. |
None |
| Pass | A (90–100) | mod InvokeTestingPlatformTaskTests. |
Simplified to one unconditional directory assertion, matching the fix's now-unconditional directory creation. | None |
| Pass | A (90–100) | mod InvokeTestingPlatformTaskTests. |
Added assertion confirms the module's results directory is created exactly once before the log is opened. | None |
| Pass | A (90–100) | new MSBuildTests_Test. |
Asserts absence of both pre-fix defect strings, exit code, pass/fail summary, failure identity, and TRX outcome across 16 packed-consumer data rows. | None |
| Pass | A (90–100) | mod TestingPlatformBuilderHookTests. |
Renamed to match the new full-registration contract; AssertFullRegistration verifies provider, host/orchestrator lifetimes, and shared consumer. |
None |
Pseudo-mutation evidence
Execute_ModuleInfoCreatesResultsDirectoryBeforeOpeningLog: Removing_fileSystem.CreateDirectory(_moduleInfo.TestResultFolder);from the fix would make the real-FileSystem-backedresultsDirectoryExists: falsedata row throwDirectoryNotFoundExceptioninstead of returningtruefromExecute(). Likely killed (inferred); not independently executed by this review.InvokeTestingPlatform_Target_With_AutoRegistered_Extensions_Should_Report_Test_Results: Reverting the fix to the unconditionalif (_.Contains(TestHostControllerPidOption, ...))(noIsAppContainer()guard) would reproduce the"MSBuild extension failed to receive test application information"output the test asserts is absent. Likely killed (inferred); not independently executed.AddExtensions_ControllerPidOptionAtAnyPosition_RegistersFullMSBuildIntegration: The same mutation would makeAssertFullRegistration's host-lifetime-count and shared-consumer checks fail, since only the command-line provider would be registered. Likely killed (inferred); not independently executed.IsAppContainer()true-branch and theGetTokenInformationfailure →Win32Exceptionpath: N/A / unverified — no test in this diff exercises these branches from the repository.
This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety
specialist with /parallel-audit.
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · 274 AIC · ⌖ 14.9 AIC · ⊞ 22.3K · ◷
|
Final test-triage resolution — Azure Pipelines build 1642265 This is the final resolution for the completed Build: 1642265 ( Finding: Not a test-product failure. Every failed leg (Windows Debug, Windows Release, Linux, Linux Release, Windows application-model acceptance, Build UWP and package-contract acceptance tests) failed at compilation with the identical error: No tests executed on any leg ( Next step: Remove or rename the duplicate No issue created — this is a one-off compile defect tied to the current PR revision, not a durable/test-infrastructure signal.
|
🔍 Build Failure AnalysisSummary — The Windows app-model acceptance build fails in Root cause 1: overlapping base-branch helper extractions#12043 added the first named tuple predicate, then #12045 added another predicate with the same name and parameter types. The analyzed merge commit therefore cannot compile and reports Affected file / error
Proposed fix Remove the second duplicate declaration from the base branch, retaining the first implementation already used by - // A tuple that supplies the trailing token is a full-arity row, not one object[] value.
- private static bool IsSingleTupleArgumentWithExplicitCancellationToken(object?[] data, ParameterInfo[] parameters)
- => parameters.Length == 2
- && data.Length == 1
- && TryHandleTupleDataSource(data[0], parameters, out object?[] tupleArguments)
- && IsDataWithExplicitCancellationToken(tupleArguments, parameters);Once Build overview
All actionable MSBuild errors (1)
🤖 Generated by the Build Failure Analysis workflow using (a href="(dev.azure.com/redacted) · analyzed build [1642265]((dev.azure.com/redacted) · PR commit
|
Pull request
Summary
MTP's MSBuild integration incorrectly suppresses registration in every controller child, so
dotnet build -t:TestandInvokeTestingPlatformlose module information and test summaries when TRX, HangDump, or CrashDump starts a controller.Restrict suppression to actual Windows AppContainer children. Ordinary and full-trust children retain their MSBuild callbacks, while sandboxed children still avoid the inaccessible outer build pipe. Add packed MSTest.Sdk regression cases for both entry points, all three options, and passing/failing tests.
Also initialize the first module's result directory before opening its MSBuild log. Restored full-trust callbacks can announce a default results path before a reporter creates it; otherwise the task fails fast before cooperative cancellation completes.
Fixes: #12032
Design and decision record
Investigation, alternatives, and paired-run evidence.
A controller PID does not imply a sandbox, and package identity does not imply AppContainer isolation. Query the actual process token, guarded for Windows 8+, without allocating a native handle. Native errors surface explicitly. Browser/WASI exclusions and the MSBuild task's required-handshake checks remain unchanged. No public API, dependency, CLI option, or IPC format changes are needed.
Demonstration
A text comparison is clearer than screenshots for this IPC regression. The same 16 package-consuming cases produced:
The passing-test regression reports
MSBuild extension failed to receive test application informationbefore the fix. With the fix, passing tests produce the expected summary/TRX and exit 0; intentionally failing tests retain their identity/summary and fail the build with exit 1.Validation and regression proof
Tested on Windows x64 with pinned SDK
12.0.100-alpha.1.26509.101, Release packs/runners, and Debug/net11.0 generated consumers. Packages: MSTest.Sdk/TestAdapter/TestFramework4.6.0-dev; MTP/MSBuild/TRX/HangDump/CrashDump2.6.0-dev.Baseline:
c83be2a87a95cdd8db8ce526d1827c5d480875f7with the new regression tests retained. Fixed source:a46ec0d7576422c3bf0969490924b397fa4b101f. The committed diff matches tested patch SHA-256360D9A0ECA36F48E9670D3CFD1CA7D7E5833E7617D995D86F49DC1E157E47731.Commands from the worktree root (
$resultsis a separate results directory per run):Pack: baseline and fixed passed, zero warnings/errors. Acceptance: 12 failed/4 passed before; 16 passed after, exits 2/0. Units: 568 passed, 142 each for net8.0/net9.0/net10.0/net11.0 targets, exit 0. No skipped cases. Independent discovery plans and fresh TRX reports were reconciled with the repository execution verifier. All 32 generated consumers matched their variant's packed runtime DLL; effective caches differed between variants.
Also exercised a real packaged AppContainer WinUI consumer through the shipping MSBuild/AppModel controller:
The same consumer/driver passed five named package/token/UI tests with each runtime variant, parent exits 0/0. The fixed runtime was taken from the freshly packed net8.0 payload; both DLL hashes were verified. Short NTFS staging avoided unrelated local XAML/AppX path limits. Exact owned registration/staging/cache/alias resources were cleaned up.
The issue evidence artifact contains the detailed assertions, component hashes, and command results; raw binlogs/TRX/provenance are retained in the session. Linux/macOS, the reporter's exact SDK 10.0.402/released 2.5.x setup, and CI were not run locally. A rel/4.5 servicing backport remains separate work.
Test-value evidence
InvokeTestingPlatform_Target_With_AutoRegistered_Extensions_Should_Report_Test_ResultsAddExtensions_ControllerPidOptionAtAnyPosition_RegistersFullMSBuildIntegrationPlainTestMethod_HasExpectedTokenIsolationand four other real WinUI testsCI follow-up:
780b5e31fa605ff8d04be0075abc95270d746026Build 1642134 passed Windows Debug, Linux Release, and real Windows app-model acceptance, but Windows Release exposed
DirectoryNotFoundExceptionin the module log writer. The repair creates the directory inside the existing first-module ownership lock; it does not weaken cancellation, timeout, or handshake checks.Tested this commit against parent
a46ec0d7576422c3bf0969490924b397fa4b101f, retaining identical tests/assertions/selectors. Tested and committed patch SHA-256:500349E9D35EE83B4B45CAE55355D4DE1A7BFFC5F064788C02C7BB3FB29EF692.Execute_ModuleInfoCreatesResultsDirectoryBeforeOpeningLog(missing/existing directory)DirectoryNotFoundException; 4 existing-directory controls pass across net8-net11; exit 2ExplicitConsumer_ControllerTimeoutCancelsHostWithoutFalseSuccessTrueExact paired selectors, with separate results directories for each variant:
Repacked with the same Release command above: zero warnings/errors. Broader validation passed 576 focused units, the original 16 controller regressions, and 12 packed consumer cases covering cancellation, mixed multi-target results, empty selection, suppressed launcher, and custom executable behavior. All executed, with zero skips; fresh TRX reports matched independently discovered plans.
The archived and repaired netstandard MSBuild task payloads have SHA-256
2A02794ECB169FEE916B1983C3FCAE9FEF69FB0035F69035EA89D2C7EF1B8301and96944FD644A208372ACC998EFD0F434233147BB27D78280BDF867097AE018E2C. Local long-path launch errors were excluded from proof; the packed consumer runs used an owned short checkout alias and fresh caches. The fixed package was restored after the counterfactual, and the alias was removed. Results and binlogs remain in the session; this local proof does not claim CI success for the new head.