Skip to content

[rel/4.5] Fix MTP MSBuild reporting for controller-backed test runs - #12057

Merged
Amaury Levé (Evangelink) merged 1 commit into
rel/4.5from
dev/amauryleve/4-5-msbuild-reporting-backport
Oct 26, 2026
Merged

Amaury Levé (Evangelink) merged 1 commit into
rel/4.5from
dev/amauryleve/4-5-msbuild-reporting-backport

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Summary

Backport only #12049, merge commit 00c2b84b751f9bf17e4f25366792c6415cb47fce, including its final results-directory repair from 780b5e31fa605ff8d04be0075abc95270d746026.

This services the MTP 2.5 / MSTest 4.5 regression reported in #12032: Test and InvokeTestingPlatform lose module information and summaries when TRX, HangDump, or CrashDump starts an ordinary controller child. Suppress callbacks only for an actual Windows AppContainer, not every controller PID. Also create the first module's results directory inside the existing ownership lock before opening its log; restoring callbacks without that repair can terminate the build before cooperative cancellation completes.

The required module/summary handshake, actual test exits, first-module ownership, cancellation-marker assertion, AppContainer child suppression, IPC format, and native-error visibility remain intact. No public API, dependency, CLI, resource, package-version, SDK-pin, or product-TFM change is included.

The release lacks the upstream hook fixture and task execution simulation helpers. Adapt only those related regressions, using the real filesystem for missing/existing directory and first-module ownership checks. The 16-row package-consuming controller test is unchanged from the source PR. Preserve the release's existing browser/OS guards rather than importing main-only WebAssembly changes. No unrelated implementation or new release design document is included; the original investigation and alternatives provide the rationale.

Release provenance and Test-value evidence

Tested/pushed head: f47969ee3ffb2f5f90d604bc4e550ecfc7aa6cc6.

Final release base: bdb22d719848d19bba86a6e3dc2e63c98e7c03aa. Rebased and repacked after the intervening release product backports; the PR diff remains the five files from #12049 only.

Windows x64; release-pinned SDK 12.0.100-alpha.1.26480.103; Release runners/packages; net8.0/net9.0 focused units; net10.0 ordinary generated consumers; net8.0-windows10.0.19041.0 real WinUI consumer. Packages are MSTest.Sdk/TestAdapter/TestFramework 4.5.2-dev and MTP/MSBuild/TRX/HangDump/CrashDump/PackagedApp 2.5.2-dev. These are fresh release-branch results, not the source PR's main-branch evidence.

Contract / retained test Counterfactual result Final release result
InvokeTestingPlatform_Target_With_AutoRegistered_Extensions_Should_Report_Test_Results: both entry points, no option/TRX/HangDump/CrashDump, passing/failing tests Actual release production: 12 intended controller failures, 4 no-controller passes, outer exit 2 Same 16 cases: 16 passed, exit 0. Inner MSBuild exits 0/1, exact counters, failing test identity, and single-TRX name/outcome are asserted.
Execute_ModuleInfoCreatesResultsDirectoryBeforeOpeningLog 2 missing-directory failures with DirectoryNotFoundException; 2 existing-directory controls passed across net8/net9; exit 2 Same 4 passed, included in the focused units below.
AddExtensions_ControllerPidOptionAtAnyPosition_RegistersFullMSBuildIntegration All 6 PID-position/TFM cases fail at missing lifecycle registration; exit 2 Same 6 passed, included below.
Related registration, lifecycle, generated integration, task execution, and first-module ownership units New regressions retained; no assertion changes between variants 198 passed, 99 per release unit TFM; exit 0.
ExplicitConsumer_ControllerTimeoutCancelsHostWithoutFalseSuccess Also tested on the final release layout with only the CreateDirectory line removed: 1 executed failure, DirectoryNotFoundException and missing cancellation marker; exit 2 Restore exact final source, repack, and repeat unchanged selector/assertion: 1 passed, exit 0. The nonzero inner exit and marker value True remain mandatory. Every restored shipping DLL matches the previously tested final package.
Shipping sidecar cancellation, custom executable, suppressed launcher, missing payload, incompatible platform, empty selection, and mixed/multi-module exits Existing release contract controls, not mock substitutes 14 passed, exit 0; includes the cancellation case above in the main batch.
PackagedAppContainerWinUI_RunsThroughMtpControllerAndPublishesTrx No local baseline AppContainer pair is claimed 1 real acceptance test passed, exit 0; five inner WinUI tests passed, including actual AppContainer token isolation, package identity, and UI dispatcher execution. The deployed runtime DLL matches the final packed net8.0 DLL.

Main final batch: 229 executed checks, zero failures/skips, plus the separate passing cancellation rerun after the one-line counterfactual. Independent filtered discovery matched every native TRX identity; SDK-merged reports were reconciled without double-counting. Existing method-level scheduling remains enabled; every matrix row owns a fresh mutable asset/results path.

Baseline packages were built from a223d46d5784d315e7bae2a04d85185b191dd2b3 with the new tests retained. The red runs' checkout had advanced to 851ffb1578b10fb7c79e6e96245a662481a43e56 through three unrelated test-only file updates; the tested production and selected test sources were unchanged. The final-layout cancellation counterfactual above avoids relying on that older composition for the directory-repair proof.

All 32 baseline/fixed ordinary consumers matched their variant's seven key shipping DLLs and aligned package versions. Effective consumer caches differed between variants. Every final package's repository commit was verified against the tested head. Build/restore/launch failures, zero-test discovery setup errors, and skips are not counted as red-phase proof.

Exact command arguments and artifacts

All environment changes were process-local. Build workers were bounded with /m:2; test processes used DOTNET_PROCESSOR_COUNT=2 and at most two parallel test modules. The short PATH avoids this host's inherited cmd.exe PATH limitation. Windows PowerShell receives a single-quoted command so -nodeReuse:$false binds as a Boolean, and its native exit is propagated explicitly.

$env:Path = "$PWD\.dotnet;$env:SystemRoot\System32\WindowsPowerShell\v1.0;$env:SystemRoot\System32;$env:SystemRoot;C:\Program Files\Git\cmd;C:\Program Files\PowerShell\7"
$env:NUGET_PACKAGES = "$PWD\artifacts\backport-12049\build-packages"
$env:NUGET_HTTP_CACHE_PATH = "$PWD\artifacts\backport-12049\http-cache"
$env:NUGET_PLUGINS_CACHE_PATH = "$PWD\artifacts\backport-12049\plugin-cache"
$env:MSBUILDUSESERVER = '0'
$env:DOTNET_MULTILEVEL_LOOKUP = '0'
$env:DOTNET_PROCESSOR_COUNT = '2'

& "$env:SystemRoot\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -ExecutionPolicy Bypass -Command '& .\eng\build.ps1 -restore -build -pack -c Release -bl -projects NonWindowsTests.slnf -nodeReuse:$false /m:2 /bl:artifacts\backport-12049\fixed-pack-2.binlog; exit $LASTEXITCODE'

Result: exit 0, zero warnings/errors. The baseline pack used the same build arguments with baseline-pack-1.binlog, also exit 0, zero warnings/errors. The one-line cancellation counterfactual and restored package each used the same command with -projects src\Platform\Microsoft.Testing.Platform.MSBuild\Microsoft.Testing.Platform.MSBuild.csproj and distinct final-no-directory-pack.binlog / final-restored-pack.binlog; both builds exited 0, zero warnings/errors.

The following expresses the recorded native discovery/execution argument vectors without repeating the common arguments. $project and $results were absolute paths. Each call used a distinct results directory; reporting defaults were cleared only for discovery, not execution.

$root = $PWD.Path
$unit = Join-Path $root 'test\UnitTests\Microsoft.Testing.Platform.MSBuild.UnitTests\Microsoft.Testing.Platform.MSBuild.UnitTests.csproj'
$mtp = Join-Path $root 'test\IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests.csproj'
$mstest = Join-Path $root 'test\IntegrationTests\MSTest.Acceptance.IntegrationTests\MSTest.Acceptance.IntegrationTests.csproj'
$matrix = 'FullyQualifiedName~InvokeTestingPlatform_Target_With_AutoRegistered_Extensions_Should_Report_Test_Results'
$related = 'FullyQualifiedName~TestingPlatformBuilderHookTests|FullyQualifiedName~MSBuildLifecycleCallbacksTests|FullyQualifiedName~MSBuildTests|FullyQualifiedName~InvokeTestingPlatformTaskTests'
$cancel = 'FullyQualifiedName~ExplicitConsumer_ControllerTimeoutCancelsHostWithoutFalseSuccess'
$controls = "$cancel|FullyQualifiedName~ExplicitConsumer_MultiTargetRunsAggregateExitCodes|FullyQualifiedName~ExplicitConsumer_ControllerRunsShippingHostAndReconcilesResults|FullyQualifiedName~ExplicitConsumer_SuppressedLauncherCannotReturnSuccess|FullyQualifiedName~ExplicitConsumer_CustomExecutableRunsWithoutControllerEnvironment|FullyQualifiedName~ExplicitConsumer_IncompatiblePlatformCannotSilentlyRun|FullyQualifiedName~ExplicitConsumer_MissingPayloadFailsWithoutFallback"

function Invoke-RecordedSelection($project, $filter, $minimum, $results) {
    $common = @('test', '--project', $project, '-c', 'Release', '--no-build', '--no-launch-profile',
        '--filter', $filter, '--max-parallel-test-modules', '2',
        '-p:UsingDotNetTest=true', '-p:TestRunnerAdditionalArguments=', "-p:ArtifactsTestResultsDir=$results")
    & .\.dotnet\dotnet.exe @common '-p:TestingPlatformCommandLineArguments=' --list-tests text "-bl:$results\discovery.binlog"
    & .\.dotnet\dotnet.exe @common --minimum-expected-tests $minimum --show-test-results all "-bl:$results\execution.binlog"
}

Invoke-RecordedSelection $mtp $matrix 16 "$root\artifacts\backport-12049\final-matrix"
Invoke-RecordedSelection $unit $related 1 "$root\artifacts\backport-12049\final-related"
Invoke-RecordedSelection $mstest $controls 14 "$root\artifacts\backport-12049\final-controls"
$env:TESTFX_RUN_WINDOWS_APP_MODEL_TESTS = '1'
Invoke-RecordedSelection $mtp 'FullyQualifiedName~PackagedAppContainerWinUI_RunsThroughMtpControllerAndPublishesTrx' 1 "$root\artifacts\backport-12049\final-appcontainer"
Invoke-RecordedSelection $mstest $cancel 1 "$root\artifacts\backport-12049\final-restored-cancellation"

Every final discovery/execution above exited 0. The related-unit discovery/identity verifier required exactly 198 executed identities despite the invocation's generic minimum of one. The matrix/boundary/hook red runs used the same vectors and the selectors shown in the evidence table, with minima 16/4/6; their intended failing executions exited 2. The final-layout cancellation red run used $cancel, minimum 1, directory final-no-directory-cancellation, and exited 2; the restored run above exited 0.

Raw build and consumer binlogs, native/merged TRX, independent plans, package hashes, and verification JSON remain under artifacts\backport-12049 and artifacts\tmp\Release\testsuite in the producing session. The real AppContainer report is artifacts\tmp\Release\testsuite\W9fud\PackagedWinUI\TestResults\appcontainer-winui.trx; its five executed identities were read back. A consolidated backport-12049-evidence.json in the session artifacts preserves the full commands, source SHAs, results, and provenance.

Limits: Windows x64 only; Linux/macOS and the reporter's exact SDK 10.0.402/released-package setup were not run locally. No CI success or baseline AppContainer green/green pair is claimed. VS discovery returned no installations, but the real dotnet-based WinUI/AppContainer fixture nevertheless built and ran successfully; it is not an unexecuted/mock-only gap.

Backport 00c2b84 (#12049), including the results-directory initialization from 780b5e3. Restrict callback suppression to actual AppContainer children and preserve the required MSBuild handshake and cooperative cancellation. Adapt only the regression fixtures missing from rel/4.5.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9a0577f6-c83e-4e2d-a0a6-e42063b3602d
@github-actions github-actions Bot added the state/needs-review Awaiting review from the team. label Oct 20, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 20, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. labels Oct 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Final test-triage resolution — Azure Pipelines build 1642575

Result: no actionable test failure (high confidence). The build succeeded, and evidence collection was complete (no fetch or artifact failures).

  • Six failed records appeared in Windows_App_Model_Diagnostics_Attempt1. All are deliberate probe or fixture outcomes from the Windows app-model diagnostics, not product regressions:
    • RetryFailsFirstAttempt (WinUI MTP packaged app) is documented as a deliberate failure on attempt 1.
    • PackagedWinUITestCases.RetryTestMethod_PassesOnSecondAppContainerHost fails the first host so Retry activates a second one.
    • NativeTests.Outcome (4 records) asserts expected host failure.
  • The Linux, macOS and Windows unit-test CTRF reports had no failures, retries or flaky tests.
  • No crash or hang diagnostics, and no timeline test signals.
  • Slow-test candidates: 46, with 0 regressions (the PR history excludes duration trends anyway).
  • History for rel/4.5 found no prior builds with matching records and was not marked incomplete.

No issue is warranted and no follow-up is needed.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · 24.1 AIC · ⌖ 0.514 AIC · ⊞ 9.8K

@Evangelink
Amaury Levé (Evangelink) merged commit d15f90a into rel/4.5 Oct 26, 2026
31 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the dev/amauryleve/4-5-msbuild-reporting-backport branch October 26, 2026 10:49
@github-actions github-actions Bot removed the state/approved Proposal approved; ready for implementation. label Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants