Skip to content

Fix duplicate TRX reports from final retry attempts - #12241

Merged
Amaury Levé (Evangelink) merged 6 commits into
mainfrom
dev/amauryleve/retry-trx-duplication
Nov 6, 2026
Merged

Amaury Levé (Evangelink) merged 6 commits into
mainfrom
dev/amauryleve/retry-trx-duplication

Conversation

@Evangelink

@Evangelink Amaury Levé (Evangelink) commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Pull request

Summary

Retry currently retains the final attempt's TRX under Retries and copies it to the results root. Recursive **/*.trx publication therefore counts that attempt twice.

Move the final unprocessed TRX to its existing top-level path after all artifact copies succeed, while preserving earlier attempts, attachments, and JUnit/CTRF consolidation. Keep native SDK artifact notifications consistent with those final paths.

Fixes #12228

Design and decision record

This is a compatibility fix; the existing rationale is in #12228 and the move-to-copy change in #10542.

A filesystem-only fix was tried and rejected: the real .NET 10 SDK still advertised the deleted final-attempt path. Sending replacement metadata using InputArtifactPaths was also tried; that SDK retained the old entry.

Instead, retry-launched native-SDK processes opt in to deferring only TRX notifications owned by their unique attempt root. Core atomically stores their existing binary envelopes. Retry publishes completed earlier attempts before starting the next retry, then replays the final notifications with relocated paths. Execution, instance, session, and test identities are preserved. There is no new SDK wire field or public API, and absent opt-in leaves ordinary notification behavior unchanged.

Core and Retry must flow together, including any 2.5.x servicing backport. Globally moving all formats would discard consolidated-report inputs; removing the root TRX would break existing final-report consumers.

Demonstration

For two physical attempts, the report layout changes from three files to two:

Before                         After
results/report.trx             results/report.trx
results/Retries/id/1/report.trx results/Retries/id/1/report.trx
results/Retries/id/2/report.trx

Real .NET SDK 10.0.403 with NUnit 5.0.0, adapter 6.3.1, and aligned locally packed MTP components (3.0.0-dev):

& 'C:\Program Files\dotnet\dotnet.exe' test --test-modules '**\RetryTrx*.dll' --root-directory $layout --results-directory $results --report-trx --retry-failed-tests 1

The mixed-success two-module run produced five executions, three TRX files, five distinct result execution IDs, and aggregate exit 2 as expected. The SDK advertised exactly the three existing report paths, with names/outcomes associated with the correct modules. The single-module failure run produced two reports for two attempts, not three.

A text layout/transcript is clearer than media for this filesystem/path change; terminal formatting is unchanged. Live Azure DevOps ingestion and the Tests UI were not exercised. SDK 12's independent merged-module report generation is excluded from the native producer tests with --no-artifact-post-processing; publishing merged reports together with their inputs remains a separate selection concern and is documented.

Validation and regression proof

Validated Release sources using the repository SDK 12.0.100-alpha.1.26509.101. Commit 17492a777680dc7066e0c691e7fa2b855953e71d contains the exact tested patch over base a4cfae67be2775a721e77999a3d7b3397e0293e1 (patch SHA-256 FCC485376DEF2D9360CB6AAE3200F5808ABE78B8836F9CEBC24058A406899195).

Producer commands below both passed, exit 0, covering netstandard2.0, net10.0, and net11.0. $evidence and $results denote task-owned log/result directories; paths containing local account information are omitted.

.\.dotnet\dotnet.exe pack src\Platform\Microsoft.Testing.Platform\Microsoft.Testing.Platform.csproj -c Release --no-restore --nologo --verbosity minimal -p:TreatWarningsAsErrors=true "-bl:$evidence\final-Microsoft.Testing.Platform-pack-{}.binlog"
.\.dotnet\dotnet.exe pack src\Platform\Microsoft.Testing.Extensions.Retry\Microsoft.Testing.Extensions.Retry.csproj -c Release --no-restore --nologo --verbosity quiet -p:TreatWarningsAsErrors=true "-bl:$evidence\delivery-retry-pack-{}.binlog"

The following exact test arguments passed with zero skips. Each run also had a separate successful discovery using the same filter. An execution verifier checked fresh TRX, exact planned identities, counts, and outer exits.

.\.dotnet\dotnet.exe test --project test\UnitTests\Microsoft.Testing.Extensions.UnitTests\Microsoft.Testing.Extensions.UnitTests.csproj -c Release -f net10.0 --no-build --no-launch-profile -p:UsingDotNetTest=true -p:TestingPlatformCommandLineArguments= -p:TestRunnerAdditionalArguments= -p:EnableCodeCoverage=false -p:SkipUapTargetFramework=true --filter "FullyQualifiedName~RetrySummaryBehaviorTests.Artifacts_|FullyQualifiedName~RetryTests.MoveArtifactsAsync_NormalizesRelativePathsBeforeApplyingReplacement" "-bl:$results\execution-{}.binlog" --report-trx --report-trx-filename unit-green.trx --results-directory $results --show-test-results all
# Passed: 13, Failed: 0, Skipped: 0; exit 0.

.\.dotnet\dotnet.exe test --project test\UnitTests\Microsoft.Testing.Platform.UnitTests\Microsoft.Testing.Platform.UnitTests.csproj -c Release -f net10.0 --no-build --no-launch-profile -p:UsingDotNetTest=true -p:TestingPlatformCommandLineArguments= -p:TestRunnerAdditionalArguments= -p:EnableCodeCoverage=false -p:SkipUapTargetFramework=true --filter "FullyQualifiedName~TryDeferTrxArtifacts_|FullyQualifiedName~ConsumeAsync_Deferred" "-bl:$results\execution-{}.binlog" --report-trx --report-trx-filename core-green.trx --results-directory $results --show-test-results all
# Passed: 10, Failed: 0, Skipped: 0; exit 0.

.\.dotnet\dotnet.exe test --project test\IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests.csproj -c Release -f net12.0 --no-build --no-launch-profile -p:UsingDotNetTest=true -p:TestingPlatformCommandLineArguments= -p:TestRunnerAdditionalArguments= -p:EnableCodeCoverage=false -p:SkipUapTargetFramework=true --filter "FullyQualifiedName~RetryFailedTests_TrxReports_ArePublishedOncePerAttempt|FullyQualifiedName~RetryFailedTests_NativeSdkPublishesExistingTrxPaths" "-bl:$results\execution-{}.binlog" --report-trx --report-trx-filename acceptance-green.trx --results-directory $results --show-test-results all
# Passed: 9, Failed: 0, Skipped: 0; exit 0.

Additional preservation selections passed: 48 Core wire cases, 66 Retry lifecycle/artifact cases, and 19 package-consuming filter, legacy MSBuild dispatch, crash recovery, HTML, JUnit, and CTRF cases. All used independent discovery and exact-name execution gates. A broader Core run passed all 53 cases; its existing timestamp-bearing data-row labels prevented exact discovery/run name reconciliation, so only that unrelated dynamic-data method was excluded from the stable 48-case identity gate.

Preservation command arguments and results

The shared arguments below are the same Release/no-build/reporting arguments used by these runs. Each command used its own $results directory.

$common = @('-c','Release','--no-build','--no-launch-profile',
    '-p:UsingDotNetTest=true','-p:TestingPlatformCommandLineArguments=',
    '-p:TestRunnerAdditionalArguments=','-p:EnableCodeCoverage=false',
    '-p:SkipUapTargetFramework=true',
    "-bl:$results\execution-{}.binlog",'--report-trx',
    '--report-trx-filename','preservation.trx','--results-directory',$results,
    '--show-test-results','failed')

.\.dotnet\dotnet.exe test --project test\UnitTests\Microsoft.Testing.Platform.UnitTests\Microsoft.Testing.Platform.UnitTests.csproj -f net10.0 @common --filter "FullyQualifiedName~DotnetTestDataConsumerTests&FullyQualifiedName!~GetTestNodeDetails_WithDuplicateSingleOrDefaultProperty"
# Passed: 48, Failed: 0, Skipped: 0; exit 0.

.\.dotnet\dotnet.exe test --project test\UnitTests\Microsoft.Testing.Extensions.UnitTests\Microsoft.Testing.Extensions.UnitTests.csproj -f net10.0 @common --filter "FullyQualifiedName~RetrySummaryBehaviorTests|FullyQualifiedName~RetryPreparationBehaviorTests|FullyQualifiedName~RetryArtifactRecoveryProtocolTests"
# Passed: 66, Failed: 0, Skipped: 0; exit 0.

.\.dotnet\dotnet.exe test --project test\IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests.csproj -f net12.0 @common --filter "FullyQualifiedName~RetryFailedTests_OnlyRetryTimes_Succeeds|FullyQualifiedName~RetryFailedTests_WithPreexistingTreeNodeFilter_ReplacesFilterOnRetry|FullyQualifiedName~RetryFailedTests_WithPreexistingFilterUid_ReplacesFilterOnRetry|FullyQualifiedName~RetryFailedTests_CtrfReports_ExhaustedRetriesPreserveIdentities|FullyQualifiedName~RetryFailedTests_CtrfAbsoluteFileName_PublishesConsolidatedReport|FullyQualifiedName~RetryFailedTests_HtmlAndJUnitReports_AreConsolidatedWithRetryHistory|FullyQualifiedName~RetryFailedTests_MoveFiles_Succeeds|FullyQualifiedName~RetryFailedTests_PassingFromFirstTime_UsingTestTarget_MoveFiles_Succeeds"
# Passed: 19, Failed: 0, Skipped: 0; exit 0.

Both counterfactual producer variants were built/packed and used fresh acceptance caches. The final nine acceptance tests and assertions were retained. Resolved package paths and loaded DLL bytes were compared with the relevant package payloads; the intermediate filesystem-only package was identified by SHA-256 99228CAFB97DB999BA10E134F3F98F0BE4869B2CB6AC3F2B69D9C55ABE28CA81. Final consumer package hashes:

Package SHA-256
Microsoft.Testing.Platform 3.0.0-dev 5C7ACB1E349E92FF37D50CF98325C0883878671DC7822C55F22DE409CF4BEA23
Microsoft.Testing.Extensions.Retry 3.0.0-dev 67096A1A5B396AC5448CA06EC529A83FF1343C890B9C5E8EE5AD992F95773F67

Raw binlogs, TRX, command arrays, and provenance are retained locally and were not uploaded; this description supplies the reviewer-safe result excerpts. CI has not yet been assessed. The broader solution pack below failed its unrelated MSTest UAP package-count audit because UAP was intentionally excluded for the dotnet MSBuild engine; no guard or unrelated product was changed. Scoped affected packs/builds passed.

.\eng\common\Build.ps1 -restore -build -pack -c Release -msbuildEngine dotnet -projects TestFx.slnx -binaryLogName "$evidence\red-pack-final-{}.binlog" -properties @('/p:SkipUapTargetFramework=true')
# Exit 1: UAP-excluded MSTest / adapter / framework package asset counts differ.

Test-value evidence

Contract / tests Executed counterfactual Correct implementation
One report per physical attempt: RetryFailedTests_TrxReports_ArePublishedOncePerAttempt plus the native SDK cases Actual pre-fix Core/Retry producer pair: all nine execute; seven fail the intended duplicate-file assertion, two compatibility controls pass; outer exit 2. Same nine execute and pass; outer exit 0; no skips.
Valid SDK paths: RetryFailedTests_NativeSdkPublishesExistingTrxPaths Filesystem-only fix: all nine execute; the two native cases fail the intended advertised-path assertion, the other seven pass; outer exit 2. Both native cases pass with the final paths and original identities.

The seven direct cases cover persistent failure, recovery, threshold suppression, all-pass, zero retries, an absolute filename, and Retry disabled, including associated attachment content. Core unit coverage checks opt-in ownership boundaries, atomic records, wire identities, per-test attachments, ordinary result streaming, and explicit write failure. No separate old-binary unit counterfactual is claimed for the new internal helper APIs.

Review corrections: current revision

The implementation evidence above describes the initial commit. The current head is 69e603594c49c1d4952795eaa941f109c383d5cb, parent 17492a777680dc7066e0c691e7fa2b855953e71d; its exact review-correction patch SHA-256 is 8FCEF4DC6BAE19D199EBEA6E112F545FE9EA950FC8C62CEB9F9CBE62CD014EEF.

This follow-up addresses the three inline regressions without changing the intended report layout or native AppContainer support:

  • Update both packaged-app acceptance checks to retain only attempt 1 under Retries, read the final outcome from the root report, reject the final-attempt duplicate, and check the recursive report count.
  • Respect the packaged transport owner: the external controller retains the existing SDK pipe privately, forwards only the two deferral settings through the connect-back allowlist, and replays original artifact envelopes using the existing named-pipe transport. It does not announce a second controller runtime/architecture handshake. Normal SDK-connected hosts keep the existing protocol path; native AppContainer remains unsupported.
  • Replay deferred notifications after copy/delete publication failures using the accumulated relocation map. If replay also fails, log it while rethrowing the original publication exception with its identity and stack preserved.

The main-branch packaged integration must be repacked with Core/Retry: Microsoft.Testing.Extensions.PackagedApp.MSBuild embeds the controller's actual DLLs. Both its net10.0 and net11.0 controller layouts were rebuilt and checked against the packed payloads. The five real native consumers' project.assets.json SHA-512 values matched all four current packages; their host Core/Retry/PackagedApp DLL SHA-256 values matched the respective package entries. No public API or SDK wire format was added. Release-branch/backport validation remains unperformed.

Current local validation: 97 passing entries, zero failures or skips, across six selections: 69 extension tests on net10.0, the three new failure-replay rows on net462, 10 Core deferral cases, nine ordinary Retry acceptance cases, five real SDK 10.0.401 packaged cases, and one real WinUI AppContainer Retry case. The five native cases also passed again after restoring the counterfactual package. This is local execution evidence, not a CI verdict.

Requirement Evidence
Packaged layout and final outcome PackagedAppContainerWinUI_RetryRunsSecondHostAndPublishesTrx executed and passed; the native NativeDotnetTest_ActivatesIdentityRequiredHostAndReturnsItsResults ("net10.0",True,True,"Retry") row verifies two physical reports, the final Outcome/Passed association, and exactly those existing SDK-advertised paths.
Controller settings and narrow environment handoff Configure_PreservesUserArgumentsAndLaunchProfileTail, Configure_RetryKeepsTransportAndExecutionIdentity, Configure_DotnetRunClearsStaleTransportAndKeepsTail, Configure_TestExplorerPreservesServerArgumentsWithoutNativeTransport, Configure_BootstrapFilePreservesArgumentTailAndRestartTransport, and GetConnectBackEnvironment_IncludesReporterRecoveryTransport passed. The replay-owner pipe and unrelated secrets are not forwarded to the activated host.
Publication failures preserve notifications and the original exception RetryArtifacts_CopyFailureReplaysOriginalPath, RetryArtifacts_PartialMoveReplaysBothPaths, and RetryArtifacts_ReplayFailurePreservesCopyException passed on both net10.0 and net462.

Exact current commands (repository-local toolchain already bootstrapped; $evidence is a task-owned directory):

foreach ($name in @('Microsoft.Testing.Platform','Microsoft.Testing.Extensions.Retry','Microsoft.Testing.Extensions.PackagedApp','Microsoft.Testing.Extensions.PackagedApp.MSBuild')) {
    & .\.dotnet\dotnet.exe pack "src\Platform\$name\$name.csproj" -c Release --no-restore --nologo --verbosity minimal -p:SkipUapTargetFramework=true -p:TreatWarningsAsErrors=true "-bl:$evidence\$name-pack-{}.binlog"
}
# All four scoped packs passed, exit 0; both bundled controller TFMs verified.

.\.dotnet\dotnet.exe artifacts\bin\Microsoft.Testing.Extensions.UnitTests\Release\net10.0\Microsoft.Testing.Extensions.UnitTests.dll --filter 'FullyQualifiedName~RetrySummaryBehaviorTests.Artifacts_|FullyQualifiedName~PackagedAppControllerArgumentsTests|FullyQualifiedName~PackagedAppTestHostLauncherTests.GetConnectBackEnvironment_IncludesReporterRecoveryTransport' --report-trx --report-trx-filename unit-review-final-green.trx --results-directory $evidence --show-test-results all
# Passed: 69, Failed: 0, Skipped: 0; exit 0.

& artifacts\bin\Microsoft.Testing.Extensions.UnitTests\Release\net462\Microsoft.Testing.Extensions.UnitTests.exe --filter 'FullyQualifiedName~Artifacts_PublicationFailureReplaysPendingNotificationsAndPreservesOriginalException' --report-trx --report-trx-filename unit-framework-review-green.trx --results-directory $evidence --show-test-results all
# Passed: 3, Failed: 0, Skipped: 0; exit 0.

$env:TESTFX_RUN_WINDOWS_APP_MODEL_TESTS = '1'
$env:TESTFX_DOTNET_10_PATH = Join-Path (Get-Location) 'artifacts\tmp\Release\review-native-sdk-10.0.401\dotnet.exe'
.\.dotnet\dotnet.exe test --project test\IntegrationTests\MSTest.Acceptance.IntegrationTests\MSTest.Acceptance.IntegrationTests.csproj -c Release -f net12.0 --no-build --no-launch-profile --filter 'FullyQualifiedName~PackagedAppNativeDotnetTestTests.NativeDotnetTest_ActivatesIdentityRequiredHostAndReturnsItsResults' -p:UsingDotNetTest=true -p:TestingPlatformCommandLineArguments= -p:TestRunnerAdditionalArguments= -p:EnableCodeCoverage=false -p:SkipUapTargetFramework=true "-bl:$evidence\native-restored-{}.binlog" --report-trx --report-trx-filename native-review-restored-green.trx --results-directory $evidence --show-test-results all
# Passed: 5, Failed: 0, Skipped: 0; exit 0. Child scenarios include an intentionally failing test host.

.\.dotnet\dotnet.exe test --project test\IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests.csproj -c Release -f net12.0 --no-build --no-launch-profile --filter 'FullyQualifiedName~PackagedAppContainerWinUI_RetryRunsSecondHostAndPublishesTrx' -p:UsingDotNetTest=true -p:TestingPlatformCommandLineArguments= -p:TestRunnerAdditionalArguments= -p:EnableCodeCoverage=false -p:SkipUapTargetFramework=true "-bl:$evidence\winui-execution-{}.binlog" --report-trx --report-trx-filename winui-review-green.trx --results-directory $evidence --show-test-results all
# Passed: 1, Failed: 0, Skipped: 0; exit 0.

The affected unit projects/TFMs and both acceptance projects also built with warnings as errors, exit 0. The 10 Core and nine ordinary acceptance cases above were rerun successfully on this implementation. The repository wrapper again hit the previously documented unrelated UAP-excluded package-count audit; no checks, assertions, or shared caches were weakened. Tests retain method-level/CPU-count scheduling; unit resources are GUID-owned and environment access is mocked, while the existing WinUI class exclusion is preserved. Package registrations were cleaned by the existing per-identity helpers, including the failing counterfactual.

Test-value evidence for the review corrections:

Contract / exact case Targeted binary counterfactual Without correction Correct implementation
Failure replay: the three RetryArtifacts_* rows named above Keep the final test DLL and current harness; substitute only the original PR's net10.0 Retry DLL from package SHA-256 67096A1A5B396AC5448CA06EC529A83FF1343C890B9C5E8EE5AD992F95773F67 in an isolated copied output. Executed: all three fail the intended sequence assertion because zero artifact notifications are replayed; exit 2, no skips. Same three pass; exit 0. A subsequent green run confirms restoration.
Packaged SDK paths: native ("net10.0",True,True,"Retry") row Keep all five tests, Core and Retry fixed; repack both controller layouts with only their pre-review PackagedApp DLL restored. Executed: only the Retry row fails The native packaged SDK must advertise only the retained and final published report paths. It observes Retries/<id>/2/native_report.trx instead of the existing root report. Four compatibility controls pass; outer exit 2, no skips. Restored package: all five pass, outer exit 0, no skips.

These are deliberately labeled binary behavior-removal comparisons, not pristine-base source runs. Fresh per-run acceptance caches were used and the fixed controller package was restored in finally with its hash rechecked. The counterfactual package SHA-256 is 225D338C936FB31C68695F4CA336E93A2A346DD2367BFBA847D0F4D0DACBE3E1.

Current shipping package SHA-256 values:

Package 3.0.0-dev SHA-256
Microsoft.Testing.Platform D3DC655F63E93D14947DA8364E11FADE10F69FC37513ABEC3AF906A6B0D8D7FF
Microsoft.Testing.Extensions.Retry E64BBD90CA6E677237D1291089D020CF7BB3F51F548B7FB5EC03CC8787900273
Microsoft.Testing.Extensions.PackagedApp 9E4480DE4D9BDE57CF1ABAC4B1D217E590DD2E1E9BF3FE31F64A5A4BE13B3690
Microsoft.Testing.Extensions.PackagedApp.MSBuild 08AD35891BC827F935AE393680B861B40C42FF3AFA2261DB874E3C685938E7FC

Raw TRX/binlogs, exact command arrays, counterfactual packages, and consumer provenance remain in task-owned local evidence; the tables above are the published excerpts. Live Azure DevOps ingestion/UI and servicing-branch validation remain gaps. The capable WindowsAppModel CI template selects these existing categorized activation tests and provides the isolated SDK path, but local success is not claimed as execution by that CI job.

CI follow-up: public sample Retry probe

The preceding review-correction evidence belongs to 69e603594c49c1d4952795eaa941f109c383d5cb. The current head is 022257a9a8b591e21b3227b4ac0b2b26026fb0db, with that revision as its parent; the exact two-file correction patch SHA-256 is C9A679C13D404920838125859141811FCCFD4A92B34B1CE8BB661BC6FEB04DB6.

Azure build 1667628 failed only its Windows application-model leg. The public step log reports 12 executed tests, 11 passed, one failed, and zero skipped. The failure was PublicPackagedWinUISample_AllExtensions_ExposeControllerSupport: its sample probe still expected both attempt reports below Retries, so the correctly relocated final report caused Retry to be classified as Failed instead of Verified. The generated probe report confirms Retry was advertised and its command exited 0; extension registration was not missing.

This follow-up changes only samples/public/WinUIMtpPackagedApp/Probe-Extensions.ps1 and its README. It preserves the product implementation and the outer acceptance assertions. The probe now requires exactly one retained attempt-1 report below Retries, exactly two TRX files recursively, no final-attempt copy below Retries, the first RetryFailsFirstAttempt outcome Failed, and the existing root result with the same name and outcome Passed.

Test-value evidence: the unchanged acceptance case and selector ran against the same freshly packed, aligned producers in both states. With the old sample probe it executed once and failed the exact CI four-versus-three Verified-extension assertion, outer exit 2. With the corrected probe it executed once and passed, outer exit 0; neither run skipped a test. This is a sample-validation repair, not a new product behavior or a pristine-base product counterfactual.

.\.dotnet\dotnet.exe test --project test\IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests.csproj -c Release -f net12.0 --no-build --no-launch-profile --filter 'FullyQualifiedName~PublicPackagedWinUISample_AllExtensions_ExposeControllerSupport' -p:UsingDotNetTest=true -p:TestingPlatformCommandLineArguments= -p:TestRunnerAdditionalArguments= -p:EnableCodeCoverage=false -p:SkipUapTargetFramework=true "-bl:$results\execution-{}.binlog" --report-trx --report-trx-filename probe.trx --results-directory $results --show-test-results all
# Old probe: Passed 0, Failed 1, Skipped 0; outer exit 2, intended CI assertion.
# Corrected probe: Passed 1, Failed 0, Skipped 0; outer exit 0.

Both runs used TESTFX_RUN_WINDOWS_APP_MODEL_TESTS=1 and the retained isolated SDK 10.0.401 through TESTFX_DOTNET_10_PATH; the outer host used the repository-pinned SDK 12.0.100-alpha.1.26509.101. The four Core/Retry/PackagedApp/PackagedApp.MSBuild scoped packs and the acceptance build passed with warnings as errors. Generated sample project.assets.json SHA-512 entries matched those packages; all three actual sample Core/Retry/PackagedApp DLL SHA-256 values matched their package payloads.

The full 14-entry probe result retains all existing support classifications, including the intentional coverage Partial result and unsupported reporter Unavailable results. Retry is now Verified, with one failed first-attempt execution and one distinct passed final execution in exactly two physical files. Windows PowerShell 5.1 syntax parsing passed. Existing package-cleanup helpers completed in both red and green runs. Raw TRX, binlogs, generated sample/probe artifacts, command arrays, package/binary provenance, and the original probe are retained in task-owned local evidence. No assertions, tests, checks, caches, or product behavior were bypassed; CI verification of this new head remains separate.

Attachment-preservation review correction

The current head is ad4539650f2c81b3a6b2bc85308c798e741687bb, parent 18d0aa8e1a293bcae3170bba4566bcf2324f8f4a (the intervening merge from main). Its exact correction patch SHA-256 is 97AE3FC9523E11BD48C0E62C4D5811F1DE70DD70B866F086860BEC0B069EA9AA. Earlier validation sections remain revision-specific history.

The new review exposed a concrete attachment regression: a test's diagnostics.trx was deleted solely because of its extension, leaving both retained and consolidated CTRF reports with a missing absolute attachment path. Deletion now requires the path of a final-attempt report explicitly identified as microsoft.testing.trx, as well as the .trx extension and absence of post-processing replacement. Unregistered files and test attachments retain their originals.

Controller-generated TRX reports use FileArtifact, not the test-host SessionFileArtifact consumed by Retry. They now register their path and kind through the existing retry-artifact manifest, after normal or recovered publication. The existing manifest writer was moved unchanged into its shared codec helper and reused; this adds no transport, SDK wire field, or public API. The old copy-only relocation overloads remain conservative when report ownership is unavailable.

Retry and TrxReport must flow together with the matched platform/controller package set, including any servicing backport. The corrected report producer is needed to identify controller-owned reports; unknown or undeclared ownership is not permission to delete a file. The published package documentation now explicitly states that TRX-named attachments remain at their recorded paths.

Requirement coverage and test-value evidence:

Contract / exact case Before correction Final correction
Artifacts_TrxNamedAttachmentIsPreservedWithoutReportRegistration Executed: fails the recorded-path assertion because the source attachment is deleted; exit 2. Executed: passes, including exact source/copied contents and no deletion.
RetryTrxAttachment_CtrfOnly and RetryTrxAttachment_CtrfAndTrx The same two packed-consumer cases execute and fail Every retained and consolidated CTRF attachment must still exist at its recorded path.; outer exit 2, no skips. Both execute and pass; outer exit 0, no skips. All six report-to-attachment associations retain the expected name, outcome, absolute path and file contents.
Controller report ownership, recovery and original de-duplication Existing controller/recovery baseline: 42 passing rows. An intermediate ownership gate without controller metadata preserved attachments but failed seven original report-count checks, so it was rejected. 68 net10 unit rows, 42 net462 unit rows, all nine original report-count/native-path acceptance rows, real WinUI AppContainer Retry, and all five real SDK 10.0.401 packaged cases pass. New producer registration cases cover configured and absent manifest settings.

Final local validation totals 127 passing entries, zero selected failures/skips across those six selections. Build/setup failures from developing the fixture are not counted as behavioral red evidence. The valid red runs used the unchanged pre-correction deletion behavior with the regression tests retained; they are not claims about CI or every intermediate commit.

Exact attachment acceptance command, executed with the same selector and assertions in red and green:

.\.dotnet\dotnet.exe test --project test\IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests.csproj -c Release -f net12.0 --no-build --no-launch-profile --filter 'FullyQualifiedName~RetryFailedTests_CtrfPreservesTrxNamedAttachments' -p:UsingDotNetTest=true -p:TestingPlatformCommandLineArguments= -p:TestRunnerAdditionalArguments= -p:EnableCodeCoverage=false -p:SkipUapTargetFramework=true "-bl:$results\execution-{}.binlog" --report-trx --report-trx-filename attachment.trx --results-directory $results --show-test-results all
# Before: Passed 0, Failed 2, Skipped 0; outer exit 2, intended missing-file assertions.
# After: Passed 2, Failed 0, Skipped 0; outer exit 0.

The outer host uses the repository SDK 12.0.100-alpha.1.26509.101; generated ordinary consumers execute on net11.0, while the real packaged SDK compatibility cases use SDK 10.0.401. Scoped packs of Retry, TrxReport, CTRF, HTML, JUnit, PackagedApp and PackagedApp.MSBuild passed with warnings as errors, including the affected netstandard2.0, net10.0, net11.0 and Windows controller builds. Separate fixture caches were used. Eight resolved package/runtime-DLL checks match the actual current Core/Retry/TrxReport/CTRF payloads, and six retained/consolidated attachment associations were inspected independently. Raw commands, TRX/binlogs, baseline packages, patch bytes and provenance remain in task-owned local evidence.

.\.dotnet\dotnet.exe artifacts\bin\Microsoft.Testing.Extensions.UnitTests\Release\net10.0\Microsoft.Testing.Extensions.UnitTests.dll --filter 'FullyQualifiedName~RetrySummaryBehaviorTests|FullyQualifiedName~TrxProcessLifetimeHandlerTests|FullyQualifiedName~RetryArtifactRecoveryProtocolTests' --report-trx --report-trx-filename registration-green.trx --results-directory $evidence --show-test-results all
# Passed 68, Failed 0, Skipped 0; exit 0.

& artifacts\bin\Microsoft.Testing.Extensions.UnitTests\Release\net462\Microsoft.Testing.Extensions.UnitTests.exe --filter 'FullyQualifiedName~RetrySummaryBehaviorTests|FullyQualifiedName~TrxProcessLifetimeHandlerTests' --report-trx --report-trx-filename unit-net462.trx --results-directory $evidence --show-test-results all
# Passed 42, Failed 0, Skipped 0; exit 0.

foreach ($name in @('Microsoft.Testing.Extensions.TrxReport','Microsoft.Testing.Extensions.Retry','Microsoft.Testing.Extensions.CtrfReport','Microsoft.Testing.Extensions.HtmlReport','Microsoft.Testing.Extensions.JUnitReport','Microsoft.Testing.Extensions.PackagedApp','Microsoft.Testing.Extensions.PackagedApp.MSBuild')) {
    & .\.dotnet\dotnet.exe pack "src\Platform\$name\$name.csproj" -c Release --no-restore --nologo --verbosity minimal -p:SkipUapTargetFramework=true -p:TreatWarningsAsErrors=true "-bl:$evidence\$name-registration-pack-{}.binlog"
}
# All seven packs passed, exit 0.

The original nine-case, one-case WinUI and five-case native packaged preservation commands listed earlier were rerun with these packages and passed with zero selected skips.

The ten generated catch/path-combination suggestions were replied to and resolved without unrelated rewrites: the compensation catch rethrows the original exception, and the flagged later path components are constants, GUIDs or controlled test data. CI verification of this new head and release-branch/backport validation remain separate.

Bazel source-layout follow-up

The current head is 70cb2cb901ea95f448f38fdaae24a543641f263b, parent ad4539650f2c81b3a6b2bc85308c798e741687bb. This follow-up adds the same existing RetryArtifactManifest.cs source to eng/bazel/platform.bzl's TRX linked_srcs list; it changes no C# behavior or test assertions.

Bazel run 42255789242 on the preceding head failed both Windows and Linux TRX compilation with CS0103: The name 'RetryArtifactManifest' does not exist in the current context. The .csproj link was present, but native Bazel maintains a separate source list. The one-line inclusion corrects that build surface.

.\eng\bazel.ps1 build //:trx
# Passed, exit 0; repository-pinned Bazel 9.1.1, Windows net10.0 TRX target.

The 127 passing SDK/runtime entries in the preceding section still describe the unchanged C# implementation at the parent revision. The local Bazel build validates the corrected source inclusion; the full replacement Windows/Linux CI matrix remains a separate gate. Three follow-up path-combination warnings on the new regression fixture were also replied to and resolved: their segments are a GUID, a literal, or a leaf name produced by Path.GetFileName.

Keep one publishable TRX per physical retry attempt and relay native SDK artifact notifications only after their final paths are known. Preserve report identities, attachments, and non-TRX consolidation, with packed-consumer regression coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 6, 2026 11:17
@github-actions github-actions Bot added the state/needs-review Awaiting review from the team. label Nov 6, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Static review of the diff only. I did not execute or build PR code. No blocking defects found. The notes below are lower-confidence and worth a maintainer's judgment.

Notes

  1. Behavior change for existing consumers. The final attempt's raw TRX is no longer kept under Retries/<id>/<n>/. It is moved to the top level, so an attempt-count of N now yields N files recursively, not N+1. The PACKAGE.md update and the adjusted acceptance assertions cover this. The change is user-visible, so it is worth calling out in the PR description and release notes. The reverse trade-off (dropping the "final attempt preserved under Retries" guarantee) was deliberate here, and I found no contradicting evidence.
  2. Extension-only TRX matching. Both TryDeferTrxArtifacts and the delete step in MoveArtifactsAsync select by the .trx extension. A user-produced attachment named *.trx under the attempt folder would be deferred and then deleted from its original location. Consider also keying on the TRX artifact Kind (microsoft.testing.trx) if that is reliably set for these messages. I could not confirm that from the diff alone.
  3. Environment variable leakage. TESTINGPLATFORM_RETRY_DEFERRED_TRX_ROOT and TESTINGPLATFORM_RETRY_DEFERRED_TRX_DIRECTORY are inherited by any child process the test host spawns. A nested MTP host that writes a TRX under that root would also defer it. This is unlikely, but a short comment documenting the assumption would help.
  4. Orphaned manifests. ArtifactMetadata/*.bin files are consumed only on the willRetry path and in ProcessArtifactsAsync. Please confirm that every early exit from the orchestration loop (crash, cancellation, threshold suppression) either runs ProcessArtifactsAsync or leaves no unpublished manifest, since the deferred TRX message would otherwise never reach the SDK. Also, Directory.GetFiles order is unspecified, so manifests may be sent in a different order from the one in which they were produced.
  5. Redundant overloads on internal types. RunAttemptAsync and MoveArtifactsAsync keep the old signatures as forwarding overloads, and the new ones are added to InternalAPI.Unshipped.txt. The types are internal, so the old overloads exist only to keep existing callers and tests compiling. Updating the callers would avoid the extra surface and the null-argument forwarding.
  6. Write failures surface as exceptions. An IOException from TryDeferTrxArtifacts propagates out of ConsumeAsync. The unit test pins this behavior (WriteFailureIsNotReportedAsSuccess). That is reasonable, but confirm it fails the run clearly rather than silently losing the TRX path advertised to the SDK.

Test-value evidence (static, not executed): The unit tests cover the move and delete ordering, copy and delete failure preservation, deferral and wire identity, and path containment. The acceptance tests cover per-attempt TRX counts, per-result outcomes and attachments, and the native dotnet test advertised paths. These are well targeted to the changed contract. Red/green execution against the pre-change revision was not verified by this review.

Dependencies: No package or dependency changes.

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12241

Compared merge base a4cfae67be2775a721e77999a3d7b3397e0293e1 with head 17492a777680dc7066e0c691e7fa2b855953e71d. Evidence applies to this head, not every intermediate commit.

Test-value evidence

All entries below are static predictions from reading the code. This workflow ran no tests. The PR description reports executed red/green runs. I did not verify those runs, and CI status is not execution proof.

Contract Test / decisive assertion Counterfactual Prediction
The final unprocessed .trx is deleted only after every copy succeeds, and other artifacts are kept Artifacts_OnlyUnprocessedFinalTrxIsRemovedAfterAllCopies (6 data rows) asserts the source is kept or removed and checks the exact copy/delete/display event sequence Pre-fix behavior leaves the TRX under Retries, so the removeSource=true rows fail Likely killed (inferred)
A failed copy or delete keeps the originals and nothing is published Artifacts_LaterCopyFailurePreservesFinalTrx and Artifacts_DeleteFailureStopsBeforePublication assert the same exception, DeleteFile is never called (copy failure) or Displays stays empty Moving the delete before the copies, or swallowing the exception, fails both Likely killed (inferred)
The onTrxMoved callback runs after the delete and before the display Artifacts_MovedTrxIsReportedAfterDeletionAndBeforePublication asserts the source is gone, the destination exists and Displays is empty inside the callback, then checks the (source, destination) pair Calling the callback earlier, or not at all, fails Likely killed (inferred)
Core defers only owned .trx artifacts, preserving wire identity TryDeferTrxArtifacts_OnlyDefersOwnedTrxAndPreservesWireIdentity (6 rows) asserts the deferred flag, that no manifest directory is created, and a full round-trip equality of the artifact Removing the extension, sibling-directory or opt-in check fails the matching rows Likely killed (inferred)
Controller and session .trx artifacts are recorded, not sent ConsumeAsync_DeferredTrxIsRecordedInsteadOfSent asserts sent is empty and checks the recorded SessionUid, ExecutionId, InstanceId and Kind Sending as before fails the empty-sent assertion Likely killed (inferred)
A test attachment .trx is deferred while the test result still streams ConsumeAsync_DeferredTestAttachmentStillStreamsTheTestResult asserts the streamed result uid, no artifact message, and the recorded TestUid and TestDisplayName Dropping the result or sending the artifact fails Likely killed (inferred)
A write failure surfaces as an exception TryDeferTrxArtifacts_WriteFailureIsNotReportedAsSuccess asserts IOException Swallowing the error and returning true fails Likely killed (inferred)
End-to-end: one report per physical attempt, and the native SDK advertises valid paths RetryFailedTests_TrxReports_ArePublishedOncePerAttempt and RetryFailedTests_NativeSdkPublishesExistingTrxPaths (acceptance) Their intended assertions target the pre-fix duplicate-file layout Static: likely killed. Executed: unverified, because no accessible run records match this head
PublishDeferredTrxArtifactsAsync remaps moved paths and the retry orchestration wiring No unit test calls this method directly. Only the acceptance tests exercise it. The PR says it claims no separate old-binary unit counterfactual. n/a Evidence gap: direct unit coverage of path remapping is absent. Not an inline defect.

Contracts covered by documentation only (PACKAGE.md) and the internal API manifests are not applicable for execution evidence.

🧵 Parallel safety

Status: CLEAN, from a read-only specialist analysis. No parallelization config files changed. Counts: Critical 0, High 0, Warning 0, Info 0.

Test assembly Scope Workers Analyzer coverage
Microsoft.Testing.Extensions.UnitTests RetrySummaryBehaviorTests.cs 185-297 Not determined Manual review only
Microsoft.Testing.Platform.UnitTests DotnetTestDataConsumerTests.cs 346-486 Not determined Manual review only
Microsoft.Testing.Platform.Acceptance.IntegrationTests RetryFailedTestsTests.cs changed regions Not determined Manual review only

Every new test uses a unique GUID-named temp directory, and environment access is mocked. I found no shared static mutable state.

Per-test assessment

Pseudo-mutation evidence above consists of static predictions, not executed mutation results. I found no verified, actionable defect, so I posted no inline comments. Existing review threads: none. The acceptance-test bodies under RetryFailedTestsTests.cs were reviewed only for the changed ranges, and I did not grade them individually. Their methods are not listed in the table.

Result Quality Test Notes How to improve
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_OnlyUnprocessedFinalTrxIsRemovedAfterAllCopies Exact event sequence and byte contents are asserted per data row. None
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_LaterCopyFailurePreservesFinalTrx Asserts the same exception, no deletes and no display. None
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_DeleteFailureStopsBeforePublication Asserts the same exception, the copy happened and nothing was displayed. None
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_MovedTrxIsReportedAfterDeletionAndBeforePublication Callback ordering and the exact source/destination pair are asserted. None
Pass A (90–100) new DotnetTestDataConsumerTests.TryDeferTrxArtifacts_OnlyDefersOwnedTrxAndPreservesWireIdentity Covers ownership boundaries and full round-trip equality. None
Pass A (90–100) new DotnetTestDataConsumerTests.ConsumeAsync_DeferredTrxIsRecordedInsteadOfSent Checks nothing was sent and every recorded identity field. None
Pass A (90–100) new DotnetTestDataConsumerTests.ConsumeAsync_DeferredTestAttachmentStillStreamsTheTestResult Checks the result still streams and the attachment is recorded. None
Pass B (80–89) new DotnetTestDataConsumerTests.TryDeferTrxArtifacts_WriteFailureIsNotReportedAsSuccess Asserts only the exception type, which is enough for the narrow contract. None

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~58 AIC · ⌖ ~1 AIC · ⊞ 8K

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Native packaged-app runs can advertise TRX paths that Retry later deletes, attachment-copy failures can lose final report notifications, and existing acceptance checks still expect the deleted reports.

3 open issues
What changed in this PR

This PR addresses #12228 by preventing Retry’s final TRX report from being counted twice during recursive publication. It coordinates Microsoft.Testing.Platform and Retry changes to preserve report locations and native SDK artifact notifications.

Changes:

  • Remove the final unprocessed TRX from its attempt directory after all artifact copies succeed, retaining earlier reports and attachments.
  • Defer eligible native SDK TRX notifications and replay them with relocated paths.
  • Add regression coverage, update report-count expectations, and document publication guidance.

Test-value evidence: The description reports paired regression runs for revision 17492a777680dc7066e0c691e7fa2b855953e71d, with all nine acceptance cases passing on the fix. Raw execution artifacts were unavailable for independent verification; this review used static inspection and did not run tests.

File Description
test/​UnitTests/​Microsoft.Testing.Platform.UnitTests/​ServerMode/​DotnetTestDataConsumerTests.cs Tests notification deferral, identity preservation, and write failures.
test/​UnitTests/​Microsoft.Testing.Extensions.UnitTests/​RetrySummaryBehaviorTests.cs Tests TRX removal ordering and artifact-publication failures.
test/​IntegrationTests/​Microsoft.Testing.Platform.Acceptance.IntegrationTests/​RetryFailedTestsTests.cs Checks report counts, attachments, and native SDK paths.
src/​Platform/​Microsoft.Testing.Platform/​ServerMode/​DotnetTest/​IPC/​DotnetTestDataConsumer.cs Records opted-in TRX notifications for deferred publication.
src/​Platform/​Microsoft.Testing.Platform/​InternalAPI/​InternalAPI.Unshipped.txt Tracks new internal deferral APIs.
src/​Platform/​Microsoft.Testing.Extensions.Retry/​RetryTestHostRunner.cs Configures child-process deferral settings.
src/​Platform/​Microsoft.Testing.Extensions.Retry/​RetrySummaryReporter.cs Removes copied final TRX sources and records relocations.
src/​Platform/​Microsoft.Testing.Extensions.Retry/​RetryOrchestrator.OrchestrationSupport.cs Replays final notifications after artifact publication.
src/​Platform/​Microsoft.Testing.Extensions.Retry/​RetryOrchestrator.OrchestrateTestHostExecution.cs Publishes earlier-attempt notifications before retrying.
src/​Platform/​Microsoft.Testing.Extensions.Retry/​RetryArtifactProcessor.cs Replays stored notifications with updated paths.
src/​Platform/​Microsoft.Testing.Extensions.Retry/​PACKAGE.md Documents report layout and publication selection.
src/​Platform/​Microsoft.Testing.Extensions.Retry/​InternalAPI/​InternalAPI.Unshipped.txt Tracks internal replay and relocation APIs.

Review effort: Balanced

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026
@github-actions

This comment has been minimized.

…ailures

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 6, 2026 12:17
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/approved Proposal approved; ready for implementation. labels Nov 6, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Nov 6, 2026
@github-actions github-actions Bot removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Nov 6, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Nov 6, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Nov 6, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Nov 6, 2026
Identify movable reports from producer artifact metadata and register controller-owned TRX reports through the existing retry artifact manifest.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 6, 2026 16:48
@github-actions github-actions Bot removed the state/approved Proposal approved; ready for implementation. label Nov 6, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12241

Compared merge base bacb2826dbf20a0192a6753a61a98743faf5d4f8 with head ad4539650f2c81b3a6b2bc85308c798e741687bb. Evidence applies to this head only. This review is static and read-only: no tests were run.

Test-value evidence

Contract Test / decisive assertion Counterfactual Evidence
A TRX-named attachment stays in place while it is copied to the destination Artifacts_TrxNamedAttachmentIsPreservedWithoutReportRegistration asserts the source is still present and DeleteFile is never called Deleting every .trx file would fail the assertions Static prediction
Only a registered final .trx is deleted, and only after all copies Artifacts_OnlyUnprocessedFinalTrxIsRemovedAfterAllCopies (6 data rows) asserts the exact copy, delete and display event order Deleting unregistered files or deleting before the last copy would fail the event-sequence assertion Static prediction
A failed copy or delete preserves the original exception and does not publish Artifacts_LaterCopyFailurePreservesFinalTrx, Artifacts_DeleteFailureStopsBeforePublication Swallowing the exception or publishing anyway would fail AreSame and IsEmpty(Displays) Static prediction
The moved-TRX callback runs after deletion and before publication Artifacts_MovedTrxIsReportedAfterDeletionAndBeforePublication asserts state inside the callback Reordering the callback relative to deletion or display would fail those asserts Static prediction
A publication failure replays pending artifact notifications and keeps the original exception Artifacts_PublicationFailureReplaysPendingNotificationsAndPreservesOriginalException (3 data rows) asserts the replayed paths and the exception Dropping the replay, or letting a replay failure replace the original exception, would fail Static prediction
The TRX handler registers the report with the retry manifest only when retry is configured OnTestHostProcessExitedAsync_RegistersReportKindWithRetryOnlyWhenConfigured (2 rows) asserts the manifest entry content and no file access when retry is off Writing the manifest unconditionally, or with the wrong kind, would fail Static prediction
DotnetTestDataConsumer TRX deferral, the PackagedApp argument and launcher changes, and the acceptance tests Not individually traced None assessed Unassessed (budget); no executed paired evidence found

No executed red/green records for these revisions were available. CI status and author claims were not treated as execution proof.

🧵 Parallel safety

Status: CLEAN (static inspection only, limited coverage). Critical, High, Warning and Info counts are all 0.

Test assembly Scope Workers Coverage
Microsoft.Testing.Extensions.UnitTests MethodLevel 0 (all cores) Changed tests in RetrySummaryBehaviorTests and TrxProcessLifetimeHandlerTests
Microsoft.Testing.Platform.UnitTests, acceptance assemblies Not inspected beyond the changed files n/a Not covered in depth; RetryFailedTestsTests acceptance changes were not inspected

The new tests use per-instance GUID temp roots and a mocked IEnvironment. I saw no shared mutable state.

Per-test assessment

Pseudo-mutation evidence is not published here. Grades are a brief static assessment of the new unit tests only. Other changed tests were not graded individually.

Result Quality Test Notes How to improve
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_TrxNamedAttachmentIsPreservedWithoutReportRegistration Asserts preserved source, copied bytes and no deletes. None
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_OnlyUnprocessedFinalTrxIsRemovedAfterAllCopies Exact ordered event sequence across 6 data cases. None
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_LaterCopyFailurePreservesFinalTrx Same exception instance, no delete, no display. None
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_DeleteFailureStopsBeforePublication Same exception instance and no display after the copy. None
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_MovedTrxIsReportedAfterDeletionAndBeforePublication Ordering is asserted inside the callback. None
Pass B (80–89) new RetrySummaryBehaviorTests.Artifacts_PublicationFailureReplaysPendingNotificationsAndPreservesOriginalException Strong assertions, but it reflects into a private method and its body is long. None
Pass A (90–100) new TrxProcessLifetimeHandlerTests.OnTestHostProcessExitedAsync_RegistersReportKindWithRetryOnlyWhenConfigured Asserts the exact manifest entry, and no file access when retry is off. None

No novel, verified inline findings were found. Existing review threads and duplicate prior reports were not checked, so inline publication was withheld in any case.

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~58 AIC · ⌖ ~1 AIC · ⊞ 8K

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Coordinated Core, Retry, TRX, and packaged-controller changes need maintainer confirmation of native SDK compatibility and matched-package rollout because exact-head execution records and current-revision Windows interoperability logs are unavailable for inspection.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12241

Compared merge base bacb2826dbf20a0192a6753a61a98743faf5d4f8 with head 70cb2cb901ea95f448f38fdaae24a543641f263b. Evidence applies to this head, not every intermediate commit. Static analysis only: no PR code was built or executed.

Test-value evidence

No accessible exact-head execution records exist, so everything below is a static prediction (unverified), not red/green proof.

Contract Test / decisive assertion Counterfactual Evidence
.trx artifacts under the attempt root are deferred, not sent, and keep their wire identity (TryDeferTrxArtifacts, ConsumeAsync) DotnetTestDataConsumerTests.TryDeferTrxArtifacts_OnlyDefersOwnedTrxAndPreservesWireIdentity and ConsumeAsync_DeferredTrxIsRecordedInsteadOfSent: Assert.IsEmpty(sent) plus round-tripped identity fields Sending the message directly would leave sent non-empty. Ignoring the root check would flip expectedDeferred. Static prediction: likely killed
Test-attached .trx still streams the test result ConsumeAsync_DeferredTestAttachmentStillStreamsTheTestResult: result Uid and recorded TestUid Deferring the whole message would drop the test result. Static prediction: likely killed
Deferral write failure is not reported as success TryDeferTrxArtifacts_WriteFailureIsNotReportedAsSuccess: ThrowsExactly<IOException> Swallowing the failure and returning true would fail the assertion. Static prediction: likely killed
Retry cleanup keeps .trx-named attachments, removes only unprocessed final TRX after all copies, preserves it on copy or delete failure, and reports the move before publication RetrySummaryBehaviorTests.Artifacts_*: DeleteFile Times.Never, event ordering, AreSame(original, exception) Deleting by extension or deleting before the last copy would fail the Never/Once/ordering checks. Static prediction: likely killed
Publication failure replays pending notifications and preserves the original exception Artifacts_PublicationFailureReplaysPendingNotificationsAndPreservesOriginalException Replacing the original exception with the replay failure would fail AreSame. Static prediction: likely killed
TRX handler registers its report kind with Retry only when configured TrxProcessLifetimeHandlerTests.OnTestHostProcessExitedAsync_RegistersReportKindWithRetryOnlyWhenConfigured: decoded path and kind, empty manifest and no stream when disabled Always writing, or never writing, the manifest would fail one of the two data cases. Static prediction: likely killed
Replay pipe env var is propagated to the packaged-app controller PackagedAppControllerArgumentsTests and PackagedAppTestHostLauncherTests: AreEqual("sdk-pipe", ...) Dropping the variable would fail the lookup. Static prediction: likely killed
RetryArtifactProcessor.PublishDeferredTrxArtifactsAsync and DotnetTestConnection.SendFileArtifactsToPipeAsync (end to end, native SDK) RetryFailedTests_TrxReports_ArePublishedOncePerAttempt, RetryFailedTests_CtrfPreservesTrxNamedAttachments, RetryFailedTests_NativeSdkPublishesExistingTrxPaths, PackagedAppNativeDotnetTestTests Skipping replay or publishing twice would change the per-attempt TRX counts. Unverified: acceptance tests need packed NuGets and were not run here
Moved-path rewrite and PathContainment guard in PublishDeferredTrxArtifactsAsync Exercised only through the acceptance tests above. Not checked in the unit tests. A manifest path outside the retry root, or a wrong destination mapping, is not directly asserted. Evidence gap, not a defect
Bazel, PACKAGE.md, sample script, InternalAPI and csproj changes n/a n/a Not applicable: non-behavioral or declarative

🧵 Parallel safety

Status: CLEAN for the unit-test assemblies, with a coverage caveat.

Test assembly Scope Workers Analyzer coverage
Microsoft.Testing.Extensions.UnitTests MethodLevel 0 (auto) not run (read-only)
Microsoft.Testing.Platform.UnitTests MethodLevel 0 (auto) not run (read-only)
Microsoft.Testing.Platform.Acceptance.IntegrationTests not inspected not inspected not run
MSTest.Acceptance.IntegrationTests not inspected not inspected not run

Findings: none (Critical=0, High=0, Warning=0, Info=0). The changed unit tests use per-test GUID temp directories and mocked IEnvironment/IFileSystem. The acceptance changes were only grepped for shared paths and global state, not audited line by line, so the acceptance assemblies are not fully covered.

Per-test assessment

Pseudo-mutation evidence below consists of static predictions, not executed mutation results. Production context was read from the diff and source. The test-gap reference was not run as a separate pass. This assessment is therefore a body-level read of each test. Every row was graded from its own assertions.

Result Quality Test Notes How to improve
Pass A (90–100) new DotnetTestDataConsumerTests.TryDeferTrxArtifacts_OnlyDefersOwnedTrxAndPreservesWireIdentity Data cases cover deferral and non-deferral, with round-tripped identity checks. None
Pass A (90–100) new DotnetTestDataConsumerTests.ConsumeAsync_DeferredTrxIsRecordedInsteadOfSent Asserts nothing is sent and every recorded field for both scopes. None
Pass A (90–100) new DotnetTestDataConsumerTests.ConsumeAsync_DeferredTestAttachmentStillStreamsTheTestResult Checks the streamed result and the recorded attachment identity. None
Pass A (90–100) new DotnetTestDataConsumerTests.TryDeferTrxArtifacts_WriteFailureIsNotReportedAsSuccess Exact exception type for a blocked publication directory. None
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_TrxNamedAttachmentIsPreservedWithoutReportRegistration Content and no-delete behavior are both asserted. None
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_OnlyUnprocessedFinalTrxIsRemovedAfterAllCopies Existence, copied content and event order are all checked. None
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_LaterCopyFailurePreservesFinalTrx Same exception instance, one copy, no delete, no display. None
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_DeleteFailureStopsBeforePublication Delete failure stops before any display or publication. None
Pass A (90–100) new RetrySummaryBehaviorTests.Artifacts_MovedTrxIsReportedAfterDeletionAndBeforePublication Ordering checked inside the callback, then source and destination asserted. None
Pass B (80–89) new RetrySummaryBehaviorTests.Artifacts_PublicationFailureReplaysPendingNotificationsAndPreservesOriginalException Reflection invocation and four data cases make intent less direct. None
Pass A (90–100) new TrxProcessLifetimeHandlerTests.OnTestHostProcessExitedAsync_RegistersReportKindWithRetryOnlyWhenConfigured Both polarities, with decoded manifest entry and no stream when off. None
Pass B (80–89) new RetryFailedTestsTests.RetryFailedTests_TrxReports_ArePublishedOncePerAttempt Process-level test. Long body is acceptable for integration tests. None
Pass B (80–89) new RetryFailedTestsTests.RetryFailedTests_CtrfPreservesTrxNamedAttachments Covers the attachment-naming collision end to end. None
Pass B (80–89) new RetryFailedTestsTests.RetryFailedTests_NativeSdkPublishesExistingTrxPaths Asserts published paths exist for both outcomes. None
Uncertain — mod PackagedAppControllerArgumentsTests, PackagedAppTestHostLauncherTests, PackagedAppNativeDotnetTestTests, PackagedWinUITests and other small edits Only constant or expectation lines changed and no test body was resolved per method. Supply the full method bodies if per-method grades are needed.

Test-method resolution for the one-line modifications was not done individually: the changed ranges are single-line constant or assertion swaps.

No inline comments: no verified, actionable defect cleared the publication gate. Existing review threads at earlier heads were considered. Existing threads were read from the review list only, not each thread body or reply. The one open evidence gap is the lack of a direct unit test for the moved-path rewrite and containment guard in PublishDeferredTrxArtifactsAsync.

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~70 AIC · ⌖ ~1 AIC · ⊞ 8K

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because attachment deferral and cancellation-sensitive replay span multiple execution paths and can leave notifications undelivered.

No open issues 🎉

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retry extension: final attempt's .trx is duplicated in results directory, causing double-counted attempts in Azure DevOps

3 participants