Repository navigation
Fix duplicate TRX reports from final retry attempts - #12241
Conversation
Keep one publishable TRX per physical retry attempt and relay native SDK artifact notifications only after their final paths are known. Preserve report identities, attachments, and non-TRX consolidation, with packed-consumer regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Note
🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.
Static review of the diff only. I did not execute or build PR code. No blocking defects found. The notes below are lower-confidence and worth a maintainer's judgment.
Notes
- Behavior change for existing consumers. The final attempt's raw TRX is no longer kept under
Retries/<id>/<n>/. It is moved to the top level, so an attempt-count ofNnow yieldsNfiles recursively, notN+1. The PACKAGE.md update and the adjusted acceptance assertions cover this. The change is user-visible, so it is worth calling out in the PR description and release notes. The reverse trade-off (dropping the "final attempt preserved under Retries" guarantee) was deliberate here, and I found no contradicting evidence. - Extension-only TRX matching. Both
TryDeferTrxArtifactsand the delete step inMoveArtifactsAsyncselect by the.trxextension. A user-produced attachment named*.trxunder the attempt folder would be deferred and then deleted from its original location. Consider also keying on the TRX artifactKind(microsoft.testing.trx) if that is reliably set for these messages. I could not confirm that from the diff alone. - Environment variable leakage.
TESTINGPLATFORM_RETRY_DEFERRED_TRX_ROOTandTESTINGPLATFORM_RETRY_DEFERRED_TRX_DIRECTORYare inherited by any child process the test host spawns. A nested MTP host that writes a TRX under that root would also defer it. This is unlikely, but a short comment documenting the assumption would help. - Orphaned manifests.
ArtifactMetadata/*.binfiles are consumed only on thewillRetrypath and inProcessArtifactsAsync. Please confirm that every early exit from the orchestration loop (crash, cancellation, threshold suppression) either runsProcessArtifactsAsyncor leaves no unpublished manifest, since the deferred TRX message would otherwise never reach the SDK. Also,Directory.GetFilesorder is unspecified, so manifests may be sent in a different order from the one in which they were produced. - Redundant overloads on internal types.
RunAttemptAsyncandMoveArtifactsAsynckeep the old signatures as forwarding overloads, and the new ones are added toInternalAPI.Unshipped.txt. The types are internal, so the old overloads exist only to keep existing callers and tests compiling. Updating the callers would avoid the extra surface and thenull-argument forwarding. - Write failures surface as exceptions. An
IOExceptionfromTryDeferTrxArtifactspropagates out ofConsumeAsync. The unit test pins this behavior (WriteFailureIsNotReportedAsSuccess). That is reasonable, but confirm it fails the run clearly rather than silently losing the TRX path advertised to the SDK.
Test-value evidence (static, not executed): The unit tests cover the move and delete ordering, copy and delete failure preservation, deferral and wire identity, and path containment. The acceptance tests cover per-attempt TRX counts, per-result outcomes and attachments, and the native dotnet test advertised paths. These are well targeted to the changed contract. Red/green execution against the pre-change revision was not verified by this review.
Dependencies: No package or dependency changes.
There was a problem hiding this comment.
🧪 Test quality review — PR #12241
Compared merge base a4cfae67be2775a721e77999a3d7b3397e0293e1 with head 17492a777680dc7066e0c691e7fa2b855953e71d. Evidence applies to this head, not every intermediate commit.
Test-value evidence
All entries below are static predictions from reading the code. This workflow ran no tests. The PR description reports executed red/green runs. I did not verify those runs, and CI status is not execution proof.
| Contract | Test / decisive assertion | Counterfactual | Prediction |
|---|---|---|---|
The final unprocessed .trx is deleted only after every copy succeeds, and other artifacts are kept |
Artifacts_OnlyUnprocessedFinalTrxIsRemovedAfterAllCopies (6 data rows) asserts the source is kept or removed and checks the exact copy/delete/display event sequence |
Pre-fix behavior leaves the TRX under Retries, so the removeSource=true rows fail |
Likely killed (inferred) |
| A failed copy or delete keeps the originals and nothing is published | Artifacts_LaterCopyFailurePreservesFinalTrx and Artifacts_DeleteFailureStopsBeforePublication assert the same exception, DeleteFile is never called (copy failure) or Displays stays empty |
Moving the delete before the copies, or swallowing the exception, fails both | Likely killed (inferred) |
The onTrxMoved callback runs after the delete and before the display |
Artifacts_MovedTrxIsReportedAfterDeletionAndBeforePublication asserts the source is gone, the destination exists and Displays is empty inside the callback, then checks the (source, destination) pair |
Calling the callback earlier, or not at all, fails | Likely killed (inferred) |
Core defers only owned .trx artifacts, preserving wire identity |
TryDeferTrxArtifacts_OnlyDefersOwnedTrxAndPreservesWireIdentity (6 rows) asserts the deferred flag, that no manifest directory is created, and a full round-trip equality of the artifact |
Removing the extension, sibling-directory or opt-in check fails the matching rows | Likely killed (inferred) |
Controller and session .trx artifacts are recorded, not sent |
ConsumeAsync_DeferredTrxIsRecordedInsteadOfSent asserts sent is empty and checks the recorded SessionUid, ExecutionId, InstanceId and Kind |
Sending as before fails the empty-sent assertion |
Likely killed (inferred) |
A test attachment .trx is deferred while the test result still streams |
ConsumeAsync_DeferredTestAttachmentStillStreamsTheTestResult asserts the streamed result uid, no artifact message, and the recorded TestUid and TestDisplayName |
Dropping the result or sending the artifact fails | Likely killed (inferred) |
| A write failure surfaces as an exception | TryDeferTrxArtifacts_WriteFailureIsNotReportedAsSuccess asserts IOException |
Swallowing the error and returning true fails |
Likely killed (inferred) |
| End-to-end: one report per physical attempt, and the native SDK advertises valid paths | RetryFailedTests_TrxReports_ArePublishedOncePerAttempt and RetryFailedTests_NativeSdkPublishesExistingTrxPaths (acceptance) |
Their intended assertions target the pre-fix duplicate-file layout | Static: likely killed. Executed: unverified, because no accessible run records match this head |
PublishDeferredTrxArtifactsAsync remaps moved paths and the retry orchestration wiring |
No unit test calls this method directly. Only the acceptance tests exercise it. The PR says it claims no separate old-binary unit counterfactual. | n/a | Evidence gap: direct unit coverage of path remapping is absent. Not an inline defect. |
Contracts covered by documentation only (PACKAGE.md) and the internal API manifests are not applicable for execution evidence.
🧵 Parallel safety
Status: CLEAN, from a read-only specialist analysis. No parallelization config files changed. Counts: Critical 0, High 0, Warning 0, Info 0.
| Test assembly | Scope | Workers | Analyzer coverage |
|---|---|---|---|
| Microsoft.Testing.Extensions.UnitTests | RetrySummaryBehaviorTests.cs 185-297 |
Not determined | Manual review only |
| Microsoft.Testing.Platform.UnitTests | DotnetTestDataConsumerTests.cs 346-486 |
Not determined | Manual review only |
| Microsoft.Testing.Platform.Acceptance.IntegrationTests | RetryFailedTestsTests.cs changed regions |
Not determined | Manual review only |
Every new test uses a unique GUID-named temp directory, and environment access is mocked. I found no shared static mutable state.
Per-test assessment
Pseudo-mutation evidence above consists of static predictions, not executed mutation results. I found no verified, actionable defect, so I posted no inline comments. Existing review threads: none. The acceptance-test bodies under RetryFailedTestsTests.cs were reviewed only for the changed ranges, and I did not grade them individually. Their methods are not listed in the table.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_OnlyUnprocessedFinalTrxIsRemovedAfterAllCopies |
Exact event sequence and byte contents are asserted per data row. | None |
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_LaterCopyFailurePreservesFinalTrx |
Asserts the same exception, no deletes and no display. | None |
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_DeleteFailureStopsBeforePublication |
Asserts the same exception, the copy happened and nothing was displayed. | None |
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_MovedTrxIsReportedAfterDeletionAndBeforePublication |
Callback ordering and the exact source/destination pair are asserted. | None |
| Pass | A (90–100) | new DotnetTestDataConsumerTests.TryDeferTrxArtifacts_OnlyDefersOwnedTrxAndPreservesWireIdentity |
Covers ownership boundaries and full round-trip equality. | None |
| Pass | A (90–100) | new DotnetTestDataConsumerTests.ConsumeAsync_DeferredTrxIsRecordedInsteadOfSent |
Checks nothing was sent and every recorded identity field. | None |
| Pass | A (90–100) | new DotnetTestDataConsumerTests.ConsumeAsync_DeferredTestAttachmentStillStreamsTheTestResult |
Checks the result still streams and the attachment is recorded. | None |
| Pass | B (80–89) | new DotnetTestDataConsumerTests.TryDeferTrxArtifacts_WriteFailureIsNotReportedAsSuccess |
Asserts only the exception type, which is enough for the narrow contract. | None |
<sub>This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~58 AIC · ⌖ ~1 AIC · ⊞ 8K
There was a problem hiding this comment.
🟡 Changes recommended
Native packaged-app runs can advertise TRX paths that Retry later deletes, attachment-copy failures can lose final report notifications, and existing acceptance checks still expect the deleted reports.
3 open issues
What changed in this PR
This PR addresses #12228 by preventing Retry’s final TRX report from being counted twice during recursive publication. It coordinates Microsoft.Testing.Platform and Retry changes to preserve report locations and native SDK artifact notifications.
Changes:
- Remove the final unprocessed TRX from its attempt directory after all artifact copies succeed, retaining earlier reports and attachments.
- Defer eligible native SDK TRX notifications and replay them with relocated paths.
- Add regression coverage, update report-count expectations, and document publication guidance.
Test-value evidence: The description reports paired regression runs for revision 17492a777680dc7066e0c691e7fa2b855953e71d, with all nine acceptance cases passing on the fix. Raw execution artifacts were unavailable for independent verification; this review used static inspection and did not run tests.
| File | Description |
|---|---|
test/UnitTests/Microsoft.Testing.Platform.UnitTests/ServerMode/DotnetTestDataConsumerTests.cs |
Tests notification deferral, identity preservation, and write failures. |
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/RetrySummaryBehaviorTests.cs |
Tests TRX removal ordering and artifact-publication failures. |
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/RetryFailedTestsTests.cs |
Checks report counts, attachments, and native SDK paths. |
src/Platform/Microsoft.Testing.Platform/ServerMode/DotnetTest/IPC/DotnetTestDataConsumer.cs |
Records opted-in TRX notifications for deferred publication. |
src/Platform/Microsoft.Testing.Platform/InternalAPI/InternalAPI.Unshipped.txt |
Tracks new internal deferral APIs. |
src/Platform/Microsoft.Testing.Extensions.Retry/RetryTestHostRunner.cs |
Configures child-process deferral settings. |
src/Platform/Microsoft.Testing.Extensions.Retry/RetrySummaryReporter.cs |
Removes copied final TRX sources and records relocations. |
src/Platform/Microsoft.Testing.Extensions.Retry/RetryOrchestrator.OrchestrationSupport.cs |
Replays final notifications after artifact publication. |
src/Platform/Microsoft.Testing.Extensions.Retry/RetryOrchestrator.OrchestrateTestHostExecution.cs |
Publishes earlier-attempt notifications before retrying. |
src/Platform/Microsoft.Testing.Extensions.Retry/RetryArtifactProcessor.cs |
Replays stored notifications with updated paths. |
src/Platform/Microsoft.Testing.Extensions.Retry/PACKAGE.md |
Documents report layout and publication selection. |
src/Platform/Microsoft.Testing.Extensions.Retry/InternalAPI/InternalAPI.Unshipped.txt |
Tracks internal replay and relocation APIs. |
Review effort: Balanced
This comment has been minimized.
This comment has been minimized.
…ailures Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Identify movable reports from producer artifact metadata and register controller-owned TRX reports through the existing retry artifact manifest. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🧪 Test quality review — PR #12241
Compared merge base bacb2826dbf20a0192a6753a61a98743faf5d4f8 with head ad4539650f2c81b3a6b2bc85308c798e741687bb. Evidence applies to this head only. This review is static and read-only: no tests were run.
Test-value evidence
| Contract | Test / decisive assertion | Counterfactual | Evidence |
|---|---|---|---|
| A TRX-named attachment stays in place while it is copied to the destination | Artifacts_TrxNamedAttachmentIsPreservedWithoutReportRegistration asserts the source is still present and DeleteFile is never called |
Deleting every .trx file would fail the assertions |
Static prediction |
Only a registered final .trx is deleted, and only after all copies |
Artifacts_OnlyUnprocessedFinalTrxIsRemovedAfterAllCopies (6 data rows) asserts the exact copy, delete and display event order |
Deleting unregistered files or deleting before the last copy would fail the event-sequence assertion | Static prediction |
| A failed copy or delete preserves the original exception and does not publish | Artifacts_LaterCopyFailurePreservesFinalTrx, Artifacts_DeleteFailureStopsBeforePublication |
Swallowing the exception or publishing anyway would fail AreSame and IsEmpty(Displays) |
Static prediction |
| The moved-TRX callback runs after deletion and before publication | Artifacts_MovedTrxIsReportedAfterDeletionAndBeforePublication asserts state inside the callback |
Reordering the callback relative to deletion or display would fail those asserts | Static prediction |
| A publication failure replays pending artifact notifications and keeps the original exception | Artifacts_PublicationFailureReplaysPendingNotificationsAndPreservesOriginalException (3 data rows) asserts the replayed paths and the exception |
Dropping the replay, or letting a replay failure replace the original exception, would fail | Static prediction |
| The TRX handler registers the report with the retry manifest only when retry is configured | OnTestHostProcessExitedAsync_RegistersReportKindWithRetryOnlyWhenConfigured (2 rows) asserts the manifest entry content and no file access when retry is off |
Writing the manifest unconditionally, or with the wrong kind, would fail | Static prediction |
DotnetTestDataConsumer TRX deferral, the PackagedApp argument and launcher changes, and the acceptance tests |
Not individually traced | None assessed | Unassessed (budget); no executed paired evidence found |
No executed red/green records for these revisions were available. CI status and author claims were not treated as execution proof.
🧵 Parallel safety
Status: CLEAN (static inspection only, limited coverage). Critical, High, Warning and Info counts are all 0.
| Test assembly | Scope | Workers | Coverage |
|---|---|---|---|
| Microsoft.Testing.Extensions.UnitTests | MethodLevel | 0 (all cores) | Changed tests in RetrySummaryBehaviorTests and TrxProcessLifetimeHandlerTests |
| Microsoft.Testing.Platform.UnitTests, acceptance assemblies | Not inspected beyond the changed files | n/a | Not covered in depth; RetryFailedTestsTests acceptance changes were not inspected |
The new tests use per-instance GUID temp roots and a mocked IEnvironment. I saw no shared mutable state.
Per-test assessment
Pseudo-mutation evidence is not published here. Grades are a brief static assessment of the new unit tests only. Other changed tests were not graded individually.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_TrxNamedAttachmentIsPreservedWithoutReportRegistration |
Asserts preserved source, copied bytes and no deletes. | None |
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_OnlyUnprocessedFinalTrxIsRemovedAfterAllCopies |
Exact ordered event sequence across 6 data cases. | None |
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_LaterCopyFailurePreservesFinalTrx |
Same exception instance, no delete, no display. | None |
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_DeleteFailureStopsBeforePublication |
Same exception instance and no display after the copy. | None |
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_MovedTrxIsReportedAfterDeletionAndBeforePublication |
Ordering is asserted inside the callback. | None |
| Pass | B (80–89) | new RetrySummaryBehaviorTests.Artifacts_PublicationFailureReplaysPendingNotificationsAndPreservesOriginalException |
Strong assertions, but it reflects into a private method and its body is long. | None |
| Pass | A (90–100) | new TrxProcessLifetimeHandlerTests.OnTestHostProcessExitedAsync_RegistersReportKindWithRetryOnlyWhenConfigured |
Asserts the exact manifest entry, and no file access when retry is off. | None |
No novel, verified inline findings were found. Existing review threads and duplicate prior reports were not checked, so inline publication was withheld in any case.
<sub>This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~58 AIC · ⌖ ~1 AIC · ⊞ 8K
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Coordinated Core, Retry, TRX, and packaged-controller changes need maintainer confirmation of native SDK compatibility and matched-package rollout because exact-head execution records and current-revision Windows interoperability logs are unavailable for inspection.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
There was a problem hiding this comment.
🧪 Test quality review — PR #12241
Compared merge base bacb2826dbf20a0192a6753a61a98743faf5d4f8 with head 70cb2cb901ea95f448f38fdaae24a543641f263b. Evidence applies to this head, not every intermediate commit. Static analysis only: no PR code was built or executed.
Test-value evidence
No accessible exact-head execution records exist, so everything below is a static prediction (unverified), not red/green proof.
| Contract | Test / decisive assertion | Counterfactual | Evidence |
|---|---|---|---|
.trx artifacts under the attempt root are deferred, not sent, and keep their wire identity (TryDeferTrxArtifacts, ConsumeAsync) |
DotnetTestDataConsumerTests.TryDeferTrxArtifacts_OnlyDefersOwnedTrxAndPreservesWireIdentity and ConsumeAsync_DeferredTrxIsRecordedInsteadOfSent: Assert.IsEmpty(sent) plus round-tripped identity fields |
Sending the message directly would leave sent non-empty. Ignoring the root check would flip expectedDeferred. |
Static prediction: likely killed |
Test-attached .trx still streams the test result |
ConsumeAsync_DeferredTestAttachmentStillStreamsTheTestResult: result Uid and recorded TestUid |
Deferring the whole message would drop the test result. | Static prediction: likely killed |
| Deferral write failure is not reported as success | TryDeferTrxArtifacts_WriteFailureIsNotReportedAsSuccess: ThrowsExactly<IOException> |
Swallowing the failure and returning true would fail the assertion. |
Static prediction: likely killed |
Retry cleanup keeps .trx-named attachments, removes only unprocessed final TRX after all copies, preserves it on copy or delete failure, and reports the move before publication |
RetrySummaryBehaviorTests.Artifacts_*: DeleteFile Times.Never, event ordering, AreSame(original, exception) |
Deleting by extension or deleting before the last copy would fail the Never/Once/ordering checks. | Static prediction: likely killed |
| Publication failure replays pending notifications and preserves the original exception | Artifacts_PublicationFailureReplaysPendingNotificationsAndPreservesOriginalException |
Replacing the original exception with the replay failure would fail AreSame. |
Static prediction: likely killed |
| TRX handler registers its report kind with Retry only when configured | TrxProcessLifetimeHandlerTests.OnTestHostProcessExitedAsync_RegistersReportKindWithRetryOnlyWhenConfigured: decoded path and kind, empty manifest and no stream when disabled |
Always writing, or never writing, the manifest would fail one of the two data cases. | Static prediction: likely killed |
| Replay pipe env var is propagated to the packaged-app controller | PackagedAppControllerArgumentsTests and PackagedAppTestHostLauncherTests: AreEqual("sdk-pipe", ...) |
Dropping the variable would fail the lookup. | Static prediction: likely killed |
RetryArtifactProcessor.PublishDeferredTrxArtifactsAsync and DotnetTestConnection.SendFileArtifactsToPipeAsync (end to end, native SDK) |
RetryFailedTests_TrxReports_ArePublishedOncePerAttempt, RetryFailedTests_CtrfPreservesTrxNamedAttachments, RetryFailedTests_NativeSdkPublishesExistingTrxPaths, PackagedAppNativeDotnetTestTests |
Skipping replay or publishing twice would change the per-attempt TRX counts. | Unverified: acceptance tests need packed NuGets and were not run here |
Moved-path rewrite and PathContainment guard in PublishDeferredTrxArtifactsAsync |
Exercised only through the acceptance tests above. Not checked in the unit tests. | A manifest path outside the retry root, or a wrong destination mapping, is not directly asserted. | Evidence gap, not a defect |
Bazel, PACKAGE.md, sample script, InternalAPI and csproj changes |
n/a | n/a | Not applicable: non-behavioral or declarative |
🧵 Parallel safety
Status: CLEAN for the unit-test assemblies, with a coverage caveat.
| Test assembly | Scope | Workers | Analyzer coverage |
|---|---|---|---|
| Microsoft.Testing.Extensions.UnitTests | MethodLevel | 0 (auto) | not run (read-only) |
| Microsoft.Testing.Platform.UnitTests | MethodLevel | 0 (auto) | not run (read-only) |
| Microsoft.Testing.Platform.Acceptance.IntegrationTests | not inspected | not inspected | not run |
| MSTest.Acceptance.IntegrationTests | not inspected | not inspected | not run |
Findings: none (Critical=0, High=0, Warning=0, Info=0). The changed unit tests use per-test GUID temp directories and mocked IEnvironment/IFileSystem. The acceptance changes were only grepped for shared paths and global state, not audited line by line, so the acceptance assemblies are not fully covered.
Per-test assessment
Pseudo-mutation evidence below consists of static predictions, not executed mutation results. Production context was read from the diff and source. The test-gap reference was not run as a separate pass. This assessment is therefore a body-level read of each test. Every row was graded from its own assertions.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | A (90–100) | new DotnetTestDataConsumerTests.TryDeferTrxArtifacts_OnlyDefersOwnedTrxAndPreservesWireIdentity |
Data cases cover deferral and non-deferral, with round-tripped identity checks. | None |
| Pass | A (90–100) | new DotnetTestDataConsumerTests.ConsumeAsync_DeferredTrxIsRecordedInsteadOfSent |
Asserts nothing is sent and every recorded field for both scopes. | None |
| Pass | A (90–100) | new DotnetTestDataConsumerTests.ConsumeAsync_DeferredTestAttachmentStillStreamsTheTestResult |
Checks the streamed result and the recorded attachment identity. | None |
| Pass | A (90–100) | new DotnetTestDataConsumerTests.TryDeferTrxArtifacts_WriteFailureIsNotReportedAsSuccess |
Exact exception type for a blocked publication directory. | None |
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_TrxNamedAttachmentIsPreservedWithoutReportRegistration |
Content and no-delete behavior are both asserted. | None |
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_OnlyUnprocessedFinalTrxIsRemovedAfterAllCopies |
Existence, copied content and event order are all checked. | None |
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_LaterCopyFailurePreservesFinalTrx |
Same exception instance, one copy, no delete, no display. | None |
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_DeleteFailureStopsBeforePublication |
Delete failure stops before any display or publication. | None |
| Pass | A (90–100) | new RetrySummaryBehaviorTests.Artifacts_MovedTrxIsReportedAfterDeletionAndBeforePublication |
Ordering checked inside the callback, then source and destination asserted. | None |
| Pass | B (80–89) | new RetrySummaryBehaviorTests.Artifacts_PublicationFailureReplaysPendingNotificationsAndPreservesOriginalException |
Reflection invocation and four data cases make intent less direct. | None |
| Pass | A (90–100) | new TrxProcessLifetimeHandlerTests.OnTestHostProcessExitedAsync_RegistersReportKindWithRetryOnlyWhenConfigured |
Both polarities, with decoded manifest entry and no stream when off. | None |
| Pass | B (80–89) | new RetryFailedTestsTests.RetryFailedTests_TrxReports_ArePublishedOncePerAttempt |
Process-level test. Long body is acceptable for integration tests. | None |
| Pass | B (80–89) | new RetryFailedTestsTests.RetryFailedTests_CtrfPreservesTrxNamedAttachments |
Covers the attachment-naming collision end to end. | None |
| Pass | B (80–89) | new RetryFailedTestsTests.RetryFailedTests_NativeSdkPublishesExistingTrxPaths |
Asserts published paths exist for both outcomes. | None |
| Uncertain | — | mod PackagedAppControllerArgumentsTests, PackagedAppTestHostLauncherTests, PackagedAppNativeDotnetTestTests, PackagedWinUITests and other small edits |
Only constant or expectation lines changed and no test body was resolved per method. | Supply the full method bodies if per-method grades are needed. |
Test-method resolution for the one-line modifications was not done individually: the changed ranges are single-line constant or assertion swaps.
No inline comments: no verified, actionable defect cleared the publication gate. Existing review threads at earlier heads were considered. Existing threads were read from the review list only, not each thread body or reply. The one open evidence gap is the lack of a direct unit test for the moved-path rewrite and containment guard in PublishDeferredTrxArtifactsAsync.
<sub>This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~70 AIC · ⌖ ~1 AIC · ⊞ 8K


Pull request
Summary
Retry currently retains the final attempt's TRX under
Retriesand copies it to the results root. Recursive**/*.trxpublication therefore counts that attempt twice.Move the final unprocessed TRX to its existing top-level path after all artifact copies succeed, while preserving earlier attempts, attachments, and JUnit/CTRF consolidation. Keep native SDK artifact notifications consistent with those final paths.
Fixes #12228
Design and decision record
This is a compatibility fix; the existing rationale is in #12228 and the move-to-copy change in #10542.
A filesystem-only fix was tried and rejected: the real .NET 10 SDK still advertised the deleted final-attempt path. Sending replacement metadata using
InputArtifactPathswas also tried; that SDK retained the old entry.Instead, retry-launched native-SDK processes opt in to deferring only TRX notifications owned by their unique attempt root. Core atomically stores their existing binary envelopes. Retry publishes completed earlier attempts before starting the next retry, then replays the final notifications with relocated paths. Execution, instance, session, and test identities are preserved. There is no new SDK wire field or public API, and absent opt-in leaves ordinary notification behavior unchanged.
Core and Retry must flow together, including any 2.5.x servicing backport. Globally moving all formats would discard consolidated-report inputs; removing the root TRX would break existing final-report consumers.
Demonstration
For two physical attempts, the report layout changes from three files to two:
Real .NET SDK 10.0.403 with NUnit 5.0.0, adapter 6.3.1, and aligned locally packed MTP components (
3.0.0-dev):The mixed-success two-module run produced five executions, three TRX files, five distinct result execution IDs, and aggregate exit
2as expected. The SDK advertised exactly the three existing report paths, with names/outcomes associated with the correct modules. The single-module failure run produced two reports for two attempts, not three.A text layout/transcript is clearer than media for this filesystem/path change; terminal formatting is unchanged. Live Azure DevOps ingestion and the Tests UI were not exercised. SDK 12's independent merged-module report generation is excluded from the native producer tests with
--no-artifact-post-processing; publishing merged reports together with their inputs remains a separate selection concern and is documented.Validation and regression proof
Validated Release sources using the repository SDK
12.0.100-alpha.1.26509.101. Commit17492a777680dc7066e0c691e7fa2b855953e71dcontains the exact tested patch over basea4cfae67be2775a721e77999a3d7b3397e0293e1(patch SHA-256FCC485376DEF2D9360CB6AAE3200F5808ABE78B8836F9CEBC24058A406899195).Producer commands below both passed, exit
0, coveringnetstandard2.0,net10.0, andnet11.0.$evidenceand$resultsdenote task-owned log/result directories; paths containing local account information are omitted.The following exact test arguments passed with zero skips. Each run also had a separate successful discovery using the same filter. An execution verifier checked fresh TRX, exact planned identities, counts, and outer exits.
Additional preservation selections passed: 48 Core wire cases, 66 Retry lifecycle/artifact cases, and 19 package-consuming filter, legacy MSBuild dispatch, crash recovery, HTML, JUnit, and CTRF cases. All used independent discovery and exact-name execution gates. A broader Core run passed all 53 cases; its existing timestamp-bearing data-row labels prevented exact discovery/run name reconciliation, so only that unrelated dynamic-data method was excluded from the stable 48-case identity gate.
Preservation command arguments and results
The shared arguments below are the same Release/no-build/reporting arguments used by these runs. Each command used its own
$resultsdirectory.Both counterfactual producer variants were built/packed and used fresh acceptance caches. The final nine acceptance tests and assertions were retained. Resolved package paths and loaded DLL bytes were compared with the relevant package payloads; the intermediate filesystem-only package was identified by SHA-256
99228CAFB97DB999BA10E134F3F98F0BE4869B2CB6AC3F2B69D9C55ABE28CA81. Final consumer package hashes:5C7ACB1E349E92FF37D50CF98325C0883878671DC7822C55F22DE409CF4BEA2367096A1A5B396AC5448CA06EC529A83FF1343C890B9C5E8EE5AD992F95773F67Raw binlogs, TRX, command arrays, and provenance are retained locally and were not uploaded; this description supplies the reviewer-safe result excerpts. CI has not yet been assessed. The broader solution pack below failed its unrelated MSTest UAP package-count audit because UAP was intentionally excluded for the dotnet MSBuild engine; no guard or unrelated product was changed. Scoped affected packs/builds passed.
Test-value evidence
RetryFailedTests_TrxReports_ArePublishedOncePerAttemptplus the native SDK cases2.0; no skips.RetryFailedTests_NativeSdkPublishesExistingTrxPaths2.The seven direct cases cover persistent failure, recovery, threshold suppression, all-pass, zero retries, an absolute filename, and Retry disabled, including associated attachment content. Core unit coverage checks opt-in ownership boundaries, atomic records, wire identities, per-test attachments, ordinary result streaming, and explicit write failure. No separate old-binary unit counterfactual is claimed for the new internal helper APIs.
Review corrections: current revision
The implementation evidence above describes the initial commit. The current head is
69e603594c49c1d4952795eaa941f109c383d5cb, parent17492a777680dc7066e0c691e7fa2b855953e71d; its exact review-correction patch SHA-256 is8FCEF4DC6BAE19D199EBEA6E112F545FE9EA950FC8C62CEB9F9CBE62CD014EEF.This follow-up addresses the three inline regressions without changing the intended report layout or native AppContainer support:
Retries, read the final outcome from the root report, reject the final-attempt duplicate, and check the recursive report count.The main-branch packaged integration must be repacked with Core/Retry:
Microsoft.Testing.Extensions.PackagedApp.MSBuildembeds the controller's actual DLLs. Both itsnet10.0andnet11.0controller layouts were rebuilt and checked against the packed payloads. The five real native consumers'project.assets.jsonSHA-512 values matched all four current packages; their host Core/Retry/PackagedApp DLL SHA-256 values matched the respective package entries. No public API or SDK wire format was added. Release-branch/backport validation remains unperformed.Current local validation: 97 passing entries, zero failures or skips, across six selections: 69 extension tests on
net10.0, the three new failure-replay rows onnet462, 10 Core deferral cases, nine ordinary Retry acceptance cases, five real SDK 10.0.401 packaged cases, and one real WinUI AppContainer Retry case. The five native cases also passed again after restoring the counterfactual package. This is local execution evidence, not a CI verdict.PackagedAppContainerWinUI_RetryRunsSecondHostAndPublishesTrxexecuted and passed; the nativeNativeDotnetTest_ActivatesIdentityRequiredHostAndReturnsItsResults ("net10.0",True,True,"Retry")row verifies two physical reports, the finalOutcome/Passedassociation, and exactly those existing SDK-advertised paths.Configure_PreservesUserArgumentsAndLaunchProfileTail,Configure_RetryKeepsTransportAndExecutionIdentity,Configure_DotnetRunClearsStaleTransportAndKeepsTail,Configure_TestExplorerPreservesServerArgumentsWithoutNativeTransport,Configure_BootstrapFilePreservesArgumentTailAndRestartTransport, andGetConnectBackEnvironment_IncludesReporterRecoveryTransportpassed. The replay-owner pipe and unrelated secrets are not forwarded to the activated host.RetryArtifacts_CopyFailureReplaysOriginalPath,RetryArtifacts_PartialMoveReplaysBothPaths, andRetryArtifacts_ReplayFailurePreservesCopyExceptionpassed on bothnet10.0andnet462.Exact current commands (repository-local toolchain already bootstrapped;
$evidenceis a task-owned directory):The affected unit projects/TFMs and both acceptance projects also built with warnings as errors, exit
0. The 10 Core and nine ordinary acceptance cases above were rerun successfully on this implementation. The repository wrapper again hit the previously documented unrelated UAP-excluded package-count audit; no checks, assertions, or shared caches were weakened. Tests retain method-level/CPU-count scheduling; unit resources are GUID-owned and environment access is mocked, while the existing WinUI class exclusion is preserved. Package registrations were cleaned by the existing per-identity helpers, including the failing counterfactual.Test-value evidence for the review corrections:
RetryArtifacts_*rows named abovenet10.0Retry DLL from package SHA-25667096A1A5B396AC5448CA06EC529A83FF1343C890B9C5E8EE5AD992F95773F67in an isolated copied output.2, no skips.0. A subsequent green run confirms restoration.("net10.0",True,True,"Retry")rowThe native packaged SDK must advertise only the retained and final published report paths.It observesRetries/<id>/2/native_report.trxinstead of the existing root report. Four compatibility controls pass; outer exit2, no skips.0, no skips.These are deliberately labeled binary behavior-removal comparisons, not pristine-base source runs. Fresh per-run acceptance caches were used and the fixed controller package was restored in
finallywith its hash rechecked. The counterfactual package SHA-256 is225D338C936FB31C68695F4CA336E93A2A346DD2367BFBA847D0F4D0DACBE3E1.Current shipping package SHA-256 values:
D3DC655F63E93D14947DA8364E11FADE10F69FC37513ABEC3AF906A6B0D8D7FFE64BBD90CA6E677237D1291089D020CF7BB3F51F548B7FB5EC03CC87879002739E4480DE4D9BDE57CF1ABAC4B1D217E590DD2E1E9BF3FE31F64A5A4BE13B369008AD35891BC827F935AE393680B861B40C42FF3AFA2261DB874E3C685938E7FCRaw TRX/binlogs, exact command arrays, counterfactual packages, and consumer provenance remain in task-owned local evidence; the tables above are the published excerpts. Live Azure DevOps ingestion/UI and servicing-branch validation remain gaps. The capable WindowsAppModel CI template selects these existing categorized activation tests and provides the isolated SDK path, but local success is not claimed as execution by that CI job.
CI follow-up: public sample Retry probe
The preceding review-correction evidence belongs to
69e603594c49c1d4952795eaa941f109c383d5cb. The current head is022257a9a8b591e21b3227b4ac0b2b26026fb0db, with that revision as its parent; the exact two-file correction patch SHA-256 isC9A679C13D404920838125859141811FCCFD4A92B34B1CE8BB661BC6FEB04DB6.Azure build 1667628 failed only its Windows application-model leg. The public step log reports 12 executed tests, 11 passed, one failed, and zero skipped. The failure was
PublicPackagedWinUISample_AllExtensions_ExposeControllerSupport: its sample probe still expected both attempt reports belowRetries, so the correctly relocated final report caused Retry to be classified asFailedinstead ofVerified. The generated probe report confirms Retry was advertised and its command exited0; extension registration was not missing.This follow-up changes only
samples/public/WinUIMtpPackagedApp/Probe-Extensions.ps1and its README. It preserves the product implementation and the outer acceptance assertions. The probe now requires exactly one retained attempt-1 report belowRetries, exactly two TRX files recursively, no final-attempt copy belowRetries, the firstRetryFailsFirstAttemptoutcomeFailed, and the existing root result with the same name and outcomePassed.Test-value evidence: the unchanged acceptance case and selector ran against the same freshly packed, aligned producers in both states. With the old sample probe it executed once and failed the exact CI four-versus-three Verified-extension assertion, outer exit
2. With the corrected probe it executed once and passed, outer exit0; neither run skipped a test. This is a sample-validation repair, not a new product behavior or a pristine-base product counterfactual.Both runs used
TESTFX_RUN_WINDOWS_APP_MODEL_TESTS=1and the retained isolated SDK 10.0.401 throughTESTFX_DOTNET_10_PATH; the outer host used the repository-pinned SDK 12.0.100-alpha.1.26509.101. The four Core/Retry/PackagedApp/PackagedApp.MSBuild scoped packs and the acceptance build passed with warnings as errors. Generated sampleproject.assets.jsonSHA-512 entries matched those packages; all three actual sample Core/Retry/PackagedApp DLL SHA-256 values matched their package payloads.The full 14-entry probe result retains all existing support classifications, including the intentional coverage
Partialresult and unsupported reporterUnavailableresults. Retry is nowVerified, with one failed first-attempt execution and one distinct passed final execution in exactly two physical files. Windows PowerShell 5.1 syntax parsing passed. Existing package-cleanup helpers completed in both red and green runs. Raw TRX, binlogs, generated sample/probe artifacts, command arrays, package/binary provenance, and the original probe are retained in task-owned local evidence. No assertions, tests, checks, caches, or product behavior were bypassed; CI verification of this new head remains separate.Attachment-preservation review correction
The current head is
ad4539650f2c81b3a6b2bc85308c798e741687bb, parent18d0aa8e1a293bcae3170bba4566bcf2324f8f4a(the intervening merge frommain). Its exact correction patch SHA-256 is97AE3FC9523E11BD48C0E62C4D5811F1DE70DD70B866F086860BEC0B069EA9AA. Earlier validation sections remain revision-specific history.The new review exposed a concrete attachment regression: a test's
diagnostics.trxwas deleted solely because of its extension, leaving both retained and consolidated CTRF reports with a missing absolute attachment path. Deletion now requires the path of a final-attempt report explicitly identified asmicrosoft.testing.trx, as well as the.trxextension and absence of post-processing replacement. Unregistered files and test attachments retain their originals.Controller-generated TRX reports use
FileArtifact, not the test-hostSessionFileArtifactconsumed by Retry. They now register their path and kind through the existing retry-artifact manifest, after normal or recovered publication. The existing manifest writer was moved unchanged into its shared codec helper and reused; this adds no transport, SDK wire field, or public API. The old copy-only relocation overloads remain conservative when report ownership is unavailable.Retry and TrxReport must flow together with the matched platform/controller package set, including any servicing backport. The corrected report producer is needed to identify controller-owned reports; unknown or undeclared ownership is not permission to delete a file. The published package documentation now explicitly states that TRX-named attachments remain at their recorded paths.
Requirement coverage and test-value evidence:
Artifacts_TrxNamedAttachmentIsPreservedWithoutReportRegistration2.RetryTrxAttachment_CtrfOnlyandRetryTrxAttachment_CtrfAndTrxEvery retained and consolidated CTRF attachment must still exist at its recorded path.; outer exit2, no skips.0, no skips. All six report-to-attachment associations retain the expected name, outcome, absolute path and file contents.Final local validation totals 127 passing entries, zero selected failures/skips across those six selections. Build/setup failures from developing the fixture are not counted as behavioral red evidence. The valid red runs used the unchanged pre-correction deletion behavior with the regression tests retained; they are not claims about CI or every intermediate commit.
Exact attachment acceptance command, executed with the same selector and assertions in red and green:
The outer host uses the repository SDK
12.0.100-alpha.1.26509.101; generated ordinary consumers execute onnet11.0, while the real packaged SDK compatibility cases use SDK10.0.401. Scoped packs of Retry, TrxReport, CTRF, HTML, JUnit, PackagedApp and PackagedApp.MSBuild passed with warnings as errors, including the affectednetstandard2.0,net10.0,net11.0and Windows controller builds. Separate fixture caches were used. Eight resolved package/runtime-DLL checks match the actual current Core/Retry/TrxReport/CTRF payloads, and six retained/consolidated attachment associations were inspected independently. Raw commands, TRX/binlogs, baseline packages, patch bytes and provenance remain in task-owned local evidence.The original nine-case, one-case WinUI and five-case native packaged preservation commands listed earlier were rerun with these packages and passed with zero selected skips.
The ten generated catch/path-combination suggestions were replied to and resolved without unrelated rewrites: the compensation catch rethrows the original exception, and the flagged later path components are constants, GUIDs or controlled test data. CI verification of this new head and release-branch/backport validation remain separate.
Bazel source-layout follow-up
The current head is
70cb2cb901ea95f448f38fdaae24a543641f263b, parentad4539650f2c81b3a6b2bc85308c798e741687bb. This follow-up adds the same existingRetryArtifactManifest.cssource toeng/bazel/platform.bzl's TRXlinked_srcslist; it changes no C# behavior or test assertions.Bazel run 42255789242 on the preceding head failed both Windows and Linux TRX compilation with
CS0103: The name 'RetryArtifactManifest' does not exist in the current context. The.csprojlink was present, but native Bazel maintains a separate source list. The one-line inclusion corrects that build surface.The 127 passing SDK/runtime entries in the preceding section still describe the unchanged C# implementation at the parent revision. The local Bazel build validates the corrected source inclusion; the full replacement Windows/Linux CI matrix remains a separate gate. Three follow-up path-combination warnings on the new regression fixture were also replied to and resolved: their segments are a GUID, a literal, or a leaf name produced by
Path.GetFileName.