Repository navigation
Update CI validation dependencies and configuration - #12047
Conversation
Align the hosting sample dependencies with the repository pins, make the Unix fallback test independent of effective user permissions, and give only affected test map collection an independent time budget. Add configuration regression guards and document the collection budget. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 90a39100-658c-4065-b3fa-e47ff912f474
There was a problem hiding this comment.
🧪 Test quality review — PR #12047
Compared merge base 780c775d357d6ca25eee67b9469133dae3ee1730 with head
a01e243ed8bf2653d2d804778ef636447e55adc0. Evidence applies to this head, not
every intermediate commit.
Test-value evidence
| Contract / case | Counterfactual | Without behavior | Correct implementation | Evidence / gap |
|---|---|---|---|---|
Unix apphost EACCES retry / StartAsyncOnUnixRetriesThroughDotnetWhenApphostInterpreterIsNotExecutable (renamed+rewritten from StartAsyncOnUnixWhenOnlyNonApplicableExecuteBitIsSetRetriesThroughDotnet) |
Reverse/remove ShouldRetryApphostThroughDotnet's EACCES predicate in MtpServerProcess.LaunchCommand.cs, or restore the old GroupExecute-bit fixture. |
Static: predicts Assert.ThrowsExactlyAsync<MtpServerConnectionClosedException> plus the log-content Assert.Contains("...retrying through 'dotnet") assertion would fail — the retry message would never be logged. |
Not independently executed by this review. | The old fixture relied on a GroupExecute-only bit denying the current process's execution, which depends on effective UID/GID relative to the file's group — unreliable under root or same-group CI runners. The new fixture arranges a shebang script whose interpreter itself lacks any execute bit, producing EACCES deterministically on Linux/macOS regardless of privilege. This is a genuine fragility fix, not cosmetic; both old and new versions exercise the same production retry path (MtpServerProcess.Startup.cs's catch (Win32Exception ex) when (ShouldRetryApphostThroughDotnet(...))). The PR description's paired red/green transcript (broken-fallback copy → assertion failure; restored client → pass) is an author claim, not verified by this read-only review. |
CI collection time-budget isolation / SharedCollectionTimeoutIsRejected, ReversedCollectionTimeoutIsRejected, ExtendedMainValidationTimeoutIsRejected, ExtendedPullRequestTimeoutIsRejected (eng/validate-affected-tests.ps1) |
Feed each described old/mutated pipeline YAML string to the unchanged Assert-CollectionJobIsolation guard. |
Not applicable (PowerShell self-test harness, not a gradable C# test method). | n/a | Verified statically: the new $mainBudget/$collectionBudget regex in Assert-CollectionJobIsolation requires the exact $[ iif(eq(variables._CollectAffectedTests, 'true'), 360, 240) ] expression paired with the unchanged else: timeoutInMinutes: 120, so a reversed condition, a flattened 240, or an inflated PR timeout would each fail to match and throw CollectionHasIndependentTimeBudget. This matches azure-pipelines.yml's actual new timeout expression. Not independently executed; PR's own report claims 53/53 self-checks passed. |
Package pin alignment (MessagePack 3.1.11 / Nerdbank.MessagePack 1.3.88 added to AspireAppHost.csproj and AspireTests.csproj) |
n/a — build/package composition only, no executable assertion. | Not applicable. | n/a | Verified statically that both added explicit <PackageReference> versions exactly match the centrally pinned Directory.Packages.props entries (Nerdbank.MessagePack 1.3.88, MessagePack 3.1.11), which per that file's own comment exist specifically to override a vulnerable transitive version pulled in via StreamJsonRpc. No paired build executed by this review; not applicable as a behavioral regression test. |
Unassessed: none — all three distinct changed contracts in this PR are covered above; the only contract with a gradable C# test method is the first row.
🧵 Parallel safety
STATUS: CLEAN
| Test assembly | Scope | Workers | Analyzer coverage |
|---|---|---|---|
| Microsoft.Testing.Platform.ServerMode.Client.Sources.UnitTests | MethodLevel ([assembly: Parallelize(Scope = ExecutionScope.MethodLevel, Workers = 0)], unchanged by this PR) |
0 (CPU count) | Attribute-based opt-in; MSTEST0074–0077 not yet on main |
No findings (Critical=0, High=0, Warning=0, Info=0). The modified test creates its interpreter and App files solely inside its own fresh per-test TempDirectory.Create() instance (unique Guid.NewGuid()-named directory) and mutates only those files' own Unix permission bits via the pre-existing File.SetUnixFileMode/MakeExecutable helpers. It touches no environment variable, working directory, static field, or path reachable by any sibling test. The class-level [DoNotParallelize] (scoped only to the unrelated FailedLaunchDisposesProcessHandles, per the class's XML remarks) is unaffected by this change, and no parallelization configuration file changed in this PR.
Per-test assessment
Pseudo-mutation evidence below consists of static predictions, not executed mutation results.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | A (90–100) | mod MtpServerProcessTests. |
Clear AAA structure; exact exception type plus decisive log-content assertion together verify the EACCES retry path. | None |
Pseudo-mutation evidence — StartAsyncOnUnixRetriesThroughDotnetWhenApphostInterpreterIsNotExecutable: Changing the production guard ShouldRetryApphostThroughDotnet in MtpServerProcess.LaunchCommand.cs to always return false (never retry) would leave the original Win32Exception (EACCES) unhandled by StartAsync's catch clause, so Assert.ThrowsExactlyAsync<MtpServerConnectionClosedException> would observe the wrong exception type and fail — Likely killed (inferred). Separately, if the retry still occurred but the warning-log call in MtpServerProcess.Startup.cs were removed or its message text changed, the subsequent Assert.Contains("The sibling apphost could not be executed; retrying through 'dotnet", log.ToString(), ...) would fail — Likely killed (inferred). No credible candidate survivor was found within this test's own claimed behavior (EACCES-triggered retry-through-dotnet with the accompanying warning log).
Production-dependent behavior coverage: assessed — the full retry path (MtpServerProcess.Startup.cs's catch (Win32Exception ex) when (ShouldRetryApphostThroughDotnet(...)) branch and MtpServerProcess.LaunchCommand.cs's ShouldRetryApphostThroughDotnet/IsApphostCandidate helpers) was resolved and traced.
This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · 181 AIC · ⌖ 9.73 AIC · ⊞ 22.3K · ◷
There was a problem hiding this comment.
🔵 Needs a closer look
Main-only timeout evaluation and successful collection artifact publication still require Azure Pipelines confirmation.
No open issues 🎉
What changed in this PR
Updates CI validation dependencies and fixtures without changing production launch behavior.
Changes:
- Aligns Aspire sample serialization dependencies with repository pins.
- Reworks the Unix fallback test to avoid effective-user permission assumptions.
- Extends only Windows affected-test collection to six hours, with regression guards preserving other validation budgets.
Test-value evidence: For a01e243ed8bf2653d2d804778ef636447e55adc0, the PR reports Unix fallback mutation checks, 53 validation self-checks, and a passing Aspire sample test. Execution artifacts were not independently verified.
Confidence at a glance
🟡 CI validation
Main-branch execution must still confirm effective timeout budgets and complete collection artifact publication.
| File | Description |
|---|---|
| test/UnitTests/Microsoft.Testing.Platform.ServerMode.Client.Sources.UnitTests/MtpServerProcessTests.cs | Uses a non-executable interpreter to exercise Unix fallback. |
| samples/public/MTPHostIntegration/AspireTests/AspireTests.csproj | Pins serialization dependencies for sample tests. |
| samples/public/MTPHostIntegration/AspireAppHost/AspireAppHost.csproj | Pins matching app-host serialization dependencies. |
| eng/validate-affected-tests.ps1 | Adds timeout isolation guards and self-checks. |
| docs/affected-test-selection.md | Documents budgets, rationale, and remaining validation. |
| azure-pipelines.yml | Gives Windows collection an independent six-hour budget. |
Review effort: Balanced
|
/backport to rel/4.5 |
|
Started backporting to rel/4.5: https://github.com/microsoft/testfx/actions/runs/40328474903 |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --3way --empty=keep --ignore-whitespace --keep-non-patch changes.patch
Patch format detection failed.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
6 similar comments
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
10 similar comments
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Amaury Levé (@Evangelink) backporting to rel/4.5 failed, the patch most likely resulted in conflicts: $ git am --continue
fatal: Resolve operation not in progress, we are not resuming.
Error: The process '/usr/bin/git' failed with exit code 128NOTE: A PR will be created, but needs to be revised manually! |
|
Potential infinite loop guard hit. Stopping |
Replace the rate-limit HTML placeholder with the release-applicable test fix from microsoft#12047. Keep runtime pins, CI configuration, samples, and production launch behavior unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Pull request
Summary
Update CI validation dependencies and configuration following unofficial run 1639792.
Design and decision record
This repairs existing validation rather than introducing a feature or changing a shipping API. The collection rationale records the observed run: collection spent about 70 minutes on unit-test maps and another 88 minutes on MTP acceptance maps before starting MSTest acceptance; the four-hour job limit then canceled collection and artifact publication.
Extending only collection preserves every suite/TFM, collector concurrency, blocking failures, rollout gates, and ordinary validation budgets. Dropping tests or normalizing collection failures to success was rejected. Six-hour completion still needs confirmation in CI.
The permission fixture now arranges EACCES directly instead of assuming a group-only execute bit denies the current process access. The test keeps the existing fallback and logging assertions and its per-test temporary-directory ownership; assembly method parallelism is unchanged.
Demonstration
A text transcript is clearer than screenshots for these CI-only changes:
The real sample consumes its configured MSTest.Sdk 4.5.0-preview.26480.9, MTP 2.5.0-preview.26480.9, and Aspire 13.5.3 packages. The resolved serialization assets in both projects'
project.assets.jsonmatch the built output DLL hashes. No repository shipping package is modified.Validation and regression proof
Tested source: base
780c775d357d6ca25eee67b9469133dae3ee1730, now committed without source changes asa01e243ed8bf2653d2d804778ef636447e55adc0. The tested and committed patches have SHA-2566716B7306FBA4B6190BCA6B70E35F2CA38006745039D6146E9370EAEFAF3AEB9. Validation used the repository-pinned SDK12.0.100-alpha.1.26509.101, Release.The commands below preserve the executed arguments;
$evidencedenotes the task-owned report directory and$projectthe absolute path of the unit-test project. Machine-specific path prefixes are omitted. Binlogs/TRX and comparison inputs remain in session artifacts and have not been uploaded.On Debian/WSL, the Windows-built
net11.0test module was executed with the pinned Linux runtime11.0.0-rc.2.26504.105, retaining the assembly's method-level schedule and 32 workers. With$testModuleselecting each isolated variant and$reportsits separate result directory:The execution verifier independently confirms the named Unix regression, its unprivileged run, and
ApiGreetingIsAvailableeach selected and passed exactly one test with no skips. The broader suites' platform skips are not counted as regression proof.Not run: macOS execution, Azure Pipelines plan execution for the final revision, and a complete multi-hour collection/artifact-publication run. The main-only collection entry will not be exercised by an ordinary PR validation run. Local checks do not establish CI success or prove the new six-hour limit is sufficient.
Test-value evidence
StartAsyncOnUnixRetriesThroughDotnetWhenApphostInterpreterIsNotExecutableEA0237D999B6D558F863E92A2AE829F8DB9032AE49B8FE92A86710C2D1A6B697.SharedCollectionTimeoutIsRejected,ReversedCollectionTimeoutIsRejected,ExtendedMainValidationTimeoutIsRejected,ExtendedPullRequestTimeoutIsRejectedAssert-CollectionJobIsolationguard.CollectionHasIndependentTimeBudget, exit 1; final YAML accepted, exit 0. All 53 self-checks pass. Full collection completion remains unverified.DistributedApplicationTests.ApiGreetingIsAvailable