Repository navigation
Migrate MSTest desktop integration tests to inline assets - #12050
Conversation
Replace the remaining standalone desktop test assets with isolated, package-consuming acceptance scenarios. Keep internal reflection and real AppDomain-unload checks in the existing platform-services unit suite. Remove the retired asset projects, desktop integration project, solution registrations, and obsolete friend-assembly declaration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4408ddf2-7a88-481a-bf06-015814037163
There was a problem hiding this comment.
🧪 Test quality review — PR #12050
Compared merge base c83be2a87a95cdd8db8ce526d1827c5d480875f7 with head b7eaed36b08343447b9d5943d8e8ffdbe856e328. Evidence applies to this head only. This was a static review: no code, builds or tests were executed.
Test-value evidence
- Contract: moving the desktop integration coverage into acceptance and unit tests. The old
PlatformServices.Desktop.IntegrationTestsproject and its test assets are removed. ItsInternalsVisibleToentry and the solution entries go with it.- Search-directory resolution has two cases, with
DisableAppDomaintrue and false.- The acceptance test builds a net-framework VSTest consumer.
- The dependency is absent from the output directory and is only reachable through the runsettings
AssemblyResolutiondirectories. - The child test asserts the dependency's
Locationequals the configured path. - The TRX counters assert exactly 1 test run and passed.
- Static prediction: if resolution ignored the configured directories,
Assembly.Loadwould throw and the TRX would show a failure, so the test would catch it. This is not executed evidence.
DisposeShouldUnloadChildAppDomainassertsAppDomainUnloadedExceptionon both the domain and the proxy afterDispose. IfDisposeskippedAppDomain.Unload, these assertions would fail. Static prediction.- The
ReflectionUtilityTestsadditions were not traced in detail. Their coverage is an unverified evidence gap.
- Search-directory resolution has two cases, with
- No executed red/green records (CI per exact revision) were available. This is an evidence gap, not a defect.
🧵 Parallel safety
PARALLEL-SAFETY RESULT
STATUS: PARTIAL
The specialist subtask was not run, so parallel-safety coverage is incomplete and is not an all-clear. Manual observation only: DesktopAssemblyResolutionTests uses a separate generated asset per method, and its comments say it relies on method-level scheduling. I did not review the assembly-wide configuration.
Per-test assessment
Pseudo-mutation evidence below consists of static predictions, not executed mutation results.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | A (90–100) | new DesktopAssemblyResolutionTests. |
Asserts the dependency location and TRX counters in the parent domain. | None |
| Pass | A (90–100) | new DesktopAssemblyResolutionTests. |
Asserts the child-domain name and the resolved dependency path. | None |
| Pass | B (80–89) | new DesktopTestSourceHostTests. |
Checks both the domain and the proxy throw after unload. | None |
| Uncertain | — | mod ReflectionUtilityTests.* (ported methods) |
Not graded individually in this budget. | Review the ported methods against the removed desktop versions. |
This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · 24.5 AIC · ⌖ 0.824 AIC · ⊞ 22.3K · ◷
There was a problem hiding this comment.
Note
🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.
The PR replaces the legacy PlatformServices.Desktop.IntegrationTests project and its test assets with a package-consuming acceptance test plus unit tests. I found no blocking issues, only a few test-robustness suggestions.
Reviewer actions
- Recommended – Use a structural check (
!AppDomain.CurrentDomain.IsDefaultAppDomain()) instead of matching the AppDomain friendly-name prefix. Done when the test no longer depends on theTestSourceHostname string. - Recommended – Improve failure diagnostics in
DesktopAssemblyResolutionTestsby including the run output and checking that the TRX file exists before loading it. Done when a CI failure shows the child run's output. - Recommended – Assert on or remove the unasserted
TestContext.WriteLinediagnostics. Done when no unasserted extra assembly loads remain in the child test.
Change classification
- Test-infrastructure cleanup. The removed project, assets and
InternalsVisibleToentry have no remaining references in the repo (I searched the checkout). - New Windows-only acceptance test, in
DesktopAssemblyResolutionTests, coveringAssemblyResolutionsearch directories with and without AppDomain isolation. - New unit tests for AppDomain unload on dispose and for MSTest attribute reflection behavior.
Build, dependencies & scripts
No dependency version changes. I checked .slnx, .slnf and the InternalsVisibleTo removal and found them consistent with the deleted projects.
Test-value evidence
This review was static only, since I did not execute untrusted PR code. I could not confirm that the new tests fail without the corresponding behavior. Please record a red/green comparison for the acceptance test and DisposeShouldUnloadChildAppDomain.
Specialist summary
- API and compatibility: no public API changes.
- Test quality: the tests assert specific outcomes, and the unit tests are net-framework-gated where needed. The remaining concerns are the three suggestions above.
There was a problem hiding this comment.
🟢 Approval recommended
The focused test migration preserves the stated contracts, strengthens lifecycle checks, and leaves no identified blocking issues.
No open issues 🎉
What changed in this PR
Consolidates MSTest desktop integration coverage into existing acceptance and unit suites, removing separately built test assets without changing shipped behavior.
Changes:
- Adds isolated, inline .NET Framework consumers for parent/child AppDomain assembly resolution.
- Migrates twelve reflection contracts and strengthens AppDomain disposal assertions.
- Removes seven asset projects, the retired integration host, and obsolete registrations.
Test-value evidence: For revision b7eaed36b08343447b9d5943d8e8ffdbe856e328, the PR reports 15 baseline, 164 final unit, and 18 acceptance tests passing. Execution artifacts were not independently inspected; mutation sensitivity and CI execution remain unverified.
Confidence at a glance
🟢 Test migration and project cleanup
The replacement coverage and removals are consistent with the stated scope. No blocking defects were identified.| File | Description |
|---|---|
| TestFx.slnx | Removes retired project registrations. |
| test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Utilities/ReflectionUtilityTests.cs | Adds twelve migrated reflection contracts. |
| test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Services/DesktopTestSourceHostTests.cs | Verifies real domain and proxy unloading. |
| test/IntegrationTests/TestAssets/TestProject/TestProjectForDiscovery.csproj | Deletes reflection asset project. |
| test/IntegrationTests/TestAssets/TestProject/Properties/AssemblyInfo.cs | Removes migrated assembly attributes. |
| test/IntegrationTests/TestAssets/TestProject/AttributeTestClasses.cs | Removes migrated attribute fixtures. |
| test/IntegrationTests/TestAssets/TestIdProject.FullyQualifiedTestStrategy/TestIdProject.FullyQualifiedStrategy.csproj | Deletes legacy identity asset project. |
| test/IntegrationTests/TestAssets/TestIdProject.DefaultStrategy/TestIdProject.DefaultStrategy.csproj | Deletes default identity asset project. |
| test/IntegrationTests/TestAssets/TestIdProject.DefaultStrategy/TestIdCases.cs | Removes legacy identity fixtures. |
| test/IntegrationTests/TestAssets/SampleProjectForAssemblyResolution/SerializableTypeThatShouldBeLoaded.cs | Removes legacy resolution fixture. |
| test/IntegrationTests/TestAssets/SampleProjectForAssemblyResolution/SampleProjectForAssemblyResolution.csproj | Deletes resolution asset project. |
| test/IntegrationTests/TestAssets/SampleFrameworkExtensions/SampleFrameworkExtensions.csproj | Deletes custom-attribute asset project. |
| test/IntegrationTests/TestAssets/SampleFrameworkExtensions/ExtensionAttributes.cs | Removes migrated custom attributes. |
| test/IntegrationTests/TestAssets/FixturesTestProject/UnitTest1.cs | Removes legacy lifecycle fixtures. |
| test/IntegrationTests/TestAssets/FixturesTestProject/FixturesTestProject.csproj | Deletes lifecycle asset project. |
| test/IntegrationTests/TestAssets/Directory.Build.targets | Removes obsolete asset build settings. |
| test/IntegrationTests/TestAssets/DesktopTestProjectx86Debug/UnitTest1.cs | Removes legacy desktop fixtures. |
| test/IntegrationTests/TestAssets/DesktopTestProjectx86Debug/DesktopTestProjectx86Debug.csproj | Deletes desktop asset project. |
| test/IntegrationTests/PlatformServices.Desktop.IntegrationTests/ReflectionUtilityTests.cs | Removes tests migrated to unit coverage. |
| test/IntegrationTests/PlatformServices.Desktop.IntegrationTests/Program.cs | Removes retired integration host entry point. |
| test/IntegrationTests/PlatformServices.Desktop.IntegrationTests/PlatformServices.Desktop.IntegrationTests.csproj | Deletes retired integration project. |
| test/IntegrationTests/PlatformServices.Desktop.IntegrationTests/DesktopTestSourceHostTests.cs | Removes replaced domain tests. |
| test/IntegrationTests/MSTest.Acceptance.IntegrationTests/DesktopAssemblyResolutionTests.cs | Adds isolated package-consuming resolution tests. |
| src/Adapter/MSTestAdapter.PlatformServices/MSTestAdapter.PlatformServices.csproj | Removes obsolete friend-assembly declaration. |
| MSTest.slnf | Removes retired projects from the filter. |
Review effort: Balanced
Remove diagnostic-only product assembly loads from the generated consumer and verify that its TRX report exists before attaching or reading it. Keep the existing exit-code helper, which already includes full run output. Document why IsDefaultAppDomain cannot identify the MSTest child: both the VSTest parent and the MSTest child are non-default domains. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4408ddf2-7a88-481a-bf06-015814037163
|
Final test-triage resolution — Azure Pipelines build 1642452 Build
|
There was a problem hiding this comment.
🧪 Test quality review — PR #12050
Compared merge base c83be2a87a95cdd8db8ce526d1827c5d480875f7 with head 532b8d91499abce976305155fcb69efb8c61777a. Evidence applies to this head, not every intermediate commit. No inline findings: no verified, actionable defect survived the publication gate.
Test-value evidence
Static analysis only; nothing was executed. The PR's own green-before/green-after numbers were not independently verified (CI/execution records unavailable here).
| Contract | Scenario / decisive assertion | Counterfactual | Evidence |
|---|---|---|---|
| Configured assembly-resolution directories honored in parent and child AppDomain | DesktopAssemblyResolutionTests (2 methods): dependency is absent from the output dir, the child test loads it and asserts the expected Location, value and domain choice; the outer test asserts the TRX counters (1 total, 1 passed) |
Dropped AssemblyResolution directory probing, or inverted DisableAppDomain handling → the child test fails and the TRX shows failed/0 passed |
Static prediction: detected. Executed: unverified |
Attribute inheritance, AttributeUsage handling, custom/array values, assembly-level identity |
12 new ReflectionUtilityTests cases with exact ordered Equal(...) expectations |
Lost base-class inheritance, wrong ordering, a truncated array, or a wrong runtime type → exact-sequence assertions fail | Static prediction: detected. Executed: unverified |
TestSourceHost.Dispose unloads the real AppDomain |
DisposeShouldUnloadChildAppDomain: AppDomainUnloadedException from both the captured domain and the proxy |
A fake dispose that only nulls AppDomain → both throw-assertions fail |
Static prediction: detected. Executed: unverified |
Removal of TestAssets/desktop integration project, slnx/slnf entries, InternalsVisibleTo |
Build/packaging only | n/a | Not applicable. Build/CI verification not run here |
No mutation score is claimed. Replacement tests have new identities, so this is not a paired red/green record.
🧵 Parallel safety
PARALLEL-SAFETY RESULT
STATUS: PARTIAL
NOTES: The dedicated specialist task was not run, so coverage is incomplete and not a clean result.
END PARALLEL-SAFETY RESULT
Informal observation, not a specialist result: each DesktopAssemblyResolutionTests method generates its own asset, settings and reports. The class has no shared mutable fields or [DoNotParallelize]. No parallelization configuration (runsettings/testconfig/assembly attributes) changed.
Per-test assessment
Pseudo-mutation evidence below consists of static predictions, not executed mutation results.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | A (90–100) | new DesktopAssemblyResolutionTests. | Asserts child-domain selection, dependency path and value, and TRX counters. | None |
| Pass | A (90–100) | new DesktopAssemblyResolutionTests. | Same contract with the AppDomain disabled. | None |
| Pass | A (90–100) | new ReflectionUtilityTests. (DesktopTestSourceHostTests) | Proves real unload via the domain and the proxy. | None |
| Pass | A (90–100) | new ReflectionUtilityTests. | Exact ordered values cover the single-use Owner and multi-use category. | None |
| Pass | A (90–100) | new ReflectionUtilityTests. | Checks runtime type and ordered values on a dynamic assembly. | None |
| Pass | A (90–100) | new ReflectionUtilityTests. | Asserts type, name and value in order. | None |
| Pass | A (90–100) | new ReflectionUtilityTests. | Exact array contents asserted. | None |
| Pass | B (80–89) | new ReflectionUtilityTests. and GetMSTestCategories* (8 further tests) | Exact-sequence assertions per scenario; several overlap in contract. | None |
This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · 37.7 AIC · ⌖ 0.829 AIC · ⊞ 22.3K · ◷
Pull request
Summary
Consolidate MSTest integration coverage into the existing acceptance suites instead of maintaining separately built asset projects and loading their sibling build outputs.
TestAssetsprojects, the retired desktop integration project, their solution/filter registrations, and the obsolete friend-assembly declaration.Only the two acceptance projects remain under
test/IntegrationTests. Shipped behavior, public APIs, package versions, and assembly scheduling are unchanged. Related issue: N/A; user-requested test-infrastructure cleanup.Design and decision record
This is a behavior-preserving test migration, not a new product feature, so a separate feature decision record is not needed. Existing
AppDomainTestsandAssemblyResolutionTestsestablished the inline Framework/multi-project patterns.Keep implementation-level reflection checks in unit tests rather than weakening them into process-exit smoke tests. The new acceptance consumers deliberately use the .NET Framework VSTest path: source-generated metadata or a modern-.NET-only run would not exercise the AppDomain boundary. Each method owns its generated projects, settings, outputs, and reports, so method-level parallelization remains enabled.
Demonstration
There is no UI change; an execution transcript is more useful than screenshots:
Each new acceptance scenario additionally verifies exactly one executed/passed
LoadsDependencyFromConfiguredSearchDirectorychild test, the actual parent/child domain choice, the configured dependency's physical location, and its exact returned value. The dependency is deliberately absent from the test output's normal probing path.Validation and regression proof
Local validation used the pinned SDK
12.0.100-alpha.1.26509.101, Release configuration, MSTest4.6.0-dev, MTP2.6.0-dev, and innerMicrosoft.NET.Test.Sdk18.10.1. Framework targets ran on the installed in-place .NET Framework 4.8.1 CLR; modern targets used matching runtimes without roll-forward overrides.The verified source is now committed as
b7eaed36b08343447b9d5943d8e8ffdbe856e328. Runs occurred before committing, against basec83be2a87a95cdd8db8ce526d1827c5d480875f7plus the byte-identical migration delta, bound by file-manifest SHA-256DDFC3E5E50462E0845DF063BC854AADF2B2F9610F35976822BE5F94C137FA3DE. Source content was rechecked before committing; tests were not rerun merely to create the PR.Commands and execution/provenance checks
$Edenotes the task-owned evidence directory; its machine-specific location is omitted. Artifact basenames below identify the retained local logs/reports.The original four required projects were built with the pinned SDK before the baseline host ran:
All four builds exited 0; the baseline executed/passed all fifteen source-inventoried cases. These commands describe the original source state, before deleting the projects.
The final pack rebuilt the solution after retiring the projects/friend entry:
Exit 0, zero warnings/errors, and successful NuGet content/metadata validation. The runtime-install skip avoids unrelated runtime installations; selected missing pinned runtimes were installed through repository tooling after launch failures, then the affected selections passed. Setup failures are not counted as test executions. PATH was shortened only in the validation process because this machine's inherited PATH prevented the build wrapper from launching PowerShell.
Final unit commands, with each row below expanded into this same argument list:
$tfm$logFinal acceptance command:
This selected both new
DesktopAssemblyResolutionTestsmethods, six existing AppDomain true/false/unspecified run/discovery cases, both existing assembly-resolution cases, the Framework resolver-recursion regression, and seven discovery-identity methods. All 18 executed/passed.Every outer TRX was reconciled against an independent source-derived identity plan using
.github\skills\testfx-acceptance-validation\scripts\verify_execution.py; JUnit exports were parsed independently and matched the counts. Reports/binlogs are retained locally, not uploaded; the inline commands and observed counts are the public evidence excerpt.For the two new generated consumers, all six restored package SHA-512 values matched the final shipping nupkgs, and all eight actually loaded framework/adapter/platform-services/MTP binaries matched the corresponding packaged payloads. Each child build/test exited 0 and executed exactly one test. Final package SHA-256 values:
3AD23ACF0BF5BDFF762E5C0FCEAB4565A4CCEA38B9C23CB62C6A4D5E66A8B5A4A149D7A9D3303BB4B9D7E12B9F032188A875E10AE139DE91D1665DF0548EA85FF9AF3F5139C764EB5F2B873782F745766E47C2E67C31B348511ED8E9F043AAD6Test-value evidence
ParentDomainShouldHonorSearchDirectoriesSpecifiedInRunsettingsandChildDomainResolutionPathsShouldHaveSearchDirectoriesSpecifiedInRunsettingspassed with real shipping packages and asserted domain selectionGetCustomAttributesOnMSTestOverrideShouldHonorAttributeUsage,GetMSTestPropertiesShouldIncludeUserDefinedAndInheritedAttributes,GetCustomArrayAttributesShouldPreserveAllValues, andGetMSTestAssemblyCategoriesShouldPreserveValuesAndTypeIdentityDisposeShouldUnloadChildAppDomainpassed on net462/net48 and additionally requires both the captured domain and remote proxy to throwAppDomainUnloadedExceptionThis is executed green-before/green-after contract-preservation evidence, not an executed mutation campaign; replacement tests have different identities. Static reasoning predicts detection of dropped probing, inverted domain selection, lost inheritance, wrong attribute identity, truncated arrays, and property-only fake disposal. No mutation score, red-phase result, or CI execution is claimed. CI execution remains unverified.