You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Mutation testing for the configured Microsoft.Testing.Platform core modules had 156 surviving mutants, leaving important edge cases and protocol behavior insufficiently verified. This change adds focused tests that exercise those paths and applies narrow Stryker suppressions only where mutations are equivalent or platform-specific.
The changes cover ServerMode process and protocol behavior plus MSBuild task generation, tool resolution, configuration, stack traces, executable parsing, Unicode boundaries, and duration suffix parsing.
Add focused tests and targeted mutation suppressions for equivalent or platform-specific behavior in the configured MTP core modules.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
Note
🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.
Summary
This PR is a mutation-testing hardening pass over Microsoft.Testing.Platform.MSBuild (DotnetMuxerLocator, InvokeTestingPlatformTask, StackTraceHelper, UnicodeCharacterUtilities, TestingPlatformAutoRegisteredExtensions), the shared StackTraceRegexHelper, and MtpServerInProcessHost, plus a large batch of new/expanded unit tests for all of the above and for TimeSpanParser, ErrorCodes/InitializeRequestArgs serialization, and MtpServerProcess. No public API, IPC wire format, or localization resources are touched.
StackTraceHelper.TryGetStackFrameLocation (src, production code): the inlined place = match.Groups["code"].Value correctly drops the dead code1-fallback branch, since TryGetStackFrameLocation is only ever invoked with matchFramesWithoutLocation: false (confirmed via GetOrCreateRegex), where the code1 group can never participate in a successful match. No regression.
Stryker disable once ... comments added across DotnetMuxerLocator*.cs, InvokeTestingPlatformTask*.cs, TestingPlatformAutoRegisteredExtensions.cs, UnicodeCharacterUtilities.cs, StackTraceRegexHelper.cs, TimeSpanParser.cs, and MtpServerInProcessHost.cs: spot-checked each rationale against the surrounding logic (e.g. TimeSpanParser's day-suffix branch is unreachable without the regex having already matched a day alias; UnicodeCharacterUtilities's <= '\u007F' check is provably equivalent to the Unicode-category path for that one code point; the Mach-O magic/cigam byte-order mapping is paired per architecture). All checked comments are accurate and explain why the mutant is semantically equivalent rather than just suppressing noise — consistent with documentation-accuracy guidance.
Null-forgiving operator change in InvokeTestingPlatformTask.ToolResolution.cs (.. EnvironmentVariables! instead of .. EnvironmentVariables ?? []): this is reached only inside the if (hasExplicitDotnetRoot || hasExplicitWindowsX86DotnetRoot) branch, and both flags are computed via ContainsEnvironmentVariable(...) which returns false whenever EnvironmentVariables is null, so EnvironmentVariables is guaranteed non-null at that point. The ??[] on the line below (reached only when neither flag is set) is correctly left untouched since EnvironmentVariables can still be null there. Safe, and improves mutation-coverage precision without behavior change.
New tests (DotnetMuxerLocatorTests, InvokeTestingPlatformTaskTests, MSBuildTests, StackTraceHelperTests, UnicodeCharacterUtilitiesTests/TimeSpanParserTests, MtpServerClientInProcessTests, MtpServerProcessTests, ProtocolMutationTests, SerializationMutationTests): use TestMethod/DataRow, reflection against private statics where needed, IsolatedAssemblyLoadContext for one-time-initialization assertions (correctly Unload()'d in finally), and polling (SpinWait.SpinUntil, bounded retry loop in AssertPortCanBeReboundAsync) rather than fixed sleeps for the port-rebind and process-tree checks — consistent with the flakiness-avoidance guidance. The SafeKillTerminatesEntireProcessTree change adds a readyFile rendezvous instead of relying on Start-Sleep -Seconds 2 alone, which removes a prior race rather than introducing one.
StackTraceHelperTests.TryFindLocationFromStackFrame_InitializesRegexWithBoundedTimeout: resets the cached s_regex static field to null before invoking TryFindLocationFromStackFrame, then re-reads it. This is a shared static touched by other tests in the same class; the assembly is marked [assembly: Parallelize(Scope = ExecutionScope.MethodLevel, Workers = 0)], and Workers = 0 disables actual parallel execution for this assembly, so there is no live cross-test race today — but flagging for awareness: if Workers is ever changed to a positive value, this test would race with any other test in the class that depends on StackTraceHelper's cached regex instance. No action required given current config.
No blocking, major, or moderate defects identified. The dependency-upgrade-assessment template does not apply (no eng/Versions.props or package-reference changes in this diff).
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🟢 Approval recommended
The focused tests and suppressions align with the stated mutation-coverage goals without introducing observable compatibility or correctness regressions.
Review effort: Balanced Findings: None
What changed in this PR
Adds focused mutation coverage for MTP ServerMode and MSBuild components while suppressing platform-specific or equivalent mutants.
Changes:
Expands protocol, process, configuration, generation, parsing, and tool-resolution tests.
Simplifies stack-frame extraction for location-only regexes.
Adds narrowly scoped Stryker suppressions.
Confidence at a glance
🟢 ServerMode and MSBuild
Reviewed lifecycle, serialization, platform-specific, parsing, and test-isolation changes; no blocking issues found.
This is a new test added by this PR (test/UnitTests/Microsoft.Testing.Platform.MSBuild.UnitTests/InvokeTestingPlatformTaskTests.cs) and it fails deterministically on Linux for both target frameworks:
Assertion failed. Expected string to contain the specified substring.
expected substring: "testingplatform.pipe."
actual: "/tmp/7bdddd678aa94243b1f8586a17496ebd"
Assert.Contains(Prefix, pipeName)
Root cause: the test's Unix branch asserts Assert.Contains(Prefix, pipeName) where Prefix = "testingplatform.pipe.", but NamedPipeServer.GetPipeName (in src/Platform/Microsoft.Testing.Platform/IPC/NamedPipeServer.cs) builds the Unix pipe path as Path.Combine(directory, name) where name is the raw GUID (e.g. 7bdddd678aa94243b1f8586a17496ebd) with notestingplatform.pipe. prefix segment — that prefix is only applied on the Windows branch ($"testingplatform.pipe.{name...}"). So on Unix the resulting path never contains the literal prefix string; only the filename format (a bare 32-char GUID "N" form) matches. The test's own fallback assertion two lines later (Assert.StartsWith(Prefix, fileName) on Path.GetFileName(pipeName)) correctly reflects this but is unreachable because the preceding Assert.Contains throws first.
Confidence: high — deterministic logic mismatch between test and implementation, reproducible on every Linux run in this build (both TFMs), not environment-dependent.
Affected matrix: Linux Release only (net8.0, net9.0); Windows legs pass because the Windows branch of GetPipeName does include the literal prefix.
Not flaky/historical: this is a brand-new test with no prior build history to compare against.
Next step: fix the new test's Unix branch to assert on fileName (e.g. drop the Assert.Contains(Prefix, pipeName) check, or assert the prefix against Path.GetFileName(pipeName) as the later lines already do) rather than the full path.
Fails on attempt 1, passes on attempt 2 in this build — and in all 10 available prior builds in history.json (including unrelated commits on main), with the same assertion ("The first AppContainer host fails so Retry must activate a second host."). This matches the test's own name and intent (verifying retry behavior that intentionally fails the first host), so it is working as designed and not related to this PR.
No issue is warranted for the retry test (consistent, by-design first-attempt failure, not a regression). An issue is warranted for the pipe-name test defect since it is a deterministic code/test bug, but since it was introduced by this same PR and is fixable by the PR author directly, filing a separate tracking issue is not necessary — please address it in this PR before merge.
Remove the reflection assertion for the deleted InvalidHandleValue field after merging the latest main branch.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use an explicit null check before StartsWith to avoid nullable lifting and the redundant Boolean comparison.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
disable once all also excludes observable mutants in the active branch, not just the inactive platform literal. For example, on Unix changing "dotnet" or selecting "dotnet.exe" would make EmbeddedRuntimeDefaults_AreInitializedInFreshAssembly fail (and the reverse is true on Windows), so the reported zero-survivor result no longer measures this behavior. Please narrow the suppression—e.g. move platform-name selection into a helper that accepts isWindows and test both inputs—rather than suppressing every mutant on this initializer.
STATUS: CLEAN. Both touched assemblies (Microsoft.Testing.Platform.MSBuild.UnitTests, Microsoft.Testing.Platform.ServerMode.Client.Sources.UnitTests) run MethodLevel parallelization with no [ResourceLock]/[DoNotParallelize] declarations added, removed, or weakened. All new/modified tests use per-test GUID temp dirs/files and ephemeral ports. The one process-global mutation (TESTINGPLATFORM_DEFAULT_HANG_TIMEOUT env var in EmbeddedRuntimeDefaults_AreInitializedInFreshAssembly, MSBuildTests.cs:722-750) is correctly guarded by [ResourceLock(WellKnownResources.EnvironmentVariables)] with capture/restore in a finally block. No Critical/High/Warning/Info findings.
Per-test grades
All 30 new or materially-modified test methods in this PR graded A. Highlights: the DotnetMuxerLocator PE-header boundary tests (RejectsOffsetJustPastLastReadableSignature, OffsetAtLastReadableSignature(InMinimalHeader)_IsNotClassifiedAsInvalidOffset) precisely pin the peHeader > fs.Length - sizeof(uint) off-by-one guard; the StackTraceRegexHelper/StackTraceHelper additions pin the exact compiled regex pattern and RegexOptions.Compiled flag; the cross-assembly ErrorCodes/Json reflection tests in ProtocolMutationTests/SerializationMutationTests verify the client-compiled copy of shared source stays wire-compatible with the server; and the MSBuild task tests (EntryPointTask_GeneratesEveryTemplateBranch, SelfRegisteredExtensions_GeneratesEveryLanguageAndNamespaceBranch) use MSTest's native [DataRow] table-driven pattern to cover every language/namespace/host-factory branch with strong value + message-log assertions and an explicit "no Stryker survivor" marker check. No assertion-strength, anti-pattern, or pseudo-mutation gaps were found; the per-branch conditional assertions in the two template-branch tests are accepted as idiomatic data-driven table tests rather than penalized as unstructured conditional logic.
Grade
Test
Mutation
Notes
How to improve
A (90–100)
new DotnetMuxerLocatorTests. GetMuxerArchitectureByPEHeaderOnWin_ RejectsOffsetJustPastLastReadableSignature
2/2 killed
Pins the exact off-by-one boundary (peHeader > length - 4) with two DataRow cases straddling the edge.
—
A (90–100)
new DotnetMuxerLocatorTests. GetMuxerArchitectureByPEHeaderOnWin_ OffsetAtLastReadableSignature_ IsNotClassifiedAsInvalidOffset
1/1 killed
Confirms the valid boundary offset is not misclassified as invalid.
—
A (90–100)
new DotnetMuxerLocatorTests. GetMuxerArchitectureByPEHeaderOnWin_ OffsetAtLastReadableSignatureInMinimalHeader_ IsNotClassifiedAsInvalidOffset
1/1 killed
Distinguishes "invalid offset" from "missing PE signature" at the minimal-header boundary.
—
A (90–100)
new DotnetMuxerLocatorTests. Constructor_UsesPlatformSpecificMuxerName
1/1 killed
Reflects on _muxerName and asserts the OS-specific value, killing the Stryker-flagged conditional.
—
A (90–100)
mod InvokeTestingPlatformTaskTests. LogEventsFromTextOutput_AzureDevOpsCommands_ AreWrittenToStdoutAtColumnZero_NotThroughMSBuildLog
1/1 killed
New assertion confirms both AzDO and normal lines always reach the captured _output buffer.
—
A (90–100)
new InvokeTestingPlatformTaskTests. AddAppHostDotnetRootEnvironmentVariable_ WithNoUserVariables_ AddsOnlyResolvedVariableAndLogsIt
2/2 killed
Verifies both the resulting environment variable and the low-importance log message.
—
A (90–100)
new InvokeTestingPlatformTaskTests. AddAppHostDotnetRootEnvironmentVariable_ UsesRequestedEnvironmentVariableComparison
2/2 killed
Exercises both the Windows (ordinal-ignore-case) and non-Windows (ordinal) comparison branches.
—
A (90–100)
new InvokeTestingPlatformTaskTests. Constructor_UsesPlatformSpecificPipeName
2/2 killed
Branches on path separator to assert the OS-appropriate pipe-name shape and embedded GUID.
—
A (90–100)
new MSBuildTests. EntryPointTask_GeneratesEveryTemplateBranch
N/A
20 DataRows cover every language/namespace/host-factory/entry-point combination with value + log assertions.
—
A (90–100)
new MSBuildTests. EntryPointTask_AcceptsMinimalQualifiedHostFactoryAndIdentifierCharacters
1/1 killed
Confirms minimal valid identifiers are accepted and correctly embedded in generated source.
—
A (90–100)
new MSBuildTests. EntryPointTask_RejectsInvalidCharactersInsideHostFactoryIdentifiers
1/1 killed
Asserts the exact error message and that no file is written on validation failure.
—
A (90–100)
new MSBuildTests. EntryPointTask_SanitizesRootNamespaceForNonVisualBasicLanguages
2/2 killed
Confirms namespace sanitization for both C# and F#.
—
A (90–100)
new MSBuildTests. SelfRegisteredExtensions_Fails_ WhenOneDuplicateMetadataValueDiffers
2/2 killed
Covers both metadata fields that can trigger the conflicting-duplicate error.
—
A (90–100)
new MSBuildTests. SelfRegisteredExtensions_ GeneratesEveryLanguageAndNamespaceBranch
N/A
5 DataRows cover every language/namespace combination with generated-call and log assertions.
—
A (90–100)
new MSBuildTests. SelfRegisteredExtensions_ FSharpWithoutHooks_EmitsUnitExpression
2/2 killed
Confirms the empty-hooks F# branch emits a unit expression and skips the hook-list log.
—
A (90–100)
new MSBuildTests. SelfRegisteredExtensions_ SanitizesRootNamespaceForNonVisualBasicLanguages
2/2 killed
Confirms namespace sanitization for both C# and F# extension generation.
—
A (90–100)
new MSBuildTests. SelfRegisteredExtensions_ StopsAfterFirstConflictingDuplicate
1/1 killed
Asserts exactly one error is reported, confirming early termination on first conflict.
—
A (90–100)
new MSBuildTests. SelfRegisteredExtensions_ MissingMetadata_ReportsExpectedItemSpec
2/2 killed
Covers both required metadata fields and asserts the exact diagnostic item-spec text.
—
A (90–100)
new MSBuildTests. ConfigurationFileTask_ MissingSourceAndDefaults_ ProducesNoOutputAndExactDiagnostics
1/1 killed
Asserts the exact ordered diagnostic message sequence for the missing-source path.
—
A (90–100)
new MSBuildTests. ConfigurationFileTask_ MissingSourceAndEmptyDefaults_ProducesNoOutput
1/1 killed
Confirms the empty-defaults path also logs the not-found message as the last entry.
—
A (90–100)
new MSBuildTests. EmbeddedRuntimeDefaults_ AreInitializedInFreshAssembly
1/1 killed
Reflection across an isolated AssemblyLoadContext pins embedded default constants; env var is saved/restored under [ResourceLock].
—
A (90–100)
new MSBuildTests. ConfigurationFileTask_ CopiesExistingSourceAndLogsEveryResolvedPath
1/1 killed
Asserts the full ordered log sequence and the copied-file content/path equality.
—
A (90–100)
new MSBuildTests. ConfigurationFileTask_ AllowsCommentsAndTrailingCommasAndWritesIndentedJson
1/1 killed
Verifies JSON-with-comments parsing, indentation, trailing newline, and comment stripping.
—
A (90–100)
mod StackTraceHelperTests. TryFindLocationFromStackFrame_ InitializesRegexWithBoundedTimeout
2/2 killed
New assertions pin the Compiled flag and the exact regex pattern string.
—
A (90–100)
new StackTraceHelperTests. CreateFrameRegexPattern_ WithOptionalLocation_MatchesBothFrameShapes
1/1 killed
Asserts the exact alternation pattern and both match/no-match frame shapes with captured groups.
—
A (90–100)
new StackTraceHelperTests. CreateFrameRegexPattern_ WithRequiredLocation_ProducesExactPattern
1/1 killed
Pins the exact required-location pattern string.
—
A (90–100)
new TimeSpanParserTests. TryParse_BareNumber_UsesMilliseconds
1/1 killed
Confirms the no-suffix default-unit dispatch to milliseconds.
—
A (90–100)
new TimeSpanParserTests. TryParse_ExplicitSuffix_ ParsesInvariantNumber
1/1 killed
Confirms fractional seconds parse with invariant culture.
—
A (90–100)
new TimeSpanParserTests. TryParse_MillisecondAliases_ UseMilliseconds
2/2 killed
Covers both the ms and mil-prefix disambiguation branches ahead of the minute prefix.
—
A (90–100)
new TimeSpanParserTests. TryParse_MinuteSuffix_UsesMinutes
1/1 killed
Confirms the minute-suffix branch.
—
A (90–100)
new TimeSpanParserTests. TryParse_HourSuffix_UsesHours
1/1 killed
Confirms the hour-suffix branch.
—
A (90–100)
new TimeSpanParserTests. TryParse_DaySuffix_UsesDays
1/1 killed
Confirms the fallthrough day-suffix branch.
—
A (90–100)
new MtpServerClientInProcessTests. LaunchInProcessAsync_ ZeroTimeoutFailedStartCleanup_ DoesNotCaptureTheCallingSynchronizationContext
1/1 killed
Reuses the class's established sync-context-pumping pattern; asserts no capture plus the exact timeout-failure message.
—
A (90–100)
mod ProtocolMutationTests. ErrorCodes_ MatchTheJsonRpcAndTestingProtocol
12/12 killed
Now also loads the client-compiled ErrorCodes copy via an isolated AssemblyLoadContext and asserts wire-value parity with the server.
—
A (90–100)
mod SerializationMutationTests. Json_DeserializeInitializeRequest_ ValidatesProtocolVersionEntries
2/2 killed
Refactored to deserialize through the client assembly's own Json type, verifying the client's validation/exception behavior directly.
—
This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.
Warning
Firewall blocked 1 domain
The following domain was blocked by the firewall during workflow execution:
github.com
To allow these domains, add them to the network.allowed list in your workflow frontmatter:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mutation testing for the configured Microsoft.Testing.Platform core modules had 156 surviving mutants, leaving important edge cases and protocol behavior insufficiently verified. This change adds focused tests that exercise those paths and applies narrow Stryker suppressions only where mutations are equivalent or platform-specific.
The changes cover ServerMode process and protocol behavior plus MSBuild task generation, tool resolution, configuration, stack traces, executable parsing, Unicode boundaries, and duration suffix parsing.
Validation
Microsoft.Testing.Platform.ServerMode.Client.Sources: 856 killed, 0 survived.Microsoft.Testing.Platform.MSBuild: 543 killed, 0 survived.git diff --checkpassed.