Skip to content

Add direct unit tests for MSBuildCompatibilityHelper - #11767

Merged
Amaury Levé (Evangelink) merged 4 commits into
mainfrom
copilot/test-improver-add-unit-tests-msbuildcompatibilityh
Oct 7, 2026
Merged

Amaury Levé (Evangelink) merged 4 commits into
mainfrom
copilot/test-improver-add-unit-tests-msbuildcompatibilityh

Conversation

Copilot AI commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

MSBuildCompatibilityHelper guards MSBuild APIs newer than the oldest supported SDK, but previously had no direct tests.

  • Feature gates: Assert multiline and extended-message support against the loaded MSBuild assembly.
  • Extended logging: Verify that a supported call logs one ExtendedBuildMessageEventArgs with the expected type, message, importance, and metadata.

Copilot AI balanced review requested due to automatic review settings October 6, 2026 08:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review any files in this pull request. Check if the Files changed in this pull request are included in default exclusions.

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 6, 2026 08:10
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Copilot AI changed the title [WIP] Add unit tests for MSBuildCompatibilityHelper Add direct unit tests for MSBuildCompatibilityHelper Oct 6, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The tests correctly cover the intended supported-version behavior and follow repository conventions.

Review effort: Balanced
Findings: None

@Evangelink
Amaury Levé (Evangelink) marked this pull request as ready for review October 6, 2026 09:30
@Evangelink
Amaury Levé (Evangelink) enabled auto-merge (squash) October 6, 2026 09:30
@github-actions github-actions Bot added the state/needs-review Awaiting review from the team. label Oct 6, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Scope: Single new test file (40 lines), no production code touched. Test-only change, no public API impact.

Correctness & test quality

  • SupportsMultiLine()/SupportsTerminalLoggerWithExtendedMessages() assertions correctly reflect the pinned MSBuild package version (≥17.10), matching the version-gating logic in MSBuildCompatibilityHelper. The explanatory comment on the pinned-version assumption is a nice touch.
  • TryWriteExtendedMessage test follows AAA structure, uses Moq's IBuildEngine appropriately (the only way to observe LogMessageEvent without a real build engine), and verifies type, message, importance, and ExtendedMetadata reference equality — good assertion diversity beyond a single Assert.IsTrue.
  • Idioms (Assert.ContainsSingle, Assert.IsInstanceOfType<T>) are consistent with sibling tests in this project (e.g. MSBuildLifecycleCallbacksTests.cs), and Moq usage is permitted here (BannedSymbols.txt only bans AwesomeAssertions, not Moq).
  • Null-value metadata entry (["empty"] = null) is a reasonable touch for exercising the Dictionary<string, string?> nullable-value contract.

Coverage gap (non-blocking)

  • Only the "supported" branches of the two feature-gate methods are exercised. The "unsupported" (older MSBuild / TryWriteExtendedMessage returning false) paths remain untested, but since the MSBuild package version is pinned centrally in this repo and can't be swapped per-test without faking the assembly's AssemblyFileVersionAttribute, this is a reasonable scope limitation rather than an omission — flagging only for awareness, not requesting changes.

No blocking issues found. This is a solid, idiomatic addition that closes a real test gap (#11761).

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 6, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 6, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. labels Oct 6, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #11767

🧵 Parallel safety

Status: CLEAN

Test assembly Scope Workers Analyzer coverage
Microsoft.Testing.Platform.MSBuild.UnitTests MethodLevel ([assembly: Parallelize(Scope = ExecutionScope.MethodLevel, Workers = 0)]) 0 → CPU count (live concurrency) Attribute-based opt-in; MSTEST0074-0077 analyzers not yet on main
  • i️ Info — MSBuildCompatibilityHelperTests.cs:9-24: the three new tests populate shared statics (s_msBuildVersion, s_supportsMultiline, s_supportsTerminalLoggerWithExtendedMessages) in MSBuildCompatibilityHelper via an unsynchronized ??= pattern. Under this assembly's MethodLevel parallelization these tests can race on initialization, but the computation is pure/deterministic (no env, CWD, or console dependency), so concurrent writers converge on the identical value — a benign race, not a correctness hazard. No other test in the assembly touches these statics today. No action needed.

No production source or parallelization-configuration files changed in this PR.

Per-test grades

GradeTestMutationNotesHow to improve
B (80–89) new MSBuildCompatibilityHelperTests.
SupportsMultiLine_
WithCurrentMSBuildVersion_
ReturnsTrue
1/1 killed Only asserts the current-environment `true` branch; no boundary case for the version/feature comparison. Add a below-17.10.0 version case (reset cached statics via reflection) to cover the `>=` boundary.
B (80–89) new MSBuildCompatibilityHelperTests.
SupportsTerminalLoggerWithExtendedMessages_
WithCurrentMSBuildVersion_
ReturnsTrue
1/1 killed Same single-branch limitation: only verifies the current pinned-version `true` outcome. Add a below-17.10.0 version case (reset cached statics via reflection) to cover the `>=` boundary.
A (90–100) new MSBuildCompatibilityHelperTests.
TryWriteExtendedMessage_
WhenSupported_
LogsExtendedMessageAndReturnsTrue
4/4 killed Verifies return value, logged event type, message, importance, and metadata reference identity. —

This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · 226.3 AIC · ⌖ 6.53 AIC · ⊞ 18K · ◷

@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Oct 6, 2026
@github-actions github-actions Bot removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Oct 6, 2026
@github-actions github-actions Bot added the state/approved Proposal approved; ready for implementation. label Oct 6, 2026
Add below, at, and above-threshold cases while isolating static caches, and verify unsupported versions do not log extended messages.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 6, 2026 11:20
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/approved Proposal approved; ready for implementation. labels Oct 6, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 6, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Oct 6, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The tests cover the intended behavior and safely restore shared state.

Review effort: Balanced
Findings: None

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #11767

🧵 Parallel safety

Test assembly Scope Workers Analyzer coverage
Microsoft.Testing.Platform.MSBuild.UnitTests MethodLevel ([assembly: Parallelize(Scope = ExecutionScope.MethodLevel, Workers = 0)]) 0 (CPU count) attribute opt-in; MSTEST0073–77 resource-lock analyzers not yet shipped on main

Status: FINDINGS (throughput-only, no live race)

Warning

  • [D · Medium confidence] test/UnitTests/Microsoft.Testing.Platform.MSBuild.UnitTests/MSBuildCompatibilityHelperTests.cs:14 — The assembly opts into MethodLevel parallelism, but the class-level [DoNotParallelize] pulls all 6 methods out of the parallel set and defers them to a serial tail after the rest of the assembly drains, instead of narrowly scoping mutual exclusion with [ResourceLock]. Confirmed via grep + reading InvokeTestingPlatformTaskTests.cs that no other class in the assembly calls into MSBuildCompatibilityHelper's cached-static path, so the class-level [DoNotParallelize] is not under-protected — it fully prevents any cross-test race on s_msBuildVersion / s_supportsMultiline / s_supportsTerminalLoggerWithExtendedMessages. This is a safety-vs-throughput tradeoff, not a correctness bug. Fix (optional): if this class's runtime becomes a measurable tail, replace the class-level attribute with a shared const resource key and [ResourceLock(...)] on each of the 6 methods.

No correctness or missing-protection issue found; the broad class-level exclusion is a safe, slightly conservative choice.

Per-test grades

GradeTestMutationNotesHow to improve
A (90–100) new MSBuildCompatibilityHelperTests.
SupportsMultiLine_
WithCurrentMSBuildVersion_
ReturnsTrue
1/1 killed Focused smoke check against the real loaded MSBuild assembly; boundary logic is delegated to the dedicated boundary test. —
A (90–100) new MSBuildCompatibilityHelperTests.
SupportsTerminalLoggerWithExtendedMessages_
WithCurrentMSBuildVersion_
ReturnsTrue
1/1 killed Focused smoke check against the real loaded MSBuild assembly; boundary logic is delegated to the dedicated boundary test. —
A (90–100) new MSBuildCompatibilityHelperTests.
SupportsMultiLine_
AtVersionBoundary_
ReturnsExpectedSupport
2/2 killed DataRow at exactly 17.10.0 kills both the `>=` boundary flip and the `&&` combinator mutation. —
A (90–100) new MSBuildCompatibilityHelperTests.
SupportsTerminalLoggerWithExtendedMessages_
AtVersionBoundary_
ReturnsExpectedSupport
1/1 killed DataRow at exactly 17.10.0 kills the `>=` boundary-flip mutation on the version comparison. —
A (90–100) new MSBuildCompatibilityHelperTests.
TryWriteExtendedMessage_
WhenSupported_
LogsExtendedMessageAndReturnsTrue
4/4 killed Asserts type, message, importance, and metadata identity — catches parameter swaps and importance/metadata mutations. —
A (90–100) new MSBuildCompatibilityHelperTests.
TryWriteExtendedMessage_
WhenUnsupported_
DoesNotLogAndReturnsFalse
2/2 killed Verifies both the `false` return and `Times.Never` on the mock, catching removal of the unsupported-path guard. —

All six new tests earned A. The reflection-based WithMSBuildVersion helper is appropriately scoped (resets all three cache fields and restores them in finally), and [DoNotParallelize] correctly prevents any cross-test race on the shared static caches — see the parallel-safety throughput note above, which is non-blocking.

This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · 164.9 AIC · ⌖ 8.64 AIC · ⊞ 18K · ◷

@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Oct 6, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Final test-triage resolution — Azure Pipelines build 1624754

This is the final resolution for the completed Windows Release/Debug and Linux Release build legs on this pull request (which only touches test/UnitTests/Microsoft.Testing.Platform.MSBuild.UnitTests/MSBuildCompatibilityHelperTests.cs, unrelated to the areas below).

Failure signatures identified (3 records, same two underlying tests):

  1. MSTest.Acceptance.IntegrationTests.WindowsUIAutomationSdkTests.EnableWindowsUIAutomation_WhenUsingMSTestRunner_RunsDesktopTests ("net10.0-windows") — Windows Release leg.

    • Root cause in the failing sub-test host: System.TimeoutException: Application 'C:\Windows\system32\charmap.exe' did not expose a window within 00:00:10 inside CharacterMapTests.WindowSetup, affecting both CharacterMap_MainWindow_IsVisible and CharacterMap_MainWindow_HasNonEmptyTitle.
    • Category: environmental one-off (UI-automation host app launch/window-activation timing), not a code defect introduced by this PR.
    • History: the last 4 completed builds in the lookback window (including two runs of the same sourceVersion as the build immediately preceding this PR's base) all show this exact test passing (durations 5.1s–8.8s). No prior occurrence of this timeout signature was found in the bounded 30-day/12-build history, so this does not meet the two-build recurrence threshold for a durable issue.
    • Confidence: moderate-high that this is a flaky Windows UI-automation launch timing issue (charmap.exe window activation), unrelated to the PR's MSBuild-helper unit test change.
  2. PackagedWinUI.PackagedWinUITestCases.RetryTestMethod_PassesOnSecondAppContainerHost — Windows App Model Diagnostics.

    • This is a by-design retry test: attempt 1 intentionally fails ("The first AppContainer host fails so Retry must activate a second host"), attempt 2 passes, and the final consolidated TRX result is passed. This matches the same pass-after-designed-retry pattern seen in every sampled history build. Not a real failure — no action needed.

Resolution: No issue is warranted. Both signals are a known-flaky UI-automation timing one-off and a test intentionally exercising retry behavior, neither tied to this PR's change nor recurring across the available history. No slowness regressions (slowRegressionCount: 0) or crash/hang diagnostics (diagnosticCount: 0) were found. Treat this build as clear for merge purposes regarding test-triage; no further action needed unless the UI-automation timeout recurs in a future build.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · 76.5 AIC · ⌖ 5.52 AIC · ⊞ 9.7K

@Evangelink
Amaury Levé (Evangelink) merged commit 21b6a7c into main Oct 7, 2026
37 of 40 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the copilot/test-improver-add-unit-tests-msbuildcompatibilityh branch October 7, 2026 09:24
@microsoft-github-policy-service microsoft-github-policy-service Bot removed the state/approved Proposal approved; ready for implementation. label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[test-improver] Add unit tests for MSBuildCompatibilityHelper

4 participants