Repository navigation
Add direct unit tests for MSBuildCompatibilityHelper - #11767
Amaury Levé (Evangelink) merged 4 commits into
Conversation
There was a problem hiding this comment.
Copilot wasn't able to review any files in this pull request. Check if the Files changed in this pull request are included in default exclusions.
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
There was a problem hiding this comment.
Note
🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.
Scope: Single new test file (40 lines), no production code touched. Test-only change, no public API impact.
Correctness & test quality
SupportsMultiLine()/SupportsTerminalLoggerWithExtendedMessages()assertions correctly reflect the pinned MSBuild package version (≥17.10), matching the version-gating logic inMSBuildCompatibilityHelper. The explanatory comment on the pinned-version assumption is a nice touch.TryWriteExtendedMessagetest follows AAA structure, usesMoq'sIBuildEngineappropriately (the only way to observeLogMessageEventwithout a real build engine), and verifies type, message, importance, andExtendedMetadatareference equality — good assertion diversity beyond a singleAssert.IsTrue.- Idioms (
Assert.ContainsSingle,Assert.IsInstanceOfType<T>) are consistent with sibling tests in this project (e.g.MSBuildLifecycleCallbacksTests.cs), andMoqusage is permitted here (BannedSymbols.txtonly bansAwesomeAssertions, not Moq). - Null-value metadata entry (
["empty"] = null) is a reasonable touch for exercising theDictionary<string, string?>nullable-value contract.
Coverage gap (non-blocking)
- Only the "supported" branches of the two feature-gate methods are exercised. The "unsupported" (older MSBuild /
TryWriteExtendedMessagereturningfalse) paths remain untested, but since the MSBuild package version is pinned centrally in this repo and can't be swapped per-test without faking the assembly'sAssemblyFileVersionAttribute, this is a reasonable scope limitation rather than an omission — flagging only for awareness, not requesting changes.
No blocking issues found. This is a solid, idiomatic addition that closes a real test gap (#11761).
There was a problem hiding this comment.
🧪 Test quality review — PR #11767
🧵 Parallel safety
Status: CLEAN
| Test assembly | Scope | Workers | Analyzer coverage |
|---|---|---|---|
| Microsoft.Testing.Platform.MSBuild.UnitTests | MethodLevel ([assembly: Parallelize(Scope = ExecutionScope.MethodLevel, Workers = 0)]) |
0 → CPU count (live concurrency) | Attribute-based opt-in; MSTEST0074-0077 analyzers not yet on main |
- i️ Info —
MSBuildCompatibilityHelperTests.cs:9-24: the three new tests populate shared statics (s_msBuildVersion,s_supportsMultiline,s_supportsTerminalLoggerWithExtendedMessages) inMSBuildCompatibilityHelpervia an unsynchronized??=pattern. Under this assembly'sMethodLevelparallelization these tests can race on initialization, but the computation is pure/deterministic (no env, CWD, or console dependency), so concurrent writers converge on the identical value — a benign race, not a correctness hazard. No other test in the assembly touches these statics today. No action needed.
No production source or parallelization-configuration files changed in this PR.
Per-test grades
| Grade | Test | Mutation | Notes | How to improve |
|---|---|---|---|---|
| B (80–89) | new MSBuildCompatibilityHelperTests. |
1/1 killed | Only asserts the current-environment `true` branch; no boundary case for the version/feature comparison. | Add a below-17.10.0 version case (reset cached statics via reflection) to cover the `>=` boundary. |
| B (80–89) | new MSBuildCompatibilityHelperTests. |
1/1 killed | Same single-branch limitation: only verifies the current pinned-version `true` outcome. | Add a below-17.10.0 version case (reset cached statics via reflection) to cover the `>=` boundary. |
| A (90–100) | new MSBuildCompatibilityHelperTests. |
4/4 killed | Verifies return value, logged event type, message, importance, and metadata reference identity. | — |
This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.
Warning
Firewall blocked 1 domain
The following domain was blocked by the firewall during workflow execution:
github.com
To allow these domains, add them to the network.allowed list in your workflow frontmatter:
network:
allowed:
- defaults
- "github.com"See Network Configuration for more information.
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · 226.3 AIC · ⌖ 6.53 AIC · ⊞ 18K · ◷
Add below, at, and above-threshold cases while isolating static caches, and verify unsupported versions do not log extended messages. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🧪 Test quality review — PR #11767
🧵 Parallel safety
| Test assembly | Scope | Workers | Analyzer coverage |
|---|---|---|---|
Microsoft.Testing.Platform.MSBuild.UnitTests |
MethodLevel ([assembly: Parallelize(Scope = ExecutionScope.MethodLevel, Workers = 0)]) |
0 (CPU count) | attribute opt-in; MSTEST0073–77 resource-lock analyzers not yet shipped on main |
Status: FINDINGS (throughput-only, no live race)
Warning
- [D · Medium confidence]
test/UnitTests/Microsoft.Testing.Platform.MSBuild.UnitTests/MSBuildCompatibilityHelperTests.cs:14— The assembly opts intoMethodLevelparallelism, but the class-level[DoNotParallelize]pulls all 6 methods out of the parallel set and defers them to a serial tail after the rest of the assembly drains, instead of narrowly scoping mutual exclusion with[ResourceLock]. Confirmed via grep + readingInvokeTestingPlatformTaskTests.csthat no other class in the assembly calls intoMSBuildCompatibilityHelper's cached-static path, so the class-level[DoNotParallelize]is not under-protected — it fully prevents any cross-test race ons_msBuildVersion/s_supportsMultiline/s_supportsTerminalLoggerWithExtendedMessages. This is a safety-vs-throughput tradeoff, not a correctness bug. Fix (optional): if this class's runtime becomes a measurable tail, replace the class-level attribute with a sharedconstresource key and[ResourceLock(...)]on each of the 6 methods.
No correctness or missing-protection issue found; the broad class-level exclusion is a safe, slightly conservative choice.
Per-test grades
| Grade | Test | Mutation | Notes | How to improve |
|---|---|---|---|---|
| A (90–100) | new MSBuildCompatibilityHelperTests. |
1/1 killed | Focused smoke check against the real loaded MSBuild assembly; boundary logic is delegated to the dedicated boundary test. | — |
| A (90–100) | new MSBuildCompatibilityHelperTests. |
1/1 killed | Focused smoke check against the real loaded MSBuild assembly; boundary logic is delegated to the dedicated boundary test. | — |
| A (90–100) | new MSBuildCompatibilityHelperTests. |
2/2 killed | DataRow at exactly 17.10.0 kills both the `>=` boundary flip and the `&&` combinator mutation. | — |
| A (90–100) | new MSBuildCompatibilityHelperTests. |
1/1 killed | DataRow at exactly 17.10.0 kills the `>=` boundary-flip mutation on the version comparison. | — |
| A (90–100) | new MSBuildCompatibilityHelperTests. |
4/4 killed | Asserts type, message, importance, and metadata identity — catches parameter swaps and importance/metadata mutations. | — |
| A (90–100) | new MSBuildCompatibilityHelperTests. |
2/2 killed | Verifies both the `false` return and `Times.Never` on the mock, catching removal of the unsupported-path guard. | — |
All six new tests earned A. The reflection-based WithMSBuildVersion helper is appropriately scoped (resets all three cache fields and restores them in finally), and [DoNotParallelize] correctly prevents any cross-test race on the shared static caches — see the parallel-safety throughput note above, which is non-blocking.
This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · 164.9 AIC · ⌖ 8.64 AIC · ⊞ 18K · ◷
|
Final test-triage resolution — Azure Pipelines build 1624754 This is the final resolution for the completed Windows Release/Debug and Linux Release build legs on this pull request (which only touches Failure signatures identified (3 records, same two underlying tests):
Resolution: No issue is warranted. Both signals are a known-flaky UI-automation timing one-off and a test intentionally exercising retry behavior, neither tied to this PR's change nor recurring across the available history. No slowness regressions (
|
MSBuildCompatibilityHelperguards MSBuild APIs newer than the oldest supported SDK, but previously had no direct tests.ExtendedBuildMessageEventArgswith the expected type, message, importance, and metadata.