Repository navigation
Improve readability of Azure DevOps skip-reason summaries - #12236
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🧪 Test quality review — PR #12236
Compared merge base 88a31ec394abe9f5c5dcf596f0a150ff998d3a0e with head ba0be6c97c7474dc8d2144177351d7040a43bb9b. Evidence applies to this head, not every intermediate commit.
Test-value evidence
All evidence is static prediction. No tests were executed and no paired before/after execution records were available.
| Contract | Scenario | Test / decisive assertion | Counterfactual | Evidence |
|---|---|---|---|---|
| Skip reasons render as list entries instead of a table | Single module, one or more reasons | BuildMarkdown_RendersTotalsAndTopFailingAndSlowest, BuildMarkdown_RendersTimedSkippedOutcome, SessionFinishing_GroupsSkipExplanationsWithoutListingSkippedNames, BuildAggregateMarkdown_SingleModuleSkipReasonsShareContext. Each compares the whole section with one exact raw-string equality. |
Restoring the table layout changes the section text, so these would fail. | Static prediction: likely detected |
| Multi-module summaries keep module and platform with each reason | Several modules, ties, attempts, the 10-group cap | BuildAggregateMarkdown_RendersOneGlobalTopTen..., ...SkipReasonsEscapeCellsAndPreserveLegacyMetadata, ...SkipReasonsCapsRowsAndAccountsForObservedOmissions, ...SkipReasonTiesUseModuleFrameworkAndAttemptIdentity. Each is an exact-equality assertion. |
Always hoisting shared context would drop the per-entry module lines and fail these. | Static prediction: likely detected |
| Escaping and long-reason wrapping | Reason containing <, &, ", |, and a long reason |
SingleModuleSkipReasonsShareContext, plus the acceptance test AssertStandaloneSummary (adds skipped names 3 and 4). |
Removing EscapeCell would change the output and fail the exact match. |
Static prediction: likely detected; the acceptance test needs -pack and was not run |
| PACKAGE.md wording | Documentation only | None | None | Not applicable |
Unassessed: nothing. Limitation: only static analysis was performed, so there are no executed red/green records.
🧵 Parallel safety
STATUS: PARTIAL. The specialist task was not launched in this run, so parallel-safety coverage is incomplete and this is not a clean result. By inspection, the changed tests call the pure static BuildMarkdown and BuildAggregateMarkdown methods with locally built inputs. The acceptance test change only adds published nodes to an existing generated fixture. No parallelization configuration changed (config_count is 0), and I found no new shared mutable state.
Per-test assessment
Pseudo-mutation evidence below consists of static predictions, not executed mutation results.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | A (90–100) | new AzureDevOpsSummaryReporterTests.BuildAggregateMarkdown_SingleModuleSkipReasonsShareContext |
An exact raw-string assertion covers the shared context, ordering, and escaping. | None |
| Pass | A (90–100) | mod AzureDevOpsSummaryReporterTests.BuildAggregateMarkdown_SkipReasonsEscapeCellsAndPreserveLegacyMetadata |
An exact whole-section equality assertion covers the multi-module layout. | None |
| Pass | A (90–100) | mod AzureDevOpsSummaryReporterTests.BuildAggregateMarkdown_SkipReasonsCapsRowsAndAccountsForObservedOmissions |
An exact equality assertion covers the cap and the omitted-test note. | None |
| Pass | A (90–100) | mod AzureDevOpsSummaryReporterTests.BuildAggregateMarkdown_SkipReasonTiesUseModuleFrameworkAndAttemptIdentity |
An exact equality assertion covers tie ordering and attempt identity. | None |
| Pass | B (80–89) | mod AzureDevOpsSummaryReporterTests.BuildMarkdown_RendersTotalsAndTopFailingAndSlowest |
A whole-markdown exact assertion covers the new layout; it is broad and brittle but protects the contract. | None |
| Pass | B (80–89) | mod AzureDevOpsSummaryReporterTests.BuildMarkdown_RendersTimedSkippedOutcome |
A whole-markdown exact assertion covers the shared-context skip section. | None |
| Pass | B (80–89) | mod AzureDevOpsSummaryReporterTests.BuildAggregateMarkdown_RendersOneGlobalTopTenWithOutcomesAndElapsedTimes |
A whole-markdown exact assertion covers the multi-module entry. | None |
| Pass | B (80–89) | mod AzureDevOpsSummaryReporterTests.SessionFinishing_GroupsSkipExplanationsWithoutListingSkippedNames |
An exact assertion covers grouping and the absence of skipped names. | None |
| Pass | B (80–89) | mod AzureDevOpsSummaryAcceptanceTests standalone-summary check (AssertStandaloneSummary helper, not a test method) |
A helper-level exact assertion over the packaged output covers escaping and long reasons. | None |
No inline findings met the publication gate.
<sub>This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~28 AIC · ⌖ ~1 AIC · ⊞ 8K
There was a problem hiding this comment.
🟡 Changes recommended
The new string.ReplaceLineEndings call is unavailable on the project’s Windows net462 and net472 targets, breaking compilation for both.
1 open issue
What changed in this PR
Improves skip-reason readability in Microsoft.Testing.Platform’s Azure DevOps summaries by replacing the wide table with wrapping lists. The separate HTML reporter is unchanged.
Changes:
- Adds bold skipped counts and shared context for single-module summaries, while retaining per-entry context for aggregates.
- Updates exact-output tests and adds long, escaped skip reasons to packed acceptance coverage.
- Documents the new presentation in the package readme.
Test-value evidence: For revision ba0be6c97c7474dc8d2144177351d7040a43bb9b, the assertions statically predict detection of the old table layout. The PR reports eight intended pre-fix failures, 37 passing corrected unit tests, and two passing packed acceptance tests; execution artifacts were not independently verified. Live Azure DevOps rendering remains unvalidated.
| File | Description |
|---|---|
| test/UnitTests/Microsoft.Testing.Extensions.UnitTests/AzureDevOpsSummaryReporterTests.cs | Updates list expectations and adds shared-context coverage. |
| test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/AzureDevOpsSummaryAcceptanceTests.cs | Extends packed-output coverage with long and escaped reasons. |
| src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/PACKAGE.md | Documents compact lists and context placement. |
| src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsSummaryReporter.Markdown.Skips.cs | Replaces the skip-reason table with Markdown lists. |
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🧪 Test quality review — PR #12236
Compared merge base 88a31ec394abe9f5c5dcf596f0a150ff998d3a0e with head 05616f9e643b3277ece10f3b556b4a0a986b6f63. Evidence applies to this head, not every intermediate commit.
Test-value evidence
All results are static predictions. No tests were run, and no paired executed records were available.
| Contract | Scenario and expected outcome | Decisive test and assertion | Counterfactual (static prediction) | Correct implementation |
|---|---|---|---|---|
| Single-module skip reasons share module and platform context once, as a list | One module with several reasons renders one **Test module:** ... · **Platform:** ... line, then one bullet per reason |
BuildAggregateMarkdown_SingleModuleSkipReasonsShareContext, an exact raw-string equality on the section |
Repeating the context per entry, or restoring the table, would change the string and fail. Likely killed (inferred). | Pass |
| Multi-module skip reasons keep module and platform context with each reason | Different modules, ties, attempts, and legacy fragments each keep their own context line | The updated escape, cap, and tie-ordering tests (exact section equality) | Dropping per-reason context or the blank-line separators would fail. Likely killed (inferred). | Pass |
| Markdown escaping and long reasons inside list items | <, |, and <br> survive in a bullet, and long text is not truncated |
The unit escape test and the acceptance AssertStandaloneSummary |
Escaping differences would change the exact text. Likely killed (inferred). | Pass |
PACKAGE.md wording |
Documentation only | n/a | Not applicable | n/a |
🧵 Parallel safety
STATUS: PARTIAL. The parallel-safety-reviewer specialist was not run, so there is no assembly row and no clean verdict.
By inspection, the changed tests compare pure string output. The acceptance change extends an existing generated asset by two published skipped tests. I found no new shared mutable state. This is not a substitute for the specialist's assembly analysis.
Per-test assessment
Pseudo-mutation evidence is a static prediction, not an executed mutation result.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | A (90–100) | new AzureDevOpsSummaryReporterTests.BuildAggregateMarkdown_SingleModuleSkipReasonsShareContext |
An exact raw-string assertion covers the shared context, the bullets, and the escaping. | None |
| Pass | A (90–100) | mod AzureDevOpsSummaryReporterTests.BuildMarkdown_RendersTotalsAndTopFailingAndSlowest |
The expected skip section now uses the shared-context list form. | None |
| Pass | A (90–100) | mod AzureDevOpsSummaryReporterTests.BuildAggregateMarkdown_RendersOneGlobalTopTenWithOutcomesAndElapsedTime |
The exact expectation covers the per-reason context line. | None |
| Pass | A (90–100) | mod AzureDevOpsSummaryReporterTests.BuildMarkdown_RendersTimedSkippedOutcome |
The exact expectation covers the shared-context list form. | None |
| Pass | A (90–100) | mod AzureDevOpsSummaryReporterTests.BuildAggregateMarkdown_SkipReasonsEscapeCellsAndPreserveLegacyMetadata |
The exact expectation covers escaping and legacy metadata, with per-reason context for two modules. | None |
| Pass | A (90–100) | mod AzureDevOpsSummaryReporterTests.BuildAggregateMarkdown_SkipReasonsCapsRowsAndAccountsForObservedOmissions |
The exact expectation covers the cap and the omitted-count note. | None |
| Pass | A (90–100) | mod AzureDevOpsSummaryReporterTests.BuildAggregateMarkdown_SkipReasonTiesUseModuleFrameworkAndAttemptIdentity |
The exact expectation covers tie ordering and attempt identity. | None |
| Pass | A (90–100) | mod AzureDevOpsSummaryReporterTests.SessionFinishing_GroupsSkipExplanationsWithoutListingSkippedNames |
The exact expectation covers grouping and the shared-context line. | None |
The acceptance test AzureDevOpsSummaryAcceptanceTests was changed only in its asset and expected markdown. It was not graded as a separate method.
No inline findings were raised. Existing-thread deduplication was not needed because nothing was published inline.
<sub>This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~27 AIC · ⌖ ~1 AIC · ⊞ 8K
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are identified for the Azure DevOps skip-reason formatting change, though execution artifacts and live rendering remain unverified.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
|
Final test-triage resolution — Azure Pipelines build 1667300 Result: inconclusive, no actionable test defect found. Build
Confidence: high that there is no product test failure in the collected evidence, medium overall because of the incomplete collection. No issue was created. If needed, re-run the pipeline to get complete artifact coverage.
|

Pull request
Summary
The Azure DevOps test summary puts long skip explanations in a four-column table that repeats the same module and platform on every row. Without useful table styling from the hosted Markdown renderer, the result is wide and difficult to scan.
Replace that table with wrapping list entries and bold counts. Single-module summaries show shared context once; aggregate entries retain their own module/framework/architecture/platform attribution. Ordering, the ten-group cap, omitted-test accounting, legacy metadata and exit semantics are unchanged. The separate
--report-htmlproduct is not affected.Related issue: N/A; this addresses the user-reported layout in the supplied screenshot.
Design and decision record
This is a bounded presentation fix, not a new feature. The rationale is recorded here and the presentation contract is documented in the package readme.
Changing header alignment alone would leave the repeated long metadata and width problem. Custom HTML/CSS would depend on Azure DevOps' sanitization and theme. Portable Markdown lists remove that dependency without changing report data, options, publishing or lifecycle behavior.
Demonstration
These are actual browser captures of Markdown emitted by the packed
3.0.0-devAzure DevOps reporter, rendered locally withConvertFrom-Markdownand neutral typography. They are not captures of the live Azure DevOps service or its theme.Desktop, 1200 CSS pixels: shared context and three distinguishable reason/count groups.
Constrained, 480 CSS pixels: long explanations and metadata wrap without clipped reason text; escaped markup, quotes, pipes and issue links remain intact.
The sample deliberately contains six results: one passed, one failed and four skipped, grouped into counts
2,1,1. Its child exit2is expected and asserted by the passing acceptance tests. The skip section also fits at 390 pixels, although the unchanged unbroken sample assembly heading elsewhere overflows by nine pixels in this neutral preview.Browser assertions checked count/reason associations, bounds, literal escaped text and keyboard focusability of issue links. Supplemental output from the packed aggregate formatter verified that metadata remains inside its associated reason's list item, missing legacy platform stays unavailable, and zero skips do not render an empty section. Reproduce the emitted summary with the packed standalone acceptance case below, then render its
summary.mdas Markdown. The original user capture supplies the before view; no live Azure DevOps publish was performed.Validation and regression proof
Executed before committing, with repository SDK
12.0.100-alpha.1.26509.101, runtime10.0.14, Release/net10.0/x64. The committed diff is byte-identical to the validated source patch. Both assemblies retained method-level parallelization (32 reported workers).BuildAggregateMarkdown_SingleModuleSkipReasonsShareContext,BuildMarkdown_RendersTimedSkippedOutcome,SessionFinishing_GroupsSkipExplanationsWithoutListingSkippedNamesAsyncBuildAggregateMarkdown_SkipReasonsEscapeCellsAndPreserveLegacyMetadata,BuildAggregateMarkdown_SkipReasonsCapsRowsAndAccountsForObservedOmissions,BuildAggregateMarkdown_SkipReasonTiesUseModuleFrameworkAndAttemptIdentityPackedStandalone_ReportsOutcomesSkipReasonsAndSummedTestTime,FixtureSdkWithoutRequiredArtifactSupport_PreservesPerModuleFallback: both passed and asserted the complete summary, expected child exit, and upload commandExecuted commands and results
$evidencedenotes the task-owned artifact directory;{}generates unique binlog names.Corrected formatter: build exit 0; 37 selected/executed/passed, 0 failed/skipped; runner exit 0. The same filter was independently discovered with
--list-tests.Shipping packages: both pack commands and repository content/metadata verification passed, exit 0. Both packages include netstandard2.0/net10.0/net11.0 assets. The build wrapper could not launch
powershellfor its post-pack check in this environment, so that same check was invoked explicitly; no build infrastructure was changed.Packed acceptance: build/discovery/runner exits 0; 2 discovered/selected/executed/passed, 0 failed/skipped. The existing execution gate independently verified both exact planned test names, fresh TRX data, unique executions and counters.
Here
$startedwas captured immediately before the acceptance execution and$testExitwas its actual exit code (0). The independently prepared expected-test file contained the two acceptance names above. The intentional failing child returned2; neither outer test failed.Both
Microsoft.Testing.PlatformandMicrosoft.Testing.Extensions.AzureDevOpsReportwere consumed as3.0.0-devfrom a fresh fixture cache.project.assets.json/deps.jsonwere inspected and the consumer's net10.0 DLL hashes matched the corresponding packed payloads:Raw TRX/binlogs and detailed provenance remain in the session artifacts, not public CI artifacts. The executed counters and failure attribution are included here rather than presenting inaccessible local paths as reviewer links.
Not run: CI or the live Azure DevOps hosted page/theme. These are local packed-product and browser-rendering results, not a hosted-UI or CI claim.
Test-value evidence
Tested base:
88a31ec394abe9f5c5dcf596f0a150ff998d3a0e. Committed source:ba0be6c97c7474dc8d2144177351d7040a43bb9b, matching validated patch SHA-256ADDEE019871D1B1DD481FDC624B666AF0DBEA4D255C622341C5C8D44CB708193.Executed: detected. All eight failures were intended exact-output assertions detecting the old four-column table, including
BuildAggregateMarkdown_SingleModuleSkipReasonsShareContext, not compilation, restore, launch, timeout or discovery failures. The unchanged original baseline also passed all 36 original tests.The retained test/support patch SHA-256 was
BCB6392792581EB24597B555AB05CA746CD530040C1D68FCC6114BCDC57D48D1. The same assertions and selector were used in both states; only production behavior differed. The red build and run were:red-final.trxrecords the eight intended failures;green.trxandacceptance.trxrecord the passing runs described above. The paired regression comparison is unit-level; packed acceptance is the additional green consumer check, not a claimed red/green package comparison.