Skip to content

[rel/4.5] Improve readability of Azure DevOps skip summaries - #12246

Merged
Amaury Levé (Evangelink) merged 1 commit into
rel/4.5from
evangelink-4-5-backport-candidates
Nov 6, 2026
Merged

Amaury Levé (Evangelink) merged 1 commit into
rel/4.5from
evangelink-4-5-backport-candidates

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Pull request

Summary

Backport #12236 to rel/4.5. Replace the wide Azure DevOps skip-reason table with wrapping list entries and bold counts. Single-module summaries show shared module/platform context once; aggregates keep that context with each reason.

Preserve sorting, the ten-group cap, omitted-test accounting, legacy unavailable metadata, result identities, publishing and exit semantics. This is a four-file backport: formatter, package documentation, exact-output unit tests and packed acceptance coverage. Keep the release's net8.0 consumer; do not import main's framework, dependency or API changes.

Design and decision record

The existing presentation-fix rationale and alternatives are in #12236; no new feature/design record is needed. The production formatter is byte-identical to the merged source (Git blob f3b508d625f6fb5f3eebe794c6e89bcac33a4fcb). The only cherry-pick conflict was the packed expectation's framework label, retained as net8.0.

  • Release base: e8db9f8d499b4d675fe176187ee483f8ffdd2b9d.
  • Source merge: 32077b2456a3c5f25ca2e96a5add96e0c4d03f74.
  • Backport/tested head: 770eba45575f97930dcfbb92ec8ab07f54c1e1ab.
  • Complete backport patch SHA-256: CB4F6B624AFD0200007D272DC502679C94258BE674CFBEF7BAA4A93FB90472DC.

Demonstration

Independently rendered the actual packed 2.5.3-dev reporter's emitted Markdown in a local browser with neutral typography at 1200px and 480px iframe viewports. Visual inspection and DOM checks confirmed readable wrapping, no horizontal overflow, correct count/reason associations, shared context, literal escaped markup/quotes and two focusable issue links. These are local release-product checks, not the live Azure DevOps service/theme. The source PR contains the original illustrative captures; the excerpt below is from the release package output:

Test module: AzureDevOpsSummaryBehavior (net8.0, x64) · Platform: Windows
2 skipped — Known issue https://example.test/issues/42
1 skipped — Awaiting <package> & "native" support | see https://example.test/issues/314
1 skipped — Packaged Windows apps are a Windows-only scenario; ...

The full output contains six results: one passed, one failed and four skipped, grouped as 2, 1, 1. Both passing outer acceptance cases assert the intentional child's exit 2 and the complete unabridged Markdown. No external publishing was performed. Aggregate/legacy/empty-output contracts are protected by unit assertions; their hosted visual presentation was not exercised.

Validation and regression proof

Windows x64, Release, repository SDK 12.0.100-alpha.1.26480.103. Unit host: net8.0 on runtime 8.0.33. Acceptance host: repository-default net11.0, using its SDK-selected 12.0.0-alpha.1.26480.103 runtime; actual packaged consumer: net8.0. Both assemblies retained MethodLevel scheduling with 32 workers. No scheduling, SDK, package versions or source configuration were changed.

State/check Executed result
Unchanged release baseline 35 discovered/executed/passed, 0 failed/skipped; exit 0.
Corrected formatter 36 discovered/executed/passed, 0 failed/skipped; exit 0.
Actual old formatter, final tests retained, isolated outputs 36 discovered/executed: 28 passed, 8 intended exact-output failures, 0 skipped; exit 2.
Correct implementation restored Same 36/36 passed, 0 failed/skipped; exit 0; production bytes equal the committed fix.
Fresh MTP and AzureDevOpsReport packs Both build/pack commands and package content/metadata verification passed, 0 warnings/errors. Both retain netstandard2.0/net8.0/net9.0 assets.
Packed standalone and unsupported-SDK fallback 2 independently discovered/executed/passed, 0 failed/skipped; outer exit 0.

TRX, JUnit and CTRF identities/counts/outcomes reconcile; JUnit's existing DataRow method/attempt aliases were mapped to their source-declared cases. The green execution verifier also checked freshness, exact planned names, unique execution IDs and counters.

Requirement coverage

Requirement Evidence
Compact output, long/escaped text and shared context BuildAggregateMarkdown_SingleModuleSkipReasonsShareContext, BuildMarkdown_RendersTimedSkippedOutcome, SessionFinishing_GroupsSkipExplanationsWithoutListingSkippedNamesAsync.
Aggregate identity, ordering, legacy metadata and omission accounting BuildAggregateMarkdown_SkipReasonTiesUseModuleFrameworkAndAttemptIdentity, BuildAggregateMarkdown_SkipReasonsEscapeCellsAndPreserveLegacyMetadata, BuildAggregateMarkdown_SkipReasonsCapsRowsAndAccountsForObservedOmissions.
Real shipping output and old-SDK fallback PackedStandalone_ReportsOutcomesSkipReasonsAndSummedTestTime, FixtureSdkWithoutRequiredArtifactSupport_PreservesPerModuleFallback; both assert the final complete summary and upload command. The latter is explicitly a protocol fixture, not a native SDK invocation.

Commands below preserve the executed flags; $evidence denotes the task-owned evidence directory rather than a machine-specific absolute prefix. Native Windows PowerShell was launched with process-only PATH entries for .dotnet, Windows PowerShell and System32; no user/global settings changed.

$unit = 'test\UnitTests\Microsoft.Testing.Extensions.UnitTests\Microsoft.Testing.Extensions.UnitTests.csproj'
$acceptance = 'test\IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests\Microsoft.Testing.Platform.Acceptance.IntegrationTests.csproj'

# Baseline and green: each exit 0, 0 warnings/errors; distinct binlogs per state.
.\eng\build.ps1 -restore -build -configuration Release -projects $unit -msbuildEngine dotnet -nodeReuse:$false "/bl:$evidence\skip-green-build-{}.binlog" /p:TargetFrameworks=net8.0 /p:SkipUapTargetFramework=true /p:EnableCodeCoverage=false

# Counterfactual: exit 0, 0 warnings/errors; only the formatter restored to its actual old contents.
.\eng\build.ps1 -restore -build -configuration Release -projects $unit -msbuildEngine dotnet -nodeReuse:$false "/bl:$evidence\skip-red-build-{}.binlog" /p:TargetFrameworks=net8.0 /p:SkipUapTargetFramework=true /p:EnableCodeCoverage=false "/p:ArtifactsBinDir=$PWD\artifacts\bin\skip-red\" "/p:ArtifactsObjDir=$PWD\artifacts\obj\skip-red\"

# After restoring the correct source: each exit 0, 0 warnings/errors, verifier passed.
.\eng\build.ps1 -restore -build -pack -configuration Release -projects src\Platform\Microsoft.Testing.Platform\Microsoft.Testing.Platform.csproj -msbuildEngine dotnet -nodeReuse:$false "/bl:$evidence\skip-core-pack-{}.binlog" /p:EnableCodeCoverage=false
.\eng\build.ps1 -restore -build -pack -configuration Release -projects src\Platform\Microsoft.Testing.Extensions.AzureDevOpsReport\Microsoft.Testing.Extensions.AzureDevOpsReport.csproj -msbuildEngine dotnet -nodeReuse:$false "/bl:$evidence\skip-azdo-pack-{}.binlog" /p:EnableCodeCoverage=false
.\eng\build.ps1 -restore -build -configuration Release -projects $acceptance -msbuildEngine dotnet -nodeReuse:$false "/bl:$evidence\skip-acceptance-build-{}.binlog" /p:SkipUapTargetFramework=true /p:EnableCodeCoverage=false

# Separate discovery: exit 0, 36 unit cases and 2 acceptance cases.
.\.dotnet\dotnet.exe artifacts\bin\Microsoft.Testing.Extensions.UnitTests\Release\net8.0\Microsoft.Testing.Extensions.UnitTests.dll --list-tests --filter 'FullyQualifiedName~Microsoft.Testing.Extensions.UnitTests.AzureDevOpsSummaryReporterTests' --no-ansi
.\.dotnet\dotnet.exe artifacts\bin\Microsoft.Testing.Platform.Acceptance.IntegrationTests\Release\net11.0\Microsoft.Testing.Platform.Acceptance.IntegrationTests.dll --list-tests --filter 'FullyQualifiedName~AzureDevOpsSummaryAcceptanceTests' --no-ansi

# Green/restored green: exit 0, 36/36 passed. The baseline uses its previously built assembly (35/35).
.\.dotnet\dotnet.exe artifacts\bin\Microsoft.Testing.Extensions.UnitTests\Release\net8.0\Microsoft.Testing.Extensions.UnitTests.dll --filter 'FullyQualifiedName~Microsoft.Testing.Extensions.UnitTests.AzureDevOpsSummaryReporterTests' --report-trx --report-trx-filename tests.trx --report-junit --report-junit-filename tests.junit.xml --report-ctrf --report-ctrf-filename tests.ctrf.json --results-directory $unitResults --show-test-results all --no-ansi

# Red: identical selector/assertions/report flags; exit 2, the eight intended failures.
.\.dotnet\dotnet.exe artifacts\bin\skip-red\Microsoft.Testing.Extensions.UnitTests\Release\net8.0\Microsoft.Testing.Extensions.UnitTests.dll --filter 'FullyQualifiedName~Microsoft.Testing.Extensions.UnitTests.AzureDevOpsSummaryReporterTests' --report-trx --report-trx-filename tests.trx --report-junit --report-junit-filename tests.junit.xml --report-ctrf --report-ctrf-filename tests.ctrf.json --results-directory $redResults --show-test-results all --no-ansi

# Packed release consumer: exit 0, 2/2 passed; both expected child exits are 2.
.\.dotnet\dotnet.exe artifacts\bin\Microsoft.Testing.Platform.Acceptance.IntegrationTests\Release\net11.0\Microsoft.Testing.Platform.Acceptance.IntegrationTests.dll --filter 'FullyQualifiedName~AzureDevOpsSummaryAcceptanceTests' --report-trx --report-trx-filename acceptance.trx --report-junit --report-junit-filename acceptance.junit.xml --report-ctrf --report-ctrf-filename acceptance.ctrf.json --results-directory $acceptanceResults --show-test-results all --no-ansi

Package provenance: the generated consumer's project.assets.json SHA-512 entries match the fresh archives, and both executed lib/net8.0 DLL SHA-256 values match those archive entries. The fixture used its unique per-run package cache, not a reused global cache. Archive SHA-256:

  • Microsoft.Testing.Platform: 4476A92F43B978D748337212372581090EF5C2830E3C8B4C38BF984C513B6196.
  • Microsoft.Testing.Extensions.AzureDevOpsReport: C5FF60798F1C14594FEF0F4B89C90D11695680476B2236699EF47DFB0DDE96E0.

Test-value evidence

Executed: detected. On tested head 770eba45575f97930dcfbb92ec8ab07f54c1e1ab, retained all final tests while replacing only the formatter with the byte-exact release-baseline blob 8e4e2f5e0ec2e87f9e251759b5045ccc570b5b5a. Counterfactual patch SHA-256: BCFBA2C2CA5764B782669E81D6BA219162506A7B70C2AE58179EB709B8BC84BE.

The same 36-case selection failed eight exact-output assertions with the old four-column table, including BuildAggregateMarkdown_SingleModuleSkipReasonsShareContext, then passed all 36 with the correct implementation. Unit and acceptance test source hashes remained unchanged between states. Failures were executed string-equality assertions, not compilation, restore, discovery, launch, timeout or skipped-test signals. Red intermediates/binaries were isolated under artifacts\obj\skip-red and artifacts\bin\skip-red.

This paired proof is at the formatter unit boundary; packed acceptance is additional green consumer proof, not a claimed red/green package comparison. Reports, binlogs, patches, package hashes, consumer assets/config and rendered output are retained in the task's session artifacts; they were not uploaded.

Not claimed: remote CI, Linux/macOS execution, a full-suite run, or the live Azure DevOps hosted page/theme. Static pipeline inspection confirms the normal Windows Release solution test step includes both assemblies when SkipTests=false; CI execution for this published head is not yet proven.

Backport #12236 while retaining the release net8.0 packed-consumer expectations. Preserve ordering, omission counts, legacy metadata and lifecycle behavior.

(cherry picked from commit 32077b2)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added the state/needs-review Awaiting review from the team. label Nov 6, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Scope: the markdown rendering of the Skipped tests by reason section in the AzureDevOpsReport extension. I also checked PACKAGE.md, the acceptance test and the unit tests. I found no blocking issues. No public API, IPC, resource or localization files are touched.

Checked

  • The single-module path (modules.Count == 1) prints the shared context once. The multi-module path prints per-entry context in an indented paragraph. Blank-line handling is consistent in both: the single-module list is tight, and the multi-module list is loose with each entry indented two spaces.
  • The omitted-tests accounting after the list is unchanged.
  • The unit tests cover both layouts, including the cap/omission case and the tie-break ordering by module, framework and attempt. The new BuildAggregateMarkdown_SingleModuleSkipReasonsShareContext test uses a raw string literal for the exact multiline comparison, as the repo guidelines ask.
  • The acceptance test adds a reason containing <, &, " and |, and a long reason, so the escaping and wrapping behavior is exercised end to end.

Non-blocking observations

  1. The reason and module identity are still escaped with the table-cell helper EscapeCell. That helper escapes | as \| and turns newlines into <br>, which only matters inside a table cell. The output is harmless in a list item (\| renders as |), and the tests pin it. A follow-up could rename the helper or split it into a general inline-text escape, so the name no longer misleads.
  2. The skip reasons are user-controlled. EscapeCell does not escape *, _ or `, so a reason with an unbalanced ** could change the formatting of the rest of the bullet. The table had the same exposure, so this is not a regression.
  3. The PACKAGE.md update accurately describes the new layout.

I did not build or run the tests; this review is from the diff only.

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. labels Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Preliminary test-triage analysis — Azure Pipelines build 1667703

Early analysis of the failed leg microsoft.testfx (Build Windows Debug). Other build legs are still running and may change this conclusion.

  • Failed test: SimplifiedConsoleOutputDeviceTests.CooperativeReporter_WhenTestIsDue_EmitsSlowDiagnostic in Microsoft.Testing.Platform.UnitTests on net9.0. It failed after about 5.5s.
  • Signature: Assert.AreSame(device.MessageReported, completedTask) at SimplifiedConsoleOutputDeviceTests.cs:406. The Task.WhenAny result was the Task.Delay timeout task, not the MessageReported task, so the slow diagnostic was not emitted before the timeout.
  • Other runs: the same test passed in about 7–11ms in the other reports in this leg, so it is not a stable assertion failure there.
  • Retry and history: no retry attempts were recorded. History is incomplete because this is an early analysis, so a prior flake can't be ruled out.
  • Assessment: the evidence fits a timing-sensitive race or a loaded agent. Confidence is low to medium. The PR is a rel/4.5 change to Azure DevOps skip summaries, and nothing here ties it to the failure.
  • Next step: re-run the leg. If it recurs, check how the test's cooperative reporter schedules its due-time check against the timeout. A recurrence across builds would justify a flaky-test issue; no issue was created.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · ~26 AIC · ⌖ ~1 AIC · ⊞ 10K

@Evangelink
Amaury Levé (Evangelink) merged commit a39964c into rel/4.5 Nov 6, 2026
33 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the evangelink-4-5-backport-candidates branch November 6, 2026 17:10
@github-actions github-actions Bot removed the state/approved Proposal approved; ready for implementation. label Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants