Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -34,19 +34,35 @@ .. modules
];

builder.Append(new string('#', headingLevel)).Append(" Skipped tests by reason\n\n");
builder.Append("| Skipped | Reason | Platform | Test module |\n");
builder.Append("| ---: | --- | --- | --- |\n");
foreach ((CiRunSummaryModule module, CiRunSummarySkipReason reason) in reasons)
bool hasSharedContext = modules.Count == 1;
if (hasSharedContext)
{
builder.Append("| ").Append(reason.Count.ToString(CultureInfo.InvariantCulture))
.Append(" | ").Append(EscapeCell(reason.Reason))
.Append(" | ").Append(EscapeCell(module.OperatingSystem ?? "Unavailable"))
.Append(" | ");
CiRunSummaryModule module = modules[0];
builder.Append("**Test module:** ");
AppendModuleIdentity(builder, modules, module);
builder.Append(" |\n");
builder.Append(" \u00b7 **Platform:** ").Append(EscapeCell(module.OperatingSystem ?? "Unavailable"))
.Append("\n\n");
}

foreach ((CiRunSummaryModule module, CiRunSummarySkipReason reason) in reasons)
{
builder.Append("- **").Append(reason.Count.ToString(CultureInfo.InvariantCulture))
.Append(" skipped** \u2014 ").Append(EscapeCell(reason.Reason));
if (!hasSharedContext)
{
builder.Append("\n\n **Test module:** ");
AppendModuleIdentity(builder, modules, module);
builder.Append(" \u00b7 **Platform:** ").Append(EscapeCell(module.OperatingSystem ?? "Unavailable"));
}

builder.Append(hasSharedContext ? "\n" : "\n\n");
}

if (hasSharedContext)
{
builder.Append('\n');
}

builder.Append('\n');
long omittedTests = skippedTests - reasons.Sum(static item => item.Reason.Count);
if (omittedTests > 0)
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ When the SDK supplies an authoritative final invocation exit, the aggregate show

Skipped tests are grouped by their reported reason and execution platform, with module/target-framework context and at most ten groups shown. Issue references in reasons are retained; skipped test names are not dumped into the summary. Omitted groups are accounted for by a skipped-test count. Older summary fragments remain supported and label unavailable outcomes, skip reasons, and platform metadata explicitly.

Skip-reason groups use compact list entries so long explanations can wrap without competing with metadata columns. A single-module summary shows the shared module and platform once; multi-module summaries keep that context with each reason.

## Usage

Enable Azure DevOps reporting with the `--report-azdo` command line option.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -112,15 +112,17 @@ private static void AssertStandaloneSummary(string markdown)
$"""
## ❌ AzureDevOpsSummaryBehavior (net8.0)

**4 tests** · ✅ **1 passed** · ❌ **1 failed** · ⏭ **2 skipped** · ⏱ Test time: **17.50s** · exit code **2** (AtLeastOneTestFailed — At least one test failed.)
**6 tests** · ✅ **1 passed** · ❌ **1 failed** · ⏭ **4 skipped** · ⏱ Test time: **17.50s** · exit code **2** (AtLeastOneTestFailed — At least one test failed.)

> Test time is the sum of reported test durations, not elapsed time; tests may run concurrently.

### Skipped tests by reason

| Skipped | Reason | Platform | Test module |
| ---: | --- | --- | --- |
| 2 | Known issue https://example.test/issues/42 | {operatingSystem} | AzureDevOpsSummaryBehavior (net8.0, {architecture}) |
**Test module:** AzureDevOpsSummaryBehavior (net8.0, {architecture}) · **Platform:** {operatingSystem}

- **2 skipped** — Known issue https://example.test/issues/42
- **1 skipped** — Awaiting <package> & "native" support \| see https://example.test/issues/314
- **1 skipped** — Packaged Windows apps are a Windows-only scenario; the launcher is intentionally disabled on other platforms so that unsupported activation is not attempted.

### Top failing classes

Expand Down Expand Up @@ -202,6 +204,12 @@ await PublishAsync(context, "PrivateSkippedName1",
new SkippedTestNodeStateProperty(" Known issue\nhttps://example.test/issues/42 "), TimeSpan.Zero);
await PublishAsync(context, "PrivateSkippedName2",
new SkippedTestNodeStateProperty("Known issue https://example.test/issues/42"), TimeSpan.Zero);
await PublishAsync(context, "PrivateSkippedName3",
new SkippedTestNodeStateProperty("Awaiting <package> & \"native\" support | see https://example.test/issues/314"),
TimeSpan.Zero);
await PublishAsync(context, "PrivateSkippedName4",
new SkippedTestNodeStateProperty("Packaged Windows apps are a Windows-only scenario; the launcher is intentionally disabled on other platforms so that unsupported activation is not attempted."),
TimeSpan.Zero);
context.Complete();
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,9 @@ public void BuildMarkdown_RendersTotalsAndTopFailingAndSlowest()

### Skipped tests by reason

| Skipped | Reason | Platform | Test module |
| ---: | --- | --- | --- |
| 1 | Reason not provided | {GetOperatingSystem()} | MyAssembly (net8.0, unknown) |
**Test module:** MyAssembly (net8.0, unknown) · **Platform:** {GetOperatingSystem()}

- **1 skipped** — Reason not provided

### Top failing classes

Expand Down Expand Up @@ -1026,9 +1026,9 @@ public void BuildAggregateMarkdown_RendersOneGlobalTopTenWithOutcomesAndElapsedT

### Skipped tests by reason

| Skipped | Reason | Platform | Test module |
| ---: | --- | --- | --- |
| 1 | Tracked issue | Linux | Beta (net8.0, x64) |
- **1 skipped** — Tracked issue

**Test module:** Beta (net8.0, x64) · **Platform:** Linux

### Test modules

Expand Down Expand Up @@ -1218,9 +1218,9 @@ public void BuildMarkdown_RendersTimedSkippedOutcome()

### Skipped tests by reason

| Skipped | Reason | Platform | Test module |
| ---: | --- | --- | --- |
| 1 | Known issue | {GetOperatingSystem()} | Tests (net8.0, unknown) |
**Test module:** Tests (net8.0, unknown) · **Platform:** {GetOperatingSystem()}

- **1 skipped** — Known issue

> Duration history unavailable for the observed tests; current timings are not a historical baseline.

Expand Down Expand Up @@ -1268,10 +1268,55 @@ public void BuildAggregateMarkdown_SkipReasonsEscapeCellsAndPreserveLegacyMetada
"""
### Skipped tests by reason

| Skipped | Reason | Platform | Test module |
| ---: | --- | --- | --- |
| 3 | Skip reason unavailable (older fragment) | Unavailable | Legacy (net8.0, arm64) |
| 2 | https://example.test/issues/42 \| \`reason\` &lt;pending&gt;&amp;<br>next | Linux\|&lt;runner&gt; | A\|&lt;&amp;&gt; (net8.0, x64) |
- **3 skipped** — Skip reason unavailable (older fragment)

**Test module:** Legacy (net8.0, arm64) · **Platform:** Unavailable

- **2 skipped** — https://example.test/issues/42 \| \`reason\` &lt;pending&gt;&amp;<br>next

**Test module:** A\|&lt;&amp;&gt; (net8.0, x64) · **Platform:** Linux\|&lt;runner&gt;


""".Replace("\r\n", "\n"),
GetSection(markdown, "### Skipped tests by reason"));
}

[TestMethod]
public void BuildAggregateMarkdown_SingleModuleSkipReasonsShareContext()
{
var module = new CiRunSummaryModule
{
AssemblyName = "Microsoft.Testing.Extensions.UnitTests",
TargetFramework = "net10.0",
Architecture = "x64",
OperatingSystem = "Linux",
TotalTests = 3,
SkippedTests = 3,
SkipReasons =
[
new CiRunSummarySkipReason
{
Reason = "Packaged Windows apps are a Windows-only scenario; the launcher is intentionally disabled elsewhere.",
Count = 2,
},
new CiRunSummarySkipReason
{
Reason = "Awaiting <package> & \"native\" support | see https://example.test/issues/314",
Count = 1,
},
],
};

string markdown = AzureDevOpsSummaryReporter.BuildAggregateMarkdown(CreateObservedAggregate(module));

Assert.AreEqual(
"""
### Skipped tests by reason

**Test module:** Microsoft.Testing.Extensions.UnitTests (net10.0, x64) · **Platform:** Linux

- **2 skipped** — Packaged Windows apps are a Windows-only scenario; the launcher is intentionally disabled elsewhere.
- **1 skipped** — Awaiting &lt;package&gt; &amp; &quot;native&quot; support \| see https://example.test/issues/314


""".Replace("\r\n", "\n"),
Expand Down Expand Up @@ -1314,18 +1359,45 @@ public void BuildAggregateMarkdown_SkipReasonsCapsRowsAndAccountsForObservedOmis
"""
### Skipped tests by reason

| Skipped | Reason | Platform | Test module |
| ---: | --- | --- | --- |
| 3 | Primary | macOS | Beta (net9.0, arm64) |
| 2 | Reason0 | Windows | Alpha (net8.0, x64) |
| 2 | Reason1 | Windows | Alpha (net8.0, x64) |
| 2 | Reason2 | Windows | Alpha (net8.0, x64) |
| 2 | Reason3 | Windows | Alpha (net8.0, x64) |
| 2 | Reason4 | Windows | Alpha (net8.0, x64) |
| 2 | Reason5 | Windows | Alpha (net8.0, x64) |
| 2 | Reason6 | Windows | Alpha (net8.0, x64) |
| 2 | Reason7 | Windows | Alpha (net8.0, x64) |
| 2 | Reason8 | Windows | Alpha (net8.0, x64) |
- **3 skipped** — Primary

**Test module:** Beta (net9.0, arm64) · **Platform:** macOS

- **2 skipped** — Reason0

**Test module:** Alpha (net8.0, x64) · **Platform:** Windows

- **2 skipped** — Reason1

**Test module:** Alpha (net8.0, x64) · **Platform:** Windows

- **2 skipped** — Reason2

**Test module:** Alpha (net8.0, x64) · **Platform:** Windows

- **2 skipped** — Reason3

**Test module:** Alpha (net8.0, x64) · **Platform:** Windows

- **2 skipped** — Reason4

**Test module:** Alpha (net8.0, x64) · **Platform:** Windows

- **2 skipped** — Reason5

**Test module:** Alpha (net8.0, x64) · **Platform:** Windows

- **2 skipped** — Reason6

**Test module:** Alpha (net8.0, x64) · **Platform:** Windows

- **2 skipped** — Reason7

**Test module:** Alpha (net8.0, x64) · **Platform:** Windows

- **2 skipped** — Reason8

**Test module:** Alpha (net8.0, x64) · **Platform:** Windows

> 9 additional skipped tests are not shown.

Expand Down Expand Up @@ -1358,12 +1430,21 @@ public void BuildAggregateMarkdown_SkipReasonTiesUseModuleFrameworkAndAttemptIde
"""
### Skipped tests by reason

| Skipped | Reason | Platform | Test module |
| ---: | --- | --- | --- |
| 1 | Same reason | Windows | A (net8.0, x64) — attempt 1, session session-1 |
| 1 | Same reason | Windows | A (net8.0, x64) — attempt 2, session session-2 |
| 1 | Same reason | Windows | A (net9.0, x64) |
| 1 | Same reason | Windows | B (net8.0, x64) |
- **1 skipped** — Same reason

**Test module:** A (net8.0, x64) — attempt 1, session session-1 · **Platform:** Windows

- **1 skipped** — Same reason

**Test module:** A (net8.0, x64) — attempt 2, session session-2 · **Platform:** Windows

- **1 skipped** — Same reason

**Test module:** A (net9.0, x64) · **Platform:** Windows

- **1 skipped** — Same reason

**Test module:** B (net8.0, x64) · **Platform:** Windows


""".Replace("\r\n", "\n"),
Expand Down Expand Up @@ -1402,10 +1483,10 @@ public async Task SessionFinishing_GroupsSkipExplanationsWithoutListingSkippedNa

### Skipped tests by reason

| Skipped | Reason | Platform | Test module |
| ---: | --- | --- | --- |
| 2 | Known issue | {GetOperatingSystem()} | MyAssembly ({targetFramework}, {RuntimeInformation.ProcessArchitecture.ToString().ToLowerInvariant()}) |
| 1 | Reason not provided | {GetOperatingSystem()} | MyAssembly ({targetFramework}, {RuntimeInformation.ProcessArchitecture.ToString().ToLowerInvariant()}) |
**Test module:** MyAssembly ({targetFramework}, {RuntimeInformation.ProcessArchitecture.ToString().ToLowerInvariant()}) · **Platform:** {GetOperatingSystem()}

- **2 skipped** — Known issue
- **1 skipped** — Reason not provided

> Duration history unavailable for the observed tests; current timings are not a historical baseline.

Expand Down
Loading