Skip to content

Add argument-only display names for data rows - #11775

Merged
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
dev/amauryleve/data-row-naming
Oct 7, 2026
Merged

Amaury Levé (Evangelink) merged 2 commits into
mainfrom
dev/amauryleve/data-row-naming

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Custom data-row DisplayName values currently replace the entire test name, making rows with the same label on different methods indistinguishable. Add an explicit opt-in ArgumentsDisplayName property so a row can render as Parse_ReturnsNull (empty input) without repeating the method name or changing existing full-name overrides.

Fixes #11770

Approach

  • Add nullable get/set properties to DataRowAttribute and TestDataRow<T>, reuse the localized default name format, and honor TestMethod.DisplayName in both discovery and folded execution.
  • Preserve existing DisplayName precedence and null/whitespace behavior. Argument labels are verbatim text, not templates; null or blank labels retain existing naming behavior. Dynamic row labels take precedence over source-level naming callbacks and require an updated adapter.
  • Add enabled-by-default MSTEST0089 for C#/VB attribute and object initializers whose constant full and argument names conflict. Unknown values, generated code, and custom derived attributes are not diagnosed. Removing either property is the documented manual remedy; runtime precedence is unchanged.
  • Update API baselines, package/RFC documentation, and analyzer resources. XLF changes were generated with the owning project's UpdateXlf target.

Validation

Passed with the pinned repository SDK 12.0.100-alpha.1.26480.103, packed MSTest components 4.5.0-dev, and Microsoft.Testing.Platform 2.5.0-dev:

  • Complete Windows build/pack and NuGet layout validation: 0 warnings, 0 errors.
  • Framework naming regressions: 38 passed; adapter metadata/execution regressions: 10 passed.
  • New conflict and existing duplicate-name analyzer coverage: 43 passed on net8.0; conflict coverage: 31 passed on net472.
  • Packed-consumer acceptance coverage: 11 passed. Real MTP/VSTest processes cover net462/net8/net10, source-generated metadata, exact display names and outcomes, TestContext names, ignored rows, categories, and VSTest serialization fallback. Framework-only consumers explicitly enable analyzers, receive MSTEST0089 through the transitive package dependency, and compile cleanly after removing either conflicting property.
Reproducible validation commands (all exited successfully)

Run from the repository root after building. Acceptance assets restore fresh copies of the packed packages.

.\build.cmd -pack -bl
.\.dotnet\dotnet.exe msbuild src\Analyzers\MSTest.Analyzers\MSTest.Analyzers.csproj /t:UpdateXlf '/bl:artifacts\log\Debug\ConflictingNames.UpdateXlf.{}.binlog' /v:minimal

$env:DOTNET_ROOT = Join-Path $PWD '.dotnet'
$results = Join-Path $PWD 'artifacts\TestResults\row-names'

# Passed: 38 framework tests.
.\.dotnet\dotnet.exe exec artifacts\bin\TestFramework.UnitTests\Debug\net8.0\Microsoft.VisualStudio.TestPlatform.TestFramework.UnitTests.dll --treenode-filter '/*/*/*DataRow*/*' --report-trx --report-trx-filename framework-row-names.trx --results-directory artifacts\TestResults\row-names

# Passed: 7 metadata helper tests and 3 execution tests.
.\.dotnet\dotnet.exe exec artifacts\bin\MSTestAdapter.PlatformServices.UnitTests\Debug\net8.0\MSTestAdapter.PlatformServices.UnitTests.dll --treenode-filter '/*/*/TestDataSourceHelpersTests/*' --report-trx --report-trx-filename adapter-helper-row-names.trx --results-directory artifacts\TestResults\row-names
.\.dotnet\dotnet.exe exec artifacts\bin\MSTestAdapter.PlatformServices.UnitTests\Debug\net8.0\MSTestAdapter.PlatformServices.UnitTests.dll --treenode-filter '/*/*/TestMethodRunnerTests/*ArgumentsDisplayName*' --report-trx --report-trx-filename adapter-runner-row-names.trx --results-directory artifacts\TestResults\row-names

# Passed: 43 analyzer tests on net8.0 and 31 on net472.
.\.dotnet\dotnet.exe exec artifacts\bin\MSTest.Analyzers.UnitTests\Debug\net8.0\MSTest.Analyzers.UnitTests.dll --filter 'FullyQualifiedName~ConflictingDataRowDisplayNamesAnalyzerTests|FullyQualifiedName~DuplicateDataRowDisplayNameAnalyzerTests' --report-trx --report-trx-filename conflicting-names-analyzer.trx --results-directory $results
.\artifacts\bin\MSTest.Analyzers.UnitTests\Debug\net472\MSTest.Analyzers.UnitTests.exe --filter 'FullyQualifiedName~ConflictingDataRowDisplayNamesAnalyzerTests' --report-trx --report-trx-filename conflicting-names-analyzer-net472.trx --results-directory $results

# Passed: 11 packed-consumer acceptance tests.
.\build.cmd -projects test\IntegrationTests\MSTest.Acceptance.IntegrationTests\MSTest.Acceptance.IntegrationTests.csproj -bl
$env:DOTNET_ROLL_FORWARD = 'Major'
.\.dotnet\dotnet.exe exec artifacts\bin\MSTest.Acceptance.IntegrationTests\Debug\net11.0\MSTest.Acceptance.IntegrationTests.dll --filter 'FullyQualifiedName~ArgumentsDisplayNameTests' --report-trx --report-trx-filename acceptance-row-names.trx --results-directory $results

TRX reports and build/binlogs are retained under artifacts. The acceptance host uses runtime roll-forward locally; its generated consumers run on their specified target frameworks.

Preserve method names with opt-in ArgumentsDisplayName metadata and add MSTEST0089 for conflicting full and argument names. Cover naming precedence, localization, compiler diagnostics, and packed MTP/VSTest consumers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 6, 2026 13:00
@github-actions github-actions Bot added the needs/merge-conflict-resolution PR has merge conflicts that must be resolved. label Oct 6, 2026
Retain both the row-label helper signature and current xxHash buffer APIs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed needs/merge-conflict-resolution PR has merge conflicts that must be resolved. labels Oct 6, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation is consistent across framework, adapter, analyzer, API tracking, documentation, localization, and regression coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Adds argument-only display names for data rows while preserving method names, existing naming precedence, and compatibility behavior.

Changes:

  • Adds ArgumentsDisplayName to DataRowAttribute and TestDataRow<T>.
  • Propagates row labels through adapter discovery and execution.
  • Adds MSTEST0089, localization, documentation, API baselines, and comprehensive tests.

Confidence at a glance

🟢 Framework, adapter, analyzer, and tests Public APIs are tracked and documented; discovery/execution paths, precedence rules, localization, analyzer behavior, and packed consumers are covered.
File Description
test/​UnitTests/​TestFramework.UnitTests/​TestDataRowTests.cs Tests row-label metadata exposure.
test/​UnitTests/​TestFramework.UnitTests/​Attributes/​DataRowAttributeTests.cs Tests naming precedence and formatting.
test/​UnitTests/​MSTestAdapter.PlatformServices.UnitTests/​Helpers/​TestDataSourceHelpersTests.cs Tests metadata extraction.
test/​UnitTests/​MSTestAdapter.PlatformServices.UnitTests/​Execution/​TestMethodRunnerTests.cs Tests folded execution naming.
test/​UnitTests/​MSTest.Analyzers.UnitTests/​ConflictingDataRowDisplayNamesAnalyzerTests.cs Covers MSTEST0089 scenarios.
test/​IntegrationTests/​MSTest.Acceptance.IntegrationTests/​ArgumentsDisplayNameTests.cs Validates packed MTP/VSTest behavior.
src/​TestFramework/​TestFramework/​TestDataRow.cs Adds dynamic-row argument labels.
src/​TestFramework/​TestFramework/​PublicAPI/​PublicAPI.Unshipped.txt Tracks new public properties.
src/​TestFramework/​TestFramework/​ITestDataRow.cs Extends internal row metadata.
src/​TestFramework/​TestFramework/​InternalAPI/​InternalAPI.Unshipped.txt Tracks internal API additions.
src/​TestFramework/​TestFramework/​Internal/​TestDataSourceUtilities.cs Centralizes localized name formatting.
src/​TestFramework/​TestFramework/​Interfaces/​ITestDataSource.cs Documents new row metadata.
src/​TestFramework/​TestFramework/​Attributes/​DataSource/​DataRowAttribute.cs Adds inline-row argument labels.
src/​TestFramework/​TestFramework.Extensions/​PACKAGE.md Documents package usage.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.zh-Hant.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.zh-Hans.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.tr.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.ru.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.pt-BR.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.pl.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.ko.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.ja.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.it.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.fr.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.es.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.de.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​xlf/​Resources.cs.xlf Adds generated localization entries.
src/​Analyzers/​MSTest.Analyzers/​Resources.resx Defines MSTEST0089 messages.
src/​Analyzers/​MSTest.Analyzers/​Helpers/​WellKnownTypeNames.cs Registers TestDataRow<T>.
src/​Analyzers/​MSTest.Analyzers/​Helpers/​DiagnosticIds.cs Allocates MSTEST0089.
src/​Analyzers/​MSTest.Analyzers/​ConflictingDataRowDisplayNamesAnalyzer.cs Detects conflicting naming properties.
src/​Analyzers/​MSTest.Analyzers/​AnalyzerReleases.Unshipped.md Records the new diagnostic.
src/​Adapter/​MSTestAdapter.PlatformServices/​InternalAPI/​InternalAPI.Unshipped.txt Tracks the adapter overload.
src/​Adapter/​MSTestAdapter.PlatformServices/​Helpers/​TestDataSourceHelpers.cs Extracts argument-label metadata.
src/​Adapter/​MSTestAdapter.PlatformServices/​Execution/​TestMethodRunner.DataSource.cs Applies labels during execution.
src/​Adapter/​MSTestAdapter.PlatformServices/​Discovery/​AssemblyEnumerator.cs Applies labels during discovery.
docs/​RFCs/​005-Framework-Extensibility-Custom-DataSource.md Documents custom-source support.
docs/​glossary.md Documents MSTEST0089.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI balanced review requested due to automatic review settings October 6, 2026 13:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation consistently preserves precedence across framework, discovery, execution, analyzer, API, localization, and test coverage.

Review effort: Balanced
Findings: None

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 6, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. labels Oct 6, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Oct 6, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Oct 6, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Oct 6, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Final test-triage resolution — Azure Pipelines build 1624906

Final analysis (completed build, microsoft.testfx check).

  • Signature observed: PackagedWinUI.PackagedWinUITestCases.RetryTestMethod_PassesOnSecondAppContainerHost (TRX, Windows App Model Diagnostics artifact) reported failed on attempt 1 with Assertion failed. The first AppContainer host fails so Retry must activate a second host., then passed on attempt 2 and in the consolidated result file.
  • Assessment: This is the test's intended behavior, not a regression or flake. The test name and message describe a scenario where the first AppContainer host is expected to fail so the retry mechanism activates a second host — the fail-then-pass sequence is the test verifying that exact retry contract, and the final outcome recorded for the build is passed.
  • Historical comparison: Checked all 12 available prior builds (last 30 days, history.json, not incomplete) — every one of them shows the identical attempt-1-fails/attempt-2-passes pattern for this same test, with no instance where it failed terminally. This confirms a stable, by-design pattern rather than a new or recurring defect.
  • Other signals: No other failed or retried records in this build (candidateCount: 49, failureOrRetryCount: 1). No slow-test regressions (slowRegressionCount: 0) and no crash/hang/dump diagnostics were collected for this build.
  • Issue search: Searched open and closed issues for this test name and the "AppContainerHost" signature — no existing match found.

Conclusion: No action needed. This supersedes the earlier preliminary comment for this check, if any. No issue is being created since the evidence shows expected retry-contract behavior consistently reproduced across history, not a defect.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · 60 AIC · ⌖ 6.99 AIC · ⊞ 9.7K

@Evangelink
Amaury Levé (Evangelink) merged commit a55373b into main Oct 7, 2026
50 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the dev/amauryleve/data-row-naming branch October 7, 2026 07:37
@github-actions github-actions Bot removed the state/approved Proposal approved; ready for implementation. label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow DataRow to customize only the arguments part of the display name

3 participants