Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/RFCs/005-Framework-Extensibility-Custom-DataSource.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,12 +140,15 @@ public class CustomTestDataSourceAttribute : Attribute, ITestDataSource, ITestDa

To attach metadata to an individual row (rather than the whole source), return a single-element `object[]` whose only item is a `TestDataRow<T>` instance. MSTest unwraps `TestDataRow<T>.Value` to obtain the actual test method arguments and applies the row's metadata:

- `DisplayName` — overrides the display name for that row.
- `DisplayName` — overrides the entire display name for that row.
- `ArgumentsDisplayName` — labels the arguments while retaining the test method's display name, for example `MyTest (empty input)`. Non-blank labels are displayed verbatim and take precedence over the source's display-name callback. A non-null `DisplayName` takes precedence over the argument label.
- `IgnoreMessage` — when non-null, that single row is reported as skipped with the provided message.
- `TestCategories` — assigns test categories to the generated test case.

`T` can be a tuple when the test method takes more than one parameter.

Use matching `MSTest.TestFramework` and `MSTest.TestAdapter` versions that support `ArgumentsDisplayName`; older adapters do not apply this metadata.

```csharp
public class CustomTestDataSourceAttribute : Attribute, ITestDataSource
{
Expand Down
4 changes: 4 additions & 0 deletions docs/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,10 @@ An MSTest analyzer (`RedundantTestMethodAttributeAnalyzer`, informational severi

An MSTest analyzer (`DuplicateDataRowDisplayNameAnalyzer`, warning severity, enabled by default) that flags a `[DataRow(DisplayName = "...")]` argument on a `[TestMethod]` when another `[DataRow]` on the same method already declares the same non-blank `DisplayName`. Duplicate display names produce ambiguous or colliding test-result identities in reports and test explorers. Comparison is ordinal (case-sensitive) and only considers explicit non-blank `DisplayName` values — missing, null, empty, or whitespace-only names are not compared. Introduced in [PR #11409](https://github.com/microsoft/testfx/pull/11409).

### MSTEST0089 (conflicting data-row display names)

An MSTest analyzer (`ConflictingDataRowDisplayNamesAnalyzer`, warning severity, enabled by default) that reports constant `DisplayName` and `ArgumentsDisplayName` values supplied together on `[DataRow]`, `new DataRowAttribute(...)`, or `new TestDataRow<T>(...)` when the full name makes the non-blank argument label ineffective. A blank full name is unset on `DataRowAttribute`, but any non-null full name takes precedence on `TestDataRow<T>`. Null or blank argument labels, values not known at compile time, generated code, and custom derived attributes are not reported. Remove `ArgumentsDisplayName` to preserve the existing full name, or remove `DisplayName` to retain the method name with the argument label. The analyzer does not change runtime precedence.

### MSTest

Microsoft's unit testing framework for .NET. Provides attributes (`[TestClass]`, `[TestMethod]`, `[DataRow]`, etc.), assertions (`Assert`, `CollectionAssert`), and lifecycle hooks for writing and organizing tests. Packaged as `MSTest.TestFramework`, `MSTest.TestAdapter`, `MSTest.Analyzers`, and `MSTest.Sdk`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -315,7 +315,7 @@ private static bool TryUnfoldITestDataSource(ITestDataSource dataSource, UnitTes
// TestDataRow<T>), falling back to the whole-source ignore message. It must be
// recomputed per row so an ignore message on one row does not leak to later rows.
string? testDataSourceIgnoreMessage = sourceLevelIgnoreMessage;
if (TestDataSourceHelpers.TryHandleITestDataRow(d, parameters, out d, out string? ignoreMessageFromTestDataRow, out string? displayNameFromTestDataRow, out IList<string>? testCategoriesFromTestDataRow))
if (TestDataSourceHelpers.TryHandleITestDataRow(d, parameters, out d, out string? ignoreMessageFromTestDataRow, out string? displayNameFromTestDataRow, out IList<string>? testCategoriesFromTestDataRow, out string? argumentsDisplayNameFromTestDataRow))
{
testDataSourceIgnoreMessage = ignoreMessageFromTestDataRow ?? sourceLevelIgnoreMessage;
}
Expand All @@ -342,6 +342,9 @@ private static bool TryUnfoldITestDataSource(ITestDataSource dataSource, UnitTes

UnitTestElement discoveredTest = test.Clone();
discoveredTest.TestMethod.DisplayName = displayNameFromTestDataRow
?? (!StringEx.IsNullOrWhiteSpace(argumentsDisplayNameFromTestDataRow)
? TestDataSourceUtilities.ComputeDisplayName(methodInfo, argumentsDisplayNameFromTestDataRow)
: null)
?? dataSource.GetDisplayName(methodInfo, d)
?? TestDataSourceUtilities.ComputeDefaultDisplayName(methodInfo, d)
?? discoveredTest.TestMethod.DisplayName;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -118,9 +118,10 @@ private async Task<TestResult[]> ExecuteTestWithDataSourceAsync(ITestContext exe
: _test.DisplayName;

string? displayNameFromTestDataRow = null;
string? argumentsDisplayNameFromTestDataRow = null;
string? ignoreFromTestDataRow = null;
if (!actualDataAlreadyHandledDuringDiscovery && data is not null &&
TestDataSourceHelpers.TryHandleITestDataRow(data, _testMethodInfo.ParameterTypes, out data, out ignoreFromTestDataRow, out displayNameFromTestDataRow))
TestDataSourceHelpers.TryHandleITestDataRow(data, _testMethodInfo.ParameterTypes, out data, out ignoreFromTestDataRow, out displayNameFromTestDataRow, out _, out argumentsDisplayNameFromTestDataRow))
{
// Handled already.
}
Expand All @@ -145,19 +146,22 @@ private async Task<TestResult[]> ExecuteTestWithDataSourceAsync(ITestContext exe
data = tupleExpandedToArray;
}

// PERF: Extract ReflectionTestMethodInfo to avoid allocating it twice when testDataSource is not null
// and both GetDisplayName and ComputeDefaultDisplayName need to be consulted.
if (displayNameFromTestDataRow is null && testDataSource is not null)
if (displayNameFromTestDataRow is not null)
{
displayName = displayNameFromTestDataRow;
}
else if (!StringEx.IsNullOrWhiteSpace(argumentsDisplayNameFromTestDataRow))
{
ReflectionTestMethodInfo reflectionMethodInfo = _cachedReflectionMethodInfo ??= new ReflectionTestMethodInfo(_testMethodInfo.MethodInfo, _test.DisplayName);
displayName = TestDataSourceUtilities.ComputeDisplayName(reflectionMethodInfo, argumentsDisplayNameFromTestDataRow);
}
else if (testDataSource is not null)
{
ReflectionTestMethodInfo reflectionMethodInfo = _cachedReflectionMethodInfo ??= new ReflectionTestMethodInfo(_testMethodInfo.MethodInfo, _test.DisplayName);
displayName = testDataSource.GetDisplayName(reflectionMethodInfo, data)
?? TestDataSourceUtilities.ComputeDefaultDisplayName(reflectionMethodInfo, data)
?? displayName;
}
else
{
displayName = displayNameFromTestDataRow ?? displayName;
}

var stopwatch = Stopwatch.StartNew();
_testMethodInfo.SetArguments(data);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,24 @@ public static bool TryHandleITestDataRow(
out string? ignoreMessageFromTestDataRow,
out string? displayNameFromTestDataRow,
out IList<string>? testCategoriesFromTestDataRow)
=> TryHandleITestDataRow(d, testMethodParameters, out data, out ignoreMessageFromTestDataRow, out displayNameFromTestDataRow, out testCategoriesFromTestDataRow, out _);

public static bool TryHandleITestDataRow(
object?[] d,
ParameterInfo[] testMethodParameters,
out object?[] data,
out string? ignoreMessageFromTestDataRow,
out string? displayNameFromTestDataRow,
out IList<string>? testCategoriesFromTestDataRow,
out string? argumentsDisplayNameFromTestDataRow)
{
if (d.Length == 1 && d[0] is ITestDataRow testDataRow)
{
object? dataFromTestDataRow = testDataRow.Value;
ignoreMessageFromTestDataRow = testDataRow.IgnoreMessage;
displayNameFromTestDataRow = testDataRow.DisplayName;
testCategoriesFromTestDataRow = testDataRow.TestCategories;
argumentsDisplayNameFromTestDataRow = testDataRow.ArgumentsDisplayName;

data = TryHandleTupleDataSource(dataFromTestDataRow, testMethodParameters, out object?[] tupleExpandedToArray)
? tupleExpandedToArray
Expand All @@ -47,6 +58,7 @@ public static bool TryHandleITestDataRow(
ignoreMessageFromTestDataRow = null;
displayNameFromTestDataRow = null;
testCategoriesFromTestDataRow = null;
argumentsDisplayNameFromTestDataRow = null;
return false;
}

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#nullable enable
static Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers.TestDataSourceHelpers.TryHandleITestDataRow(object?[]! d, System.Reflection.ParameterInfo![]! testMethodParameters, out object?[]! data, out string? ignoreMessageFromTestDataRow, out string? displayNameFromTestDataRow, out System.Collections.Generic.IList<string!>? testCategoriesFromTestDataRow, out string? argumentsDisplayNameFromTestDataRow) -> bool
System.IO.Hashing.XxHashShared.State.Accumulators -> System.IO.Hashing.XxHashShared.State.AccumulatorsBuffer
System.IO.Hashing.XxHashShared.State.AccumulatorsBuffer
System.IO.Hashing.XxHashShared.State.AccumulatorsBuffer.AccumulatorsBuffer() -> void
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ MSTEST0085 | Usage | Info | TestClassAttributeShouldNotBeAppliedToAbstractClassA
MSTEST0086 | Usage | Info | RedundantTestMethodAttributeAnalyzer, [Documentation](https://learn.microsoft.com/dotnet/core/testing/mstest-analyzers/mstest0086)
MSTEST0087 | Usage | Warning | DuplicateDataRowDisplayNameAnalyzer, [Documentation](https://learn.microsoft.com/dotnet/core/testing/mstest-analyzers/mstest0087)
MSTEST0088 | Usage | Error | TestClassConstructorShouldBeValidAnalyzer, [Documentation](https://learn.microsoft.com/dotnet/core/testing/mstest-analyzers/mstest0088)
MSTEST0089 | Usage | Warning | ConflictingDataRowDisplayNamesAnalyzer, [Documentation](https://learn.microsoft.com/dotnet/core/testing/mstest-analyzers/mstest0089)
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using System.Collections.Immutable;

using Analyzer.Utilities.Extensions;

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Diagnostics;
using Microsoft.CodeAnalysis.Operations;

using MSTest.Analyzers.Helpers;

namespace MSTest.Analyzers;

/// <summary>
/// MSTEST0089: <inheritdoc cref="Resources.ConflictingDataRowDisplayNamesTitle"/>.
/// </summary>
[DiagnosticAnalyzer(LanguageNames.CSharp, LanguageNames.VisualBasic)]
public sealed class ConflictingDataRowDisplayNamesAnalyzer : DiagnosticAnalyzer
{
internal static readonly DiagnosticDescriptor Rule = DiagnosticDescriptorHelper.Create(
DiagnosticIds.ConflictingDataRowDisplayNamesRuleId,
new LocalizableResourceString(nameof(Resources.ConflictingDataRowDisplayNamesTitle), Resources.ResourceManager, typeof(Resources)),
new LocalizableResourceString(nameof(Resources.ConflictingDataRowDisplayNamesMessageFormat), Resources.ResourceManager, typeof(Resources)),
new LocalizableResourceString(nameof(Resources.ConflictingDataRowDisplayNamesDescription), Resources.ResourceManager, typeof(Resources)),
Category.Usage,
DiagnosticSeverity.Warning,
isEnabledByDefault: true);

/// <inheritdoc />
public override ImmutableArray<DiagnosticDescriptor> SupportedDiagnostics => ImmutableArray.Create(Rule);

/// <inheritdoc />
public override void Initialize(AnalysisContext context)
{
context.ConfigureGeneratedCodeAnalysis(GeneratedCodeAnalysisFlags.None);
context.EnableConcurrentExecution();

context.RegisterCompilationStartAction(context =>
{
INamedTypeSymbol? dataRowAttribute = context.Compilation.GetOrCreateTypeByMetadataName(WellKnownTypeNames.MicrosoftVisualStudioTestToolsUnitTestingDataRowAttribute);
INamedTypeSymbol? testDataRow = context.Compilation.GetOrCreateTypeByMetadataName(WellKnownTypeNames.MicrosoftVisualStudioTestToolsUnitTestingTestDataRow1);
if (dataRowAttribute is not null || testDataRow is not null)
{
context.RegisterOperationAction(context => AnalyzeCreation(context, dataRowAttribute, testDataRow), OperationKind.ObjectCreation);
}
});
}

private static void AnalyzeCreation(OperationAnalysisContext context, INamedTypeSymbol? dataRowAttribute, INamedTypeSymbol? testDataRow)
{
var creation = (IObjectCreationOperation)context.Operation;
bool isDataRowAttribute = dataRowAttribute is not null && dataRowAttribute.Equals(creation.Type, SymbolEqualityComparer.Default);
bool isTestDataRow = testDataRow is not null && testDataRow.Equals(creation.Type?.OriginalDefinition, SymbolEqualityComparer.Default);
if ((!isDataRowAttribute && !isTestDataRow) || creation.Initializer is null)
{
return;
}

ISimpleAssignmentOperation? displayName = null;
ISimpleAssignmentOperation? argumentsDisplayName = null;
foreach (IOperation initializer in creation.Initializer.Initializers)
{
if (initializer is ISimpleAssignmentOperation { Target: IPropertyReferenceOperation property } assignment)
{
switch (property.Property.Name)
{
case "DisplayName":
displayName = assignment;
break;
case "ArgumentsDisplayName":
argumentsDisplayName = assignment;
break;
}
}
}
Comment thread
Evangelink marked this conversation as resolved.
Resolved

if (displayName?.Value.ConstantValue is not { HasValue: true, Value: string fullName }
|| argumentsDisplayName?.Value.ConstantValue is not { HasValue: true, Value: string argumentName }
|| string.IsNullOrWhiteSpace(argumentName)
|| (isDataRowAttribute && string.IsNullOrWhiteSpace(fullName)))
{
return;
}

context.ReportDiagnostic(creation.CreateDiagnostic(Rule));
}
}
1 change: 1 addition & 0 deletions src/Analyzers/MSTest.Analyzers/Helpers/DiagnosticIds.cs
Original file line number Diff line number Diff line change
Expand Up @@ -93,4 +93,5 @@ internal static class DiagnosticIds
public const string RedundantTestMethodAttributeRuleId = "MSTEST0086";
public const string DuplicateDataRowDisplayNameRuleId = "MSTEST0087";
public const string MSTestHostTestClassInjectionNotSupportedRuleId = "MSTEST0088";
public const string ConflictingDataRowDisplayNamesRuleId = "MSTEST0089";
}
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ internal static class WellKnownTypeNames
public const string MicrosoftVisualStudioTestToolsUnitTestingTestClassAttribute = "Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute";
public const string MicrosoftVisualStudioTestToolsUnitTestingTestCleanupAttribute = "Microsoft.VisualStudio.TestTools.UnitTesting.TestCleanupAttribute";
public const string MicrosoftVisualStudioTestToolsUnitTestingTestContext = "Microsoft.VisualStudio.TestTools.UnitTesting.TestContext";
public const string MicrosoftVisualStudioTestToolsUnitTestingTestDataRow1 = "Microsoft.VisualStudio.TestTools.UnitTesting.TestDataRow`1";
public const string MicrosoftVisualStudioTestToolsUnitTestingTestFilterProviderAttribute = "Microsoft.VisualStudio.TestTools.UnitTesting.TestFilterProviderAttribute";
public const string MicrosoftVisualStudioTestToolsUnitTestingTestFilterProviderAttribute1 = "Microsoft.VisualStudio.TestTools.UnitTesting.TestFilterProviderAttribute`1";
public const string MicrosoftVisualStudioTestToolsUnitTestingTestInitializeAttribute = "Microsoft.VisualStudio.TestTools.UnitTesting.TestInitializeAttribute";
Expand Down
12 changes: 12 additions & 0 deletions src/Analyzers/MSTest.Analyzers/Resources.resx
Original file line number Diff line number Diff line change
Expand Up @@ -775,6 +775,18 @@ The type declaring these methods should also respect the following rules:
<value>Do not duplicate the 'DataRow' display name '{0}'. Each 'DataRow' on a test method should use a unique display name.</value>
<comment>{0} is the duplicated display name. {Locked="DataRow"}</comment>
</data>
<data name="ConflictingDataRowDisplayNamesTitle" xml:space="preserve">
<value>Do not combine 'DisplayName' and 'ArgumentsDisplayName'</value>
<comment>{Locked="DisplayName"}{Locked="ArgumentsDisplayName"}</comment>
</data>
<data name="ConflictingDataRowDisplayNamesMessageFormat" xml:space="preserve">
<value>'ArgumentsDisplayName' is ignored because 'DisplayName' overrides the entire test case name</value>
<comment>{Locked="DisplayName"}{Locked="ArgumentsDisplayName"}</comment>
</data>
<data name="ConflictingDataRowDisplayNamesDescription" xml:space="preserve">
<value>Specify either 'DisplayName' for a full test case name or 'ArgumentsDisplayName' for an argument label that retains the method display name, but not both. Remove the ignored 'ArgumentsDisplayName' to preserve the current name, or remove 'DisplayName' to use the argument label.</value>
<comment>{Locked="DisplayName"}{Locked="ArgumentsDisplayName"}</comment>
</data>
<data name="UseRetryWithTestMethodDescription" xml:space="preserve">
<value>Retry attributes have no effect on methods or classes that MSTest does not recognize as tests.</value>
<comment>{Locked="MSTest"}</comment>
Expand Down
Loading
Loading