Skip to content

Enable dotnet test for device platforms via DeviceRunners.Testing.Targets - #106

Merged
mattleibow merged 96 commits into
mainfrom
mattleibow/investigate-dotnet-test-mobile
May 14, 2026
Merged

mattleibow merged 96 commits into
mainfrom
mattleibow/investigate-dotnet-test-mobile

Conversation

@mattleibow

@mattleibow mattleibow commented May 6, 2026 •

Copy link
Copy Markdown
Owner

Summary

Adds first-class dotnet test support for device test apps on Android, iOS, macOS Catalyst, and Windows via a new DeviceRunners.Testing.Targets NuGet package.

Quick start

1. Add the package:

<PackageReference Include="DeviceRunners.Testing.Targets" />

2. Configure your app (in MauiProgram.cs):

.UseVisualTestRunner(conf => conf
    .AddCliConfiguration()       // reads config from DeviceRunners CLI or dotnet test
    .AddConsoleResultChannel()
    .AddTestAssembly(typeof(MauiProgram).Assembly)
    .AddXunit())

3. Run tests:

dotnet test MyApp.DeviceTests.csproj -f net10.0-android
dotnet test MyApp.DeviceTests.csproj -f net10.0-ios
dotnet test MyApp.DeviceTests.csproj -f net10.0-maccatalyst
dotnet test MyApp.DeviceTests.csproj -f net10.0-windows10.0.19041.0

How it works

The package replaces the default VSTest target for device TFMs and orchestrates: Build → Deploy → Launch → Collect results → Report.

It ships self-contained, trimmed, single-file CLI binaries per RID (~20 MB each): osx-arm64, osx-x64, win-x64, win-arm64, linux-x64, linux-arm64. The targets auto-detect the host OS and architecture at build time.

Configuration injection

Each platform uses a different mechanism to pass configuration to the app:

Platform Mechanism
Android Env vars baked into APK at build time via _GenerateEnvironmentFiles
iOS Env vars passed via SimCtl.LaunchAppAsync (AppleDev 0.8.10+)
macOS Catalyst Env vars via ProcessStartInfo (direct process launch)
Windows (EXE) Env vars via ProcessStartInfo (direct process launch)
Windows (MSIX loose) CLI arguments via winapp.exe --args (env vars can't be forwarded to packaged apps)

AddCliConfiguration() reads env vars first, then falls back to --device-runners-* CLI arguments. When neither source provides DEVICE_RUNNERS_AUTORUN, it's a no-op and the visual runner behaves normally.

Windows support

Mode Detection How it runs
Unpackaged .exe in build output CLI launches the EXE directly (-p:WindowsPackageType=None)
Packaged AppxManifest.xml in build output CLI registers via winapp.exe loose-file deploy (no publish/cert needed)

Output

Standard dotnet test format:

Results File: /path/to/test-results.trx
Test summary: total: 44, failed: 5, succeeded: 36, skipped: 3, duration: 14.8s

Exit codes: 0 = pass, 1 = test failures, 2 = app crashed (TCP connection lost without end event).

Configuration

Property Default Description
DeviceRunnersPort 16384 TCP port for result collection
DeviceRunnersConnectionTimeout 120 Seconds to wait for the app to connect
DeviceRunnersDataTimeout 30 Seconds of silence before assuming done
DeviceRunnersDevice (auto) Target device/simulator ID
DeviceRunnersBin (bundled) Override the CLI tool path

CI strategy

Three independent test paths are validated per platform:

Path What it tests Workflows
CLI direct device-runners test --app ... test-tcp-*
dotnet test MSBuild targets integration test-dotnet-test-* (new)
XHarness Legacy XHarness tool test-xharness-*

What's new

  • DeviceRunners.Testing.Targets — MSBuild package with .props, .targets, and per-RID CLI binaries
  • AddCliConfiguration() — Reads env vars first, falls back to CLI args for MSIX support
  • CLI trim support — PublishTrimmed, TrimmerRootAssembly, JsonSerializerIsReflectionEnabledByDefault
  • 10 new CI jobs — test-dotnet-test-* workflows for GH Actions and AzDO (5 each)
  • Documentation — using-dotnet-test.md plus per-platform guides; updated README, TOC, architecture overview, CI docs

Reviewer notes

Suggested review order:

  1. src/DeviceRunners.Testing.Targets/build/*.targets and *.props — core MSBuild targets
  2. src/DeviceRunners.Testing.Targets/DeviceRunners.Testing.Targets.csproj — PublishCliTools target with item batching
  3. src/DeviceRunners.Cli/DeviceRunners.Cli.csproj — trim properties, PublishForTargetsPackage
  4. src/DeviceRunners.VisualRunners/VisualTestRunnerConfigurationBuilderExtensions.cs — AddCliConfiguration()
  5. sample/test/DeviceTestingKitApp.DeviceTests/ — MauiProgram.cs + csproj changes
  6. CI workflows (test-dotnet-test-*)
  7. Documentation (docs/articles/dotnet-test-*.md, using-dotnet-test.md)

mattleibow and others added 25 commits May 8, 2026 18:27
….Targets

Introduces a new MSBuild targets NuGet package that makes dotnet test work
for Android and iOS MAUI test apps with a single command:

  dotnet test MyTests.csproj -f net10.0-android

New: DeviceRunners.Testing.Targets NuGet package
- build/DeviceRunners.Testing.Targets.props injects DEVICE_RUNNERS_AUTORUN,
  DEVICE_RUNNERS_PORT, DEVICE_RUNNERS_HOST_NAMES into the app bundle at build
  time via AndroidEnvironmentVariables / iOSEnvironmentVariables MSBuild items
- build/DeviceRunners.Testing.Targets.targets overrides the VSTest target for
  Android/iOS to invoke the bundled CLI tool; selects the correct self-contained
  binary for the host RID (win-x64, linux-x64, osx-x64, osx-arm64)
- PublishCliTools MSBuild target publishes the CLI self-contained + trimmed for
  all 4 RIDs before packing the NuGet package

New: UseTestRunnerEnvironment() extension method
Replaces the compile-time #if MODE_NON_INTERACTIVE_VISUAL block with a runtime
env var check. When DEVICE_RUNNERS_AUTORUN is set (injected by the .targets file),
the app auto-enables headless mode with TCP result streaming.

Updated: sample test project
- Removes #if MODE_NON_INTERACTIVE_VISUAL and TestingMode/MODE_NON_INTERACTIVE_VISUAL
- Calls UseTestRunnerEnvironment() instead (no-op when env var is absent)
- Imports DeviceRunners.Testing.Targets .props and .targets directly for in-repo use

Updated: CI workflows (test-tcp-android-linux, test-tcp-ios)
- Replaces 2-step dotnet publish + device-runners CLI with a single dotnet test call
- Adds PublishCliTools step to build bundled native binaries before dotnet test

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dotnet pack triggers PublishCliTools via BeforeTargets=GenerateNuspec, which
is the natural way to prepare the NuGet package and its bundled binaries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… support

- Replace non-existent AndroidEnvironmentVariables ItemGroup with correct
  _GeneratedAndroidEnvironment items in a BeforeTargets=_GenerateEnvironmentFiles target.
  The Android SDK compiles these into the native libxamarin-app.so so that Mono
  makes them available via Environment.GetEnvironmentVariable() at app startup.

- Use %3B to escape semicolons in HOST_NAMES value, preventing MSBuild from
  treating the list separator as an item separator.

- Update iOS LaunchAppAsync to accept environmentVariables parameter and inject
  them using the xcrun simctl SIMCTL_CHILD_* mechanism when provided.

- Update iOS TestCommand to pass DEVICE_RUNNERS_* env vars at app launch time
  (matching how Windows and macOS CLI commands already inject them).

- Add DeviceRunners.Cli FDD publish output to tools/ for in-repo testing.

Android dotnet test now works end-to-end: 44 tests, 36 pass, 5 intentional fail,
TRX written, correct non-zero exit code.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix inaccurate comment in MacOSService: 'open --env' does support env
  vars; direct launch is used for process handle retention and simplicity
- Read stderr on xcrun simctl launch failure for better CI diagnostics
- Make iOS CI dotnet pack use 'release' consistently with macOS CI
- Fix .app bundle discovery: MSBuild Include glob (*.app/) does not
  reliably match directories on macOS; use Directory.GetDirectories
  instead — verified working for both iOS and macOS Catalyst
- Remove committed binary DLLs from tools/ and add .gitignore so
  they are regenerated by dotnet pack (PublishCliTools target)
- Validated end-to-end: dotnet test works for macOS Catalyst (44 tests,
  TRX generated) and iOS simulator (44 tests, TRX generated)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rgets

Mirror the same simplification already applied to GitHub Actions:
- Replace 'dotnet tool install + dotnet publish + device-runners <plat> test'
  with 'dotnet pack Testing.Targets + dotnet test'
- Remove TestingMode=NonInteractiveVisual (replaced by env var auto-config)
- Remove manual .app/.apk/.exe discovery steps
- All four platforms updated: Android, iOS, macOS, Windows

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- global.json: pin SDK to 10.0.100 with rollForward=latestFeature
  so any 10.0.* SDK satisfies the requirement.
- Directory.Build.props: set ValidateXcodeVersion=false so minor
  Xcode version mismatches don't break builds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of the raw MSB3073 'command exited with code 1' error:
- All Exec tasks now use IgnoreExitCode=true and capture the exit code
- New _DeviceRunnersReportResults target parses the TRX via XmlPeek
  and emits a dotnet-test-style summary line, e.g.:
    Failed! - Failed: 5, Passed: 36, Skipped: 3, Total: 44
- On failure, emits a clean Error task message instead of MSB3073
- Fallback error for the case where CLI crashes before writing TRX
- Fix: count skipped via UnitTestResult[@outcome='NotExecuted'] elements
  because xunit/nunit skipped tests don't increment Counters/@notExecuted

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…gle Exec

Replace CallTarget with proper DependsOnTargets chains and eliminate
duplicate Exec blocks. New target hierarchy:

  VSTest
    → _DeviceRunnersCheckCli  (fail fast before building)
    → Build
    → _DeviceRunnersRunTests
        → _DeviceRunnersPrepareArgs
            → _DeviceRunnersCommonArgs   (TRX path, device flag, shared CLI flags)
            → _DeviceRunnersAndroidArgs  (Condition=android)
            → _DeviceRunnersiOSArgs      (Condition=ios)
            → _DeviceRunnersMacOSArgs    (Condition=maccatalyst)
            → _DeviceRunnersWindowsArgs  (Condition=windows)
        → _DeviceRunnersExecTests        (single Exec, consumes $(_DeviceRunnersCliArgs))
        → _DeviceRunnersReportResults    (XmlPeek TRX, emit summary)

Each platform arg target sets $(_DeviceRunnersCliArgs); the optional
$(_DeviceRunnersDeviceArg) is built once in _DeviceRunnersCommonArgs and
appended where the CLI supports it (Android, iOS). No duplicate Exec pairs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace CallTarget usage with proper DependsOnTargets chains
- VSTest -> _DeviceRunnersCheckCli;Build;_DeviceRunnersRunTests
- _DeviceRunnersRunTests -> _DeviceRunnersPrepareArgs;_DeviceRunnersExecTests;_DeviceRunnersReportResults
- _DeviceRunnersPrepareArgs -> _DeviceRunnersCommonArgs + 4 platform targets
- Introduce _DeviceRunnersPlatform and _DeviceRunnersIsSupportedPlatform properties
  so all conditions use property comparisons instead of repeated function calls
- Single _DeviceRunnersExecTests consumes fully-assembled $(_DeviceRunnersCliArgs)
- Single _DeviceRunnersDeviceArg with leading space for clean concatenation
- ASCII-only comments (no box-drawing chars, no -- inside XML comments)

Validated: dotnet test macOS Catalyst - 44 tests (36 pass, 5 fail, 3 skip)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Emit the standard dotnet test two-line format:
  Results File: /path/to/test-results.trx
  Test summary: total: 44, failed: 5, succeeded: 36, skipped: 3, duration: 9.3s

Changes:
- Output 'Results File:' line first (before summary, not after)
- Use 'succeeded' instead of 'passed' to match dotnet test terminology
- Read scalar TRX counters as PropertyName (not ItemName) for clean interpolation
- Compute duration from TRX Times/@start and Times/@finish attributes
- Keep bare 'Test run failed.' Error as the MSBuild failure signal

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds Code="TESTERROR" to the two test-failure Error tasks so the
output matches what dotnet test emits for failing tests:

  error TESTERROR: Test run failed.

Tools that parse dotnet test output keyed on the TESTERROR code will
now correctly recognise device test failures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dotnet CLI terminal logger (default in .NET 8+) filters MSBuild
Message tasks, so Results File and Test summary were invisible at
default verbosity. Fix by embedding the full summary in the TESTERROR
error message which is always surfaced:

  error TESTERROR: Test run failed: 5 of 44 tests failed
    (36 succeeded, 3 skipped, duration: 9.7s). Results File: /path/...

The Message tasks are kept for --tl:off / --verbosity normal users
where they render as the clean two-line dotnet test format:
  Results File: /path/to/test-results.trx
  Test summary: total: 44, failed: 5, succeeded: 36, skipped: 3, duration: 9.7s

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use %0A (MSBuild newline escape) to split the error into two lines
matching the standard dotnet test output format:

  error TESTERROR: Results File: /path/to/test-results.trx
  error TESTERROR: Test summary: total: 44, failed: 5, succeeded: 36, skipped: 3, duration: 18.5s

Both lines are always visible via the terminal logger.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Test classes are discovered via reflection at runtime. Without explicit
trimmer roots the iOS/Android linker strips the referenced test assemblies
(MauiLibrary.XunitTests, Library.NUnitTests), causing only tests in the
main DeviceTests assembly to run (~8 instead of ~44).

TrimmerRootAssembly tells the linker to keep the full assembly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The event stream protocol sends begin/end events to bracket the test run.
If the TCP connection closes after receiving results but without the end
event, the app crashed or was killed mid-run.

Previously this produced partial results with no indication of a crash.
Now the CLI explicitly reports:

  The application appears to have crashed during the test run.
  Only 8 test result(s) were received before the connection was lost.
  Check the device log for crash details.

And forces a non-zero exit code even if all received tests passed, so
CI pipelines correctly fail.

Changes:
- EventStreamService: add HasStarted and HasEnded tracking
- BaseTestCommand: check for begin-without-end after listener completes

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CLI exit codes:
  0 = all tests passed
  1 = test failures
  2 = app crashed (begin event received, results streamed, but no end event)

MSBuild targets check the exit code directly (no file reading needed):
  exit code 2 + TRX:    'Test summary: ... (incomplete: app crashed)'
  exit code 2 + no TRX: 'The application crashed before producing any test results'
  exit code 1 + TRX:    'Test summary: ...' (normal failure)

Validated:
  iOS:   8 tests, crash detected, '(incomplete: app crashed)' shown
  macOS: 44 tests, no crash, normal TESTERROR shown

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the (int testFailures, string? testResults) tuple with a
protected record TestListenerResult(int FailedCount, string? ResultsFile, bool Crashed).

This makes the crash detection explicit via a bool rather than a
sentinel value, and gives clear property names at all call sites.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Centralise exit code mapping and success logic on the record itself:
  - ToExitCode(): 0 = success, 1 = test failures, 2 = app crashed
  - Success: true when no failures and no crash

All callers now use listener.ToExitCode() and listener.Success instead
of duplicating the conditional logic.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MAUI 10.0.20 had a NullReferenceException in Button.LayoutButton when
the platform button was null during a layout pass on the UI thread.
This crashed the app mid-test-run on iOS, producing only partial results.

The fix (dotnet/maui#35284, fixing dotnet/maui#31048) adds a null guard
in LayoutButton. MAUI 10.0.60 includes this fix.

Results: iOS now completes all 44 tests (previously crashed at 8).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the existing AppleDev predicate parameter on GetLogsPlainAsync to
filter logs to just the app process:

  predicate: 'process == "DeviceTestingKitApp.DeviceTests"'

Results:
  Before: 17,000 lines of OS-level noise, no managed exceptions
  After:  190 lines with all PASS/FAIL output and full managed stack
          traces when the app crashes (NullReferenceException, etc.)

Also restores MAUI 10.0.60 (was temporarily reverted for testing).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AppleDev 0.8.10 adds SimCtl.LaunchAppAsync overload with env vars,
so we can replace the manual ProcessStartInfo + SIMCTL_CHILD_* workaround
with the native API call.

AndroidSdk 0.35.1 is the latest stable release.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Testing.Targets .props defaults WindowsPackageType=None (for dotnet test
unpackaged mode). The TCP Windows packaged and XHarness Windows workflows
need MSIX packaging, so pass -p:WindowsPackageType=MSIX explicitly.

Fixes: TCP Windows (GH+AzDO), XHarness Windows (GH+AzDO) CI failures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New articles:
- using-dotnet-test.md: Main guide covering setup, configuration,
  output format, crash detection, MSBuild properties, and how it works
- dotnet-test-android.md: Emulator setup, env var injection, APK config
- dotnet-test-ios.md: Simulator setup, device logs, crash troubleshooting
- dotnet-test-macos.md: Direct launch, no simulator needed
- dotnet-test-windows.md: Unpackaged EXE mode, WindowsPackageType

Updated:
- toc.yml: 'dotnet test (Recommended)' section at top, CLI renamed to
  'Advanced', XHarness renamed to 'Legacy' (from 'CLI Runners')
- index.md: Getting Started leads with dotnet test
- README.md: dotnet test as primary, links to docs site
- technical-architecture-overview.md: Testing.Targets package section
  with MSBuild target chain, exit code protocol, package structure
- ci-pipeline.md: dotnet test CI examples for GH Actions and AzDO

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove WindowsPackageType=None default from .props entirely. Instead,
the _DeviceRunnersWindowsArgs target detects what the build produced:

- If .exe exists (unpackaged): uses 'windows test --app path.exe'
- If .msix exists (packaged): uses 'windows test --app path.msix'
- If neither: clear error message

This means dotnet publish for MSIX packaging is no longer broken by the
Testing.Targets package. CI workflows no longer need -p:WindowsPackageType
overrides. Also search both artifacts/ and sample/ for .msix in CI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mattleibow
mattleibow force-pushed the mattleibow/investigate-dotnet-test-mobile branch from 932a033 to aa1cb54 Compare May 8, 2026 17:34
mattleibow and others added 4 commits May 8, 2026 20:02
Create DeviceRunners.VisualRunners.Core (net9.0;net10.0 only) with the
cross-platform testing types that the CLI needs:

Moved from VisualRunners:
- Testing/ITestResultInfo, ITestCaseInfo, ITestAssemblyInfo,
  ITestAssemblyConfiguration, TestResultStatus
- Testing/Channels/IResultChannel, IResultChannelFormatter,
  FileResultChannel, FileResultChannelOptions, TestResultEvent,
  EventStreamFormatter, TextResultChannelFormatter,
  TrxResultChannelFormatter, TextWriterResultChannel

The CLI now references VisualRunners.Core instead of VisualRunners,
breaking the dependency on multi-target mobile TFMs. This unblocks
self-contained and AOT publish (no more Mono runtime pack errors).

VisualRunners references VisualRunners.Core transitively, so all
existing consumers are unaffected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add TrimmerRootAssembly for Spectre.Console.Cli and DeviceRunners.Cli
to preserve types discovered via reflection (command settings classes).
Spectre.Console.Cli is explicitly not trim-safe (spectreconsole/spectre.console#1690)
so the entire assembly must be rooted.

Suppress trim analysis warnings since Spectre.Console.Cli generates
unavoidable IL2026/IL2090 warnings that are covered by the rooting.

Result: 19MB self-contained single-file binary (down from 21MB multi-file).
  dotnet publish -r osx-arm64 --self-contained -p:PublishSingleFile=true

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CLI csproj now has SelfContained, PublishSingleFile, PublishTrimmed,
and RuntimeIdentifiers (osx-arm64;win-x64) set by default. Plain
'dotnet publish -r osx-arm64' produces a 19MB single-file binary.

The NuGet package PublishCliTools target overrides these back to
framework-dependent (SelfContained=false, PublishSingleFile=false,
PublishTrimmed=false) since the bundled tool runs via 'dotnet <dll>'.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NuGet package now ships self-contained, trimmed, single-file native
binaries for each platform in tools/<rid>/:

  tools/osx-arm64/DeviceRunners.Cli     (~20 MB)
  tools/win-x64/DeviceRunners.Cli.exe   (~20 MB)
  tools/linux-x64/DeviceRunners.Cli     (~20 MB)

The MSBuild targets detect the host OS and select the right binary.
Falls back to framework-dependent DLL at tools/DeviceRunners.Cli.dll
for in-repo dev or older packages.

Changes:
- CLI csproj: add JsonSerializerIsReflectionEnabledByDefault, linux-x64 RID
- Testing.Targets csproj: PublishCliTools publishes 3 RIDs (self-contained)
- .targets: detect host OS -> pick tools/<rid>/DeviceRunners.Cli[.exe]

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mattleibow and others added 7 commits May 13, 2026 22:00
The static <None Include="tools\**"> glob was evaluated at project load
time, before PublishCliTools populated the tools/ directory. This meant
a clean pack produced a nupkg with build/ but zero tools/ files.

Fix: move the tools glob into the PublishCliTools target as a
_PackageFiles item, so it runs AFTER the per-RID publishes complete.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dotnet test Android workflow was missing several setup steps that the
working TCP workflow has:
- Java 21 installation (required by Android SDK/emulator)
- mkdir for AVD directory (ANDROID_AVD_HOME must exist)
- ANDROID_EMULATOR_HOME env var
- AndroidSdkDirectory MSBuild property for build

These omissions caused the emulator to not boot or the build to not find
the Android SDK, resulting in the job hanging indefinitely.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e-dotnet-test-mobile

# Conflicts:
#	sample/test/DeviceTestingKitApp.DeviceTests/DeviceTestingKitApp.DeviceTests.csproj
PR #124 enabled TreatWarningsAsErrors globally. The CLI project has
known trim warnings (IL2026, IL2090) from reflection-based JSON/XML
serialization and Spectre.Console.Cli. These are suppressed during
publish via SuppressTrimAnalysisWarnings but TreatWarningsAsErrors
promoted them to errors during normal builds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace reflection-based JsonSerializer calls with source-generated
CliJsonContext for all CommandResult types and Dictionary<string,object>.
This eliminates IL2026 warnings for JSON serialization and removes the
need for JsonSerializerIsReflectionEnabledByDefault.

XML serialization (XmlSerializer) has no source-gen equivalent, so
those call sites use #pragma to suppress IL2026.

Reflection-based text output (GetProperties) uses
DynamicallyAccessedMembers annotations to satisfy IL2090.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restored comments that were accidentally removed during the source-gen
refactor. Removed the dead JSON fallback path since all CommandResult
types are registered in CliJsonContext.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DeviceRunners.Testing.Targets has IncludeBuildOutput=false because it
ships only build/ and tools/ content (no lib/ assemblies). NU5128 warns
about the mismatch, which TreatWarningsAsErrors promotes to an error.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new DeviceRunners.Testing.Targets package to enable dotnet test for device-platform MAUI test apps by overriding VSTest, launching the DeviceRunners CLI, collecting TCP results, and reporting TRX output.

Changes:

  • Adds MSBuild props/targets and packaging support for bundled per-RID CLI binaries.
  • Adds CLI/visual-runner support for environment/argument-driven test autorun and trim-safe JSON serialization.
  • Adds sample, CI, and documentation coverage for Android, iOS, macOS Catalyst, and Windows dotnet test workflows.

Reviewed changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
src/DeviceRunners.VisualRunners/VisualTestRunnerConfigurationBuilderExtensions.cs Adds AddCliConfiguration() for env/CLI argument configuration.
src/DeviceRunners.Testing.Targets/DeviceRunners.Testing.Targets.csproj Defines targets-only NuGet package and CLI publish target.
src/DeviceRunners.Testing.Targets/build/DeviceRunners.Testing.Targets.targets Adds custom VSTest orchestration and platform-specific run args.
src/DeviceRunners.Testing.Targets/build/DeviceRunners.Testing.Targets.props Adds default MSBuild properties and disables default VSTest/MTP behavior.
src/DeviceRunners.Testing.Targets/.gitignore Ignores generated packaged CLI tools.
src/DeviceRunners.Cli/Services/MacOSService.cs Switches plist JSON deserialization to source-generated context.
src/DeviceRunners.Cli/Models/CliJsonContext.cs Adds source-generation metadata for CLI JSON output.
src/DeviceRunners.Cli/DeviceRunners.Cli.csproj Adds trim/single-file publish settings for packaged CLI binaries.
src/DeviceRunners.Cli/Commands/Windows/TestCommand.cs Passes DeviceRunners CLI arguments to loose MSIX launches.
src/DeviceRunners.Cli/Commands/BaseCommand.cs Updates JSON/XML/text output paths for trimming.
src/DeviceRunners.Cli/Commands/BaseAsyncCommand.cs Updates async command output serialization for trimming.
sample/test/DeviceTestingKitApp.DeviceTests/MauiProgram.cs Uses AddCliConfiguration() in the sample app.
sample/test/DeviceTestingKitApp.DeviceTests/DeviceTestingKitApp.DeviceTests.csproj Imports new in-repo testing targets for sample use.
README.md Promotes dotnet test workflow and updates runner guidance.
global.json Pins the repository SDK to .NET 10.
docs/articles/using-dotnet-test.md Adds primary dotnet test documentation.
docs/articles/toc.yml Adds dotnet test docs to the TOC.
docs/articles/technical-architecture-overview.md Documents the new targets package architecture.
docs/articles/index.md Adds dotnet test overview to docs landing page.
docs/articles/dotnet-test-windows.md Adds Windows-specific dotnet test guide.
docs/articles/dotnet-test-macos.md Adds macOS Catalyst-specific guide.
docs/articles/dotnet-test-ios.md Adds iOS simulator-specific guide.
docs/articles/dotnet-test-android.md Adds Android-specific guide.
docs/articles/ci-pipeline.md Documents new CI paths.
DeviceRunners.slnx Adds the new targets package project to the solution.
.github/workflows/test-dotnet-test-windows/action.yml Adds GitHub Action for Windows MSIX dotnet test.
.github/workflows/test-dotnet-test-windows-exe/action.yml Adds GitHub Action for Windows EXE dotnet test.
.github/workflows/test-dotnet-test-macos/action.yml Adds GitHub Action for macOS Catalyst dotnet test.
.github/workflows/test-dotnet-test-ios/action.yml Adds GitHub Action for iOS simulator dotnet test.
.github/workflows/test-dotnet-test-android-linux/action.yml Adds GitHub Action for Android emulator dotnet test.
.github/workflows/ci.yml Wires new GitHub Actions jobs into CI.
.azure/templates/test-dotnet-test-windows.yml Adds Azure template for Windows MSIX dotnet test.
.azure/templates/test-dotnet-test-windows-exe.yml Adds Azure template for Windows EXE dotnet test.
.azure/templates/test-dotnet-test-macos.yml Adds Azure template for macOS Catalyst dotnet test.
.azure/templates/test-dotnet-test-ios.yml Adds Azure template for iOS simulator dotnet test.
.azure/templates/test-dotnet-test-android.yml Adds Azure template for Android emulator dotnet test.
.azure/azure-pipelines.yml Wires new Azure dotnet test stage into CI.

mattleibow and others added 2 commits May 14, 2026 01:37
- Quote CLI tool path in Exec command to handle spaces in NuGet cache
  paths (e.g. user profiles with spaces)
- Use AppPort/AppHostNames settings in Windows MSIX launch args instead
  of hardcoding Port and localhost, matching GetAppEnvironmentVariables
- Remove TestingMode=NonInteractiveVisual from dotnet test Windows MSIX
  workflows to avoid duplicate TCP channels (AddCliConfiguration already
  configures the channel via CLI args)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Android env vars (DEVICE_RUNNERS_AUTORUN etc.) were always baked into
the APK when the Testing.Targets package was referenced, which meant
normal builds from the IDE or dotnet build would auto-run tests.

Now the injection is gated on DeviceRunnersTestRun=true, which is set
by the _DeviceRunnersMarkTestRun target that runs before Build in the
VSTest chain. Normal builds skip injection entirely.

Users can also pass -p:DeviceRunnersTestRun=true for manual CLI testing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mattleibow and others added 11 commits May 14, 2026 01:58
EmbedAssembliesIntoApk was always set to true when the Testing.Targets
package was referenced, disabling Fast Deployment for ALL Android builds.
This made normal debug builds from the IDE slow.

Now it's set in _DeviceRunnersMarkTestRun (which only runs during
dotnet test), so normal dotnet build / dotnet run / IDE builds keep
Fast Deployment enabled for fast iteration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
EmbedAssembliesIntoApk is evaluated at project load time by the Android
SDK, so setting it inside a target (_DeviceRunnersMarkTestRun) was too
late — the APK was built without embedded assemblies and the app crashed.

Moved it back to the .props file (unconditional for Android when the
package is referenced). This affects all Android builds but is required
until #125 (use platform Run/Install targets) is implemented.

The env var injection (DeviceRunnersTestRun) stays target-gated since
_GenerateEnvironmentFiles is a target that runs after _DeviceRunnersMarkTestRun.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of setting EmbedAssembliesIntoApk=true and having the CLI do
adb install (which breaks Fast Deployment), use the Android SDK's own
Install target which handles both regular install and Fast Deployment.

The CLI now receives --package instead of --app, so it just launches
the already-installed app by package name and listens for TCP results.

This eliminates the EmbedAssembliesIntoApk override entirely, keeping
debug builds fast (Fast Deployment enabled) while dotnet test works.

Verified locally: 44 tests (36 passed, 5 failed, 3 skipped) with
Fast Deployment — no embedded assemblies needed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Removed EmbedAssembliesIntoApk documentation and Fast Deployment
troubleshooting. Replaced with explanation of the platform Install
target approach where Fast Deployment works normally.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1. macOS docs: removed false claims about process handle retention,
   exit code capture, and stdout/stderr redirection (not implemented)
2. AzDO templates: added artifact cleanup (rm bin/obj) to Android,
   iOS, and macOS dotnet test templates (were uploading GB of output)
3. ci-pipeline.md: swapped Windows dotnet test support — loose MSIX
   is supported (✅), packaged MSIX is not (N/A)
4. WriteJson: replaced NRE-prone cast with explicit error message when
   a CommandResult subtype isn't registered in CliJsonContext
5. winapp.exe: only bundled for Windows RIDs, saving ~190MB from the
   NuGet package (was duplicated across all 6 RID folders)
6. Pack Exec: use $(DotNetHostPath) instead of bare 'dotnet' to work
   when dotnet is not on PATH

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DotNetHostPath is not always set (empty on Windows CI), causing pack to
fail with exit code 9009. Fall back to bare 'dotnet' when not available.

winapp.exe condition was too strict — only matched explicit win-* RIDs
but not the common case of dotnet build/tool install without a RID on
Windows. Added fallback for IsOSPlatform('windows') when no RID is set.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both changes were based on unvalidated review suggestions:
- DotNetHostPath is not reliably set; bare dotnet works fine in CI
- winapp.exe CopyToOutputDirectory is unconditional by design; the
  duplication in the NuGet package is from dotnet publish copying all
  output files to each RID folder, not from the ItemGroup

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dotnet publish copies all output files (including winapp/) to every RID
folder. winapp.exe is only used on Windows but was duplicated across all
6 RIDs, adding ~190MB to the package. RemoveDir after publish cleans
the non-Windows folders. Package size: 169MB -> 90MB.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CLI csproj now conditions winapp.exe copy on RuntimeIdentifier:
- No RID (dotnet tool install): both archs to winapp/<arch>/winapp.exe
- win-x64: just win-x64 to winapp/winapp.exe
- win-arm64: just win-arm64 to winapp/winapp.exe
- Non-Windows RID: nothing copied

WinAppService.GetWinAppPath() checks both layouts (direct then arch subfolder).

Package size: 169MB -> 70MB (only 1 winapp.exe per Windows RID, none for others).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
No subfolder needed — winapp.exe sits directly alongside the CLI binary
in tools/win-x64/ and tools/win-arm64/. GetWinAppPath() checks next to
the exe first, then falls back to winapp/<arch>/ for dotnet tool installs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mattleibow
mattleibow merged commit e39f5d8 into main May 14, 2026
24 checks passed
@mattleibow
mattleibow deleted the mattleibow/investigate-dotnet-test-mobile branch May 14, 2026 08:09
mattleibow added a commit that referenced this pull request May 14, 2026
Resolve 16 merge conflicts from main (which now includes PR #106).
Fix CS0067 warning-as-error for CanExecuteChanged in ViewModelBindingTests.
Fix reflection discoverer test count (PreEnumerateTheories=false yields 6 not 8).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mattleibow added a commit that referenced this pull request May 16, 2026
- Merge main (Testing.Targets #106, Blazor browser runner #110)
- Add browser/WASM platform detection using UsingMicrosoftNETSdkBlazorWebAssembly
- Add index.html.template for Blazor WebAssembly projects
- Update BrowserTests sample to use Bootstrap for index.html generation
- Fix platform detection to use GetTargetPlatformIdentifier() function
- Condition icons/splash to MAUI-only (not browser)
- Remove unused AndroidManifest property (SDK provides default + overlay)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants