Repository navigation
Enable dotnet test for device platforms via DeviceRunners.Testing.Targets - #106
Merged
Merged
Conversation
….Targets Introduces a new MSBuild targets NuGet package that makes dotnet test work for Android and iOS MAUI test apps with a single command: dotnet test MyTests.csproj -f net10.0-android New: DeviceRunners.Testing.Targets NuGet package - build/DeviceRunners.Testing.Targets.props injects DEVICE_RUNNERS_AUTORUN, DEVICE_RUNNERS_PORT, DEVICE_RUNNERS_HOST_NAMES into the app bundle at build time via AndroidEnvironmentVariables / iOSEnvironmentVariables MSBuild items - build/DeviceRunners.Testing.Targets.targets overrides the VSTest target for Android/iOS to invoke the bundled CLI tool; selects the correct self-contained binary for the host RID (win-x64, linux-x64, osx-x64, osx-arm64) - PublishCliTools MSBuild target publishes the CLI self-contained + trimmed for all 4 RIDs before packing the NuGet package New: UseTestRunnerEnvironment() extension method Replaces the compile-time #if MODE_NON_INTERACTIVE_VISUAL block with a runtime env var check. When DEVICE_RUNNERS_AUTORUN is set (injected by the .targets file), the app auto-enables headless mode with TCP result streaming. Updated: sample test project - Removes #if MODE_NON_INTERACTIVE_VISUAL and TestingMode/MODE_NON_INTERACTIVE_VISUAL - Calls UseTestRunnerEnvironment() instead (no-op when env var is absent) - Imports DeviceRunners.Testing.Targets .props and .targets directly for in-repo use Updated: CI workflows (test-tcp-android-linux, test-tcp-ios) - Replaces 2-step dotnet publish + device-runners CLI with a single dotnet test call - Adds PublishCliTools step to build bundled native binaries before dotnet test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dotnet pack triggers PublishCliTools via BeforeTargets=GenerateNuspec, which is the natural way to prepare the NuGet package and its bundled binaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… support - Replace non-existent AndroidEnvironmentVariables ItemGroup with correct _GeneratedAndroidEnvironment items in a BeforeTargets=_GenerateEnvironmentFiles target. The Android SDK compiles these into the native libxamarin-app.so so that Mono makes them available via Environment.GetEnvironmentVariable() at app startup. - Use %3B to escape semicolons in HOST_NAMES value, preventing MSBuild from treating the list separator as an item separator. - Update iOS LaunchAppAsync to accept environmentVariables parameter and inject them using the xcrun simctl SIMCTL_CHILD_* mechanism when provided. - Update iOS TestCommand to pass DEVICE_RUNNERS_* env vars at app launch time (matching how Windows and macOS CLI commands already inject them). - Add DeviceRunners.Cli FDD publish output to tools/ for in-repo testing. Android dotnet test now works end-to-end: 44 tests, 36 pass, 5 intentional fail, TRX written, correct non-zero exit code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix inaccurate comment in MacOSService: 'open --env' does support env vars; direct launch is used for process handle retention and simplicity - Read stderr on xcrun simctl launch failure for better CI diagnostics - Make iOS CI dotnet pack use 'release' consistently with macOS CI - Fix .app bundle discovery: MSBuild Include glob (*.app/) does not reliably match directories on macOS; use Directory.GetDirectories instead — verified working for both iOS and macOS Catalyst - Remove committed binary DLLs from tools/ and add .gitignore so they are regenerated by dotnet pack (PublishCliTools target) - Validated end-to-end: dotnet test works for macOS Catalyst (44 tests, TRX generated) and iOS simulator (44 tests, TRX generated) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rgets Mirror the same simplification already applied to GitHub Actions: - Replace 'dotnet tool install + dotnet publish + device-runners <plat> test' with 'dotnet pack Testing.Targets + dotnet test' - Remove TestingMode=NonInteractiveVisual (replaced by env var auto-config) - Remove manual .app/.apk/.exe discovery steps - All four platforms updated: Android, iOS, macOS, Windows Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- global.json: pin SDK to 10.0.100 with rollForward=latestFeature so any 10.0.* SDK satisfies the requirement. - Directory.Build.props: set ValidateXcodeVersion=false so minor Xcode version mismatches don't break builds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of the raw MSB3073 'command exited with code 1' error:
- All Exec tasks now use IgnoreExitCode=true and capture the exit code
- New _DeviceRunnersReportResults target parses the TRX via XmlPeek
and emits a dotnet-test-style summary line, e.g.:
Failed! - Failed: 5, Passed: 36, Skipped: 3, Total: 44
- On failure, emits a clean Error task message instead of MSB3073
- Fallback error for the case where CLI crashes before writing TRX
- Fix: count skipped via UnitTestResult[@outcome='NotExecuted'] elements
because xunit/nunit skipped tests don't increment Counters/@notExecuted
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…gle Exec
Replace CallTarget with proper DependsOnTargets chains and eliminate
duplicate Exec blocks. New target hierarchy:
VSTest
→ _DeviceRunnersCheckCli (fail fast before building)
→ Build
→ _DeviceRunnersRunTests
→ _DeviceRunnersPrepareArgs
→ _DeviceRunnersCommonArgs (TRX path, device flag, shared CLI flags)
→ _DeviceRunnersAndroidArgs (Condition=android)
→ _DeviceRunnersiOSArgs (Condition=ios)
→ _DeviceRunnersMacOSArgs (Condition=maccatalyst)
→ _DeviceRunnersWindowsArgs (Condition=windows)
→ _DeviceRunnersExecTests (single Exec, consumes $(_DeviceRunnersCliArgs))
→ _DeviceRunnersReportResults (XmlPeek TRX, emit summary)
Each platform arg target sets $(_DeviceRunnersCliArgs); the optional
$(_DeviceRunnersDeviceArg) is built once in _DeviceRunnersCommonArgs and
appended where the CLI supports it (Android, iOS). No duplicate Exec pairs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace CallTarget usage with proper DependsOnTargets chains - VSTest -> _DeviceRunnersCheckCli;Build;_DeviceRunnersRunTests - _DeviceRunnersRunTests -> _DeviceRunnersPrepareArgs;_DeviceRunnersExecTests;_DeviceRunnersReportResults - _DeviceRunnersPrepareArgs -> _DeviceRunnersCommonArgs + 4 platform targets - Introduce _DeviceRunnersPlatform and _DeviceRunnersIsSupportedPlatform properties so all conditions use property comparisons instead of repeated function calls - Single _DeviceRunnersExecTests consumes fully-assembled $(_DeviceRunnersCliArgs) - Single _DeviceRunnersDeviceArg with leading space for clean concatenation - ASCII-only comments (no box-drawing chars, no -- inside XML comments) Validated: dotnet test macOS Catalyst - 44 tests (36 pass, 5 fail, 3 skip) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Emit the standard dotnet test two-line format: Results File: /path/to/test-results.trx Test summary: total: 44, failed: 5, succeeded: 36, skipped: 3, duration: 9.3s Changes: - Output 'Results File:' line first (before summary, not after) - Use 'succeeded' instead of 'passed' to match dotnet test terminology - Read scalar TRX counters as PropertyName (not ItemName) for clean interpolation - Compute duration from TRX Times/@start and Times/@finish attributes - Keep bare 'Test run failed.' Error as the MSBuild failure signal Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds Code="TESTERROR" to the two test-failure Error tasks so the output matches what dotnet test emits for failing tests: error TESTERROR: Test run failed. Tools that parse dotnet test output keyed on the TESTERROR code will now correctly recognise device test failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dotnet CLI terminal logger (default in .NET 8+) filters MSBuild
Message tasks, so Results File and Test summary were invisible at
default verbosity. Fix by embedding the full summary in the TESTERROR
error message which is always surfaced:
error TESTERROR: Test run failed: 5 of 44 tests failed
(36 succeeded, 3 skipped, duration: 9.7s). Results File: /path/...
The Message tasks are kept for --tl:off / --verbosity normal users
where they render as the clean two-line dotnet test format:
Results File: /path/to/test-results.trx
Test summary: total: 44, failed: 5, succeeded: 36, skipped: 3, duration: 9.7s
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use %0A (MSBuild newline escape) to split the error into two lines matching the standard dotnet test output format: error TESTERROR: Results File: /path/to/test-results.trx error TESTERROR: Test summary: total: 44, failed: 5, succeeded: 36, skipped: 3, duration: 18.5s Both lines are always visible via the terminal logger. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Test classes are discovered via reflection at runtime. Without explicit trimmer roots the iOS/Android linker strips the referenced test assemblies (MauiLibrary.XunitTests, Library.NUnitTests), causing only tests in the main DeviceTests assembly to run (~8 instead of ~44). TrimmerRootAssembly tells the linker to keep the full assembly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The event stream protocol sends begin/end events to bracket the test run. If the TCP connection closes after receiving results but without the end event, the app crashed or was killed mid-run. Previously this produced partial results with no indication of a crash. Now the CLI explicitly reports: The application appears to have crashed during the test run. Only 8 test result(s) were received before the connection was lost. Check the device log for crash details. And forces a non-zero exit code even if all received tests passed, so CI pipelines correctly fail. Changes: - EventStreamService: add HasStarted and HasEnded tracking - BaseTestCommand: check for begin-without-end after listener completes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CLI exit codes: 0 = all tests passed 1 = test failures 2 = app crashed (begin event received, results streamed, but no end event) MSBuild targets check the exit code directly (no file reading needed): exit code 2 + TRX: 'Test summary: ... (incomplete: app crashed)' exit code 2 + no TRX: 'The application crashed before producing any test results' exit code 1 + TRX: 'Test summary: ...' (normal failure) Validated: iOS: 8 tests, crash detected, '(incomplete: app crashed)' shown macOS: 44 tests, no crash, normal TESTERROR shown Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the (int testFailures, string? testResults) tuple with a protected record TestListenerResult(int FailedCount, string? ResultsFile, bool Crashed). This makes the crash detection explicit via a bool rather than a sentinel value, and gives clear property names at all call sites. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Centralise exit code mapping and success logic on the record itself: - ToExitCode(): 0 = success, 1 = test failures, 2 = app crashed - Success: true when no failures and no crash All callers now use listener.ToExitCode() and listener.Success instead of duplicating the conditional logic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MAUI 10.0.20 had a NullReferenceException in Button.LayoutButton when the platform button was null during a layout pass on the UI thread. This crashed the app mid-test-run on iOS, producing only partial results. The fix (dotnet/maui#35284, fixing dotnet/maui#31048) adds a null guard in LayoutButton. MAUI 10.0.60 includes this fix. Results: iOS now completes all 44 tests (previously crashed at 8). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the existing AppleDev predicate parameter on GetLogsPlainAsync to
filter logs to just the app process:
predicate: 'process == "DeviceTestingKitApp.DeviceTests"'
Results:
Before: 17,000 lines of OS-level noise, no managed exceptions
After: 190 lines with all PASS/FAIL output and full managed stack
traces when the app crashes (NullReferenceException, etc.)
Also restores MAUI 10.0.60 (was temporarily reverted for testing).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AppleDev 0.8.10 adds SimCtl.LaunchAppAsync overload with env vars, so we can replace the manual ProcessStartInfo + SIMCTL_CHILD_* workaround with the native API call. AndroidSdk 0.35.1 is the latest stable release. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Testing.Targets .props defaults WindowsPackageType=None (for dotnet test unpackaged mode). The TCP Windows packaged and XHarness Windows workflows need MSIX packaging, so pass -p:WindowsPackageType=MSIX explicitly. Fixes: TCP Windows (GH+AzDO), XHarness Windows (GH+AzDO) CI failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New articles: - using-dotnet-test.md: Main guide covering setup, configuration, output format, crash detection, MSBuild properties, and how it works - dotnet-test-android.md: Emulator setup, env var injection, APK config - dotnet-test-ios.md: Simulator setup, device logs, crash troubleshooting - dotnet-test-macos.md: Direct launch, no simulator needed - dotnet-test-windows.md: Unpackaged EXE mode, WindowsPackageType Updated: - toc.yml: 'dotnet test (Recommended)' section at top, CLI renamed to 'Advanced', XHarness renamed to 'Legacy' (from 'CLI Runners') - index.md: Getting Started leads with dotnet test - README.md: dotnet test as primary, links to docs site - technical-architecture-overview.md: Testing.Targets package section with MSBuild target chain, exit code protocol, package structure - ci-pipeline.md: dotnet test CI examples for GH Actions and AzDO Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove WindowsPackageType=None default from .props entirely. Instead, the _DeviceRunnersWindowsArgs target detects what the build produced: - If .exe exists (unpackaged): uses 'windows test --app path.exe' - If .msix exists (packaged): uses 'windows test --app path.msix' - If neither: clear error message This means dotnet publish for MSIX packaging is no longer broken by the Testing.Targets package. CI workflows no longer need -p:WindowsPackageType overrides. Also search both artifacts/ and sample/ for .msix in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mattleibow
force-pushed
the
mattleibow/investigate-dotnet-test-mobile
branch
from
May 8, 2026 17:34
932a033 to
aa1cb54
Compare
Create DeviceRunners.VisualRunners.Core (net9.0;net10.0 only) with the cross-platform testing types that the CLI needs: Moved from VisualRunners: - Testing/ITestResultInfo, ITestCaseInfo, ITestAssemblyInfo, ITestAssemblyConfiguration, TestResultStatus - Testing/Channels/IResultChannel, IResultChannelFormatter, FileResultChannel, FileResultChannelOptions, TestResultEvent, EventStreamFormatter, TextResultChannelFormatter, TrxResultChannelFormatter, TextWriterResultChannel The CLI now references VisualRunners.Core instead of VisualRunners, breaking the dependency on multi-target mobile TFMs. This unblocks self-contained and AOT publish (no more Mono runtime pack errors). VisualRunners references VisualRunners.Core transitively, so all existing consumers are unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add TrimmerRootAssembly for Spectre.Console.Cli and DeviceRunners.Cli to preserve types discovered via reflection (command settings classes). Spectre.Console.Cli is explicitly not trim-safe (spectreconsole/spectre.console#1690) so the entire assembly must be rooted. Suppress trim analysis warnings since Spectre.Console.Cli generates unavoidable IL2026/IL2090 warnings that are covered by the rooting. Result: 19MB self-contained single-file binary (down from 21MB multi-file). dotnet publish -r osx-arm64 --self-contained -p:PublishSingleFile=true Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CLI csproj now has SelfContained, PublishSingleFile, PublishTrimmed, and RuntimeIdentifiers (osx-arm64;win-x64) set by default. Plain 'dotnet publish -r osx-arm64' produces a 19MB single-file binary. The NuGet package PublishCliTools target overrides these back to framework-dependent (SelfContained=false, PublishSingleFile=false, PublishTrimmed=false) since the bundled tool runs via 'dotnet <dll>'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NuGet package now ships self-contained, trimmed, single-file native binaries for each platform in tools/<rid>/: tools/osx-arm64/DeviceRunners.Cli (~20 MB) tools/win-x64/DeviceRunners.Cli.exe (~20 MB) tools/linux-x64/DeviceRunners.Cli (~20 MB) The MSBuild targets detect the host OS and select the right binary. Falls back to framework-dependent DLL at tools/DeviceRunners.Cli.dll for in-repo dev or older packages. Changes: - CLI csproj: add JsonSerializerIsReflectionEnabledByDefault, linux-x64 RID - Testing.Targets csproj: PublishCliTools publishes 3 RIDs (self-contained) - .targets: detect host OS -> pick tools/<rid>/DeviceRunners.Cli[.exe] Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The static <None Include="tools\**"> glob was evaluated at project load time, before PublishCliTools populated the tools/ directory. This meant a clean pack produced a nupkg with build/ but zero tools/ files. Fix: move the tools glob into the PublishCliTools target as a _PackageFiles item, so it runs AFTER the per-RID publishes complete. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dotnet test Android workflow was missing several setup steps that the working TCP workflow has: - Java 21 installation (required by Android SDK/emulator) - mkdir for AVD directory (ANDROID_AVD_HOME must exist) - ANDROID_EMULATOR_HOME env var - AndroidSdkDirectory MSBuild property for build These omissions caused the emulator to not boot or the build to not find the Android SDK, resulting in the job hanging indefinitely. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e-dotnet-test-mobile # Conflicts: # sample/test/DeviceTestingKitApp.DeviceTests/DeviceTestingKitApp.DeviceTests.csproj
PR #124 enabled TreatWarningsAsErrors globally. The CLI project has known trim warnings (IL2026, IL2090) from reflection-based JSON/XML serialization and Spectre.Console.Cli. These are suppressed during publish via SuppressTrimAnalysisWarnings but TreatWarningsAsErrors promoted them to errors during normal builds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace reflection-based JsonSerializer calls with source-generated CliJsonContext for all CommandResult types and Dictionary<string,object>. This eliminates IL2026 warnings for JSON serialization and removes the need for JsonSerializerIsReflectionEnabledByDefault. XML serialization (XmlSerializer) has no source-gen equivalent, so those call sites use #pragma to suppress IL2026. Reflection-based text output (GetProperties) uses DynamicallyAccessedMembers annotations to satisfy IL2090. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restored comments that were accidentally removed during the source-gen refactor. Removed the dead JSON fallback path since all CommandResult types are registered in CliJsonContext. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DeviceRunners.Testing.Targets has IncludeBuildOutput=false because it ships only build/ and tools/ content (no lib/ assemblies). NU5128 warns about the mismatch, which TreatWarningsAsErrors promotes to an error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new DeviceRunners.Testing.Targets package to enable dotnet test for device-platform MAUI test apps by overriding VSTest, launching the DeviceRunners CLI, collecting TCP results, and reporting TRX output.
Changes:
- Adds MSBuild props/targets and packaging support for bundled per-RID CLI binaries.
- Adds CLI/visual-runner support for environment/argument-driven test autorun and trim-safe JSON serialization.
- Adds sample, CI, and documentation coverage for Android, iOS, macOS Catalyst, and Windows
dotnet testworkflows.
Reviewed changes
Copilot reviewed 37 out of 37 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
src/DeviceRunners.VisualRunners/VisualTestRunnerConfigurationBuilderExtensions.cs |
Adds AddCliConfiguration() for env/CLI argument configuration. |
src/DeviceRunners.Testing.Targets/DeviceRunners.Testing.Targets.csproj |
Defines targets-only NuGet package and CLI publish target. |
src/DeviceRunners.Testing.Targets/build/DeviceRunners.Testing.Targets.targets |
Adds custom VSTest orchestration and platform-specific run args. |
src/DeviceRunners.Testing.Targets/build/DeviceRunners.Testing.Targets.props |
Adds default MSBuild properties and disables default VSTest/MTP behavior. |
src/DeviceRunners.Testing.Targets/.gitignore |
Ignores generated packaged CLI tools. |
src/DeviceRunners.Cli/Services/MacOSService.cs |
Switches plist JSON deserialization to source-generated context. |
src/DeviceRunners.Cli/Models/CliJsonContext.cs |
Adds source-generation metadata for CLI JSON output. |
src/DeviceRunners.Cli/DeviceRunners.Cli.csproj |
Adds trim/single-file publish settings for packaged CLI binaries. |
src/DeviceRunners.Cli/Commands/Windows/TestCommand.cs |
Passes DeviceRunners CLI arguments to loose MSIX launches. |
src/DeviceRunners.Cli/Commands/BaseCommand.cs |
Updates JSON/XML/text output paths for trimming. |
src/DeviceRunners.Cli/Commands/BaseAsyncCommand.cs |
Updates async command output serialization for trimming. |
sample/test/DeviceTestingKitApp.DeviceTests/MauiProgram.cs |
Uses AddCliConfiguration() in the sample app. |
sample/test/DeviceTestingKitApp.DeviceTests/DeviceTestingKitApp.DeviceTests.csproj |
Imports new in-repo testing targets for sample use. |
README.md |
Promotes dotnet test workflow and updates runner guidance. |
global.json |
Pins the repository SDK to .NET 10. |
docs/articles/using-dotnet-test.md |
Adds primary dotnet test documentation. |
docs/articles/toc.yml |
Adds dotnet test docs to the TOC. |
docs/articles/technical-architecture-overview.md |
Documents the new targets package architecture. |
docs/articles/index.md |
Adds dotnet test overview to docs landing page. |
docs/articles/dotnet-test-windows.md |
Adds Windows-specific dotnet test guide. |
docs/articles/dotnet-test-macos.md |
Adds macOS Catalyst-specific guide. |
docs/articles/dotnet-test-ios.md |
Adds iOS simulator-specific guide. |
docs/articles/dotnet-test-android.md |
Adds Android-specific guide. |
docs/articles/ci-pipeline.md |
Documents new CI paths. |
DeviceRunners.slnx |
Adds the new targets package project to the solution. |
.github/workflows/test-dotnet-test-windows/action.yml |
Adds GitHub Action for Windows MSIX dotnet test. |
.github/workflows/test-dotnet-test-windows-exe/action.yml |
Adds GitHub Action for Windows EXE dotnet test. |
.github/workflows/test-dotnet-test-macos/action.yml |
Adds GitHub Action for macOS Catalyst dotnet test. |
.github/workflows/test-dotnet-test-ios/action.yml |
Adds GitHub Action for iOS simulator dotnet test. |
.github/workflows/test-dotnet-test-android-linux/action.yml |
Adds GitHub Action for Android emulator dotnet test. |
.github/workflows/ci.yml |
Wires new GitHub Actions jobs into CI. |
.azure/templates/test-dotnet-test-windows.yml |
Adds Azure template for Windows MSIX dotnet test. |
.azure/templates/test-dotnet-test-windows-exe.yml |
Adds Azure template for Windows EXE dotnet test. |
.azure/templates/test-dotnet-test-macos.yml |
Adds Azure template for macOS Catalyst dotnet test. |
.azure/templates/test-dotnet-test-ios.yml |
Adds Azure template for iOS simulator dotnet test. |
.azure/templates/test-dotnet-test-android.yml |
Adds Azure template for Android emulator dotnet test. |
.azure/azure-pipelines.yml |
Wires new Azure dotnet test stage into CI. |
- Quote CLI tool path in Exec command to handle spaces in NuGet cache paths (e.g. user profiles with spaces) - Use AppPort/AppHostNames settings in Windows MSIX launch args instead of hardcoding Port and localhost, matching GetAppEnvironmentVariables - Remove TestingMode=NonInteractiveVisual from dotnet test Windows MSIX workflows to avoid duplicate TCP channels (AddCliConfiguration already configures the channel via CLI args) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Android env vars (DEVICE_RUNNERS_AUTORUN etc.) were always baked into the APK when the Testing.Targets package was referenced, which meant normal builds from the IDE or dotnet build would auto-run tests. Now the injection is gated on DeviceRunnersTestRun=true, which is set by the _DeviceRunnersMarkTestRun target that runs before Build in the VSTest chain. Normal builds skip injection entirely. Users can also pass -p:DeviceRunnersTestRun=true for manual CLI testing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
EmbedAssembliesIntoApk was always set to true when the Testing.Targets package was referenced, disabling Fast Deployment for ALL Android builds. This made normal debug builds from the IDE slow. Now it's set in _DeviceRunnersMarkTestRun (which only runs during dotnet test), so normal dotnet build / dotnet run / IDE builds keep Fast Deployment enabled for fast iteration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
EmbedAssembliesIntoApk is evaluated at project load time by the Android SDK, so setting it inside a target (_DeviceRunnersMarkTestRun) was too late — the APK was built without embedded assemblies and the app crashed. Moved it back to the .props file (unconditional for Android when the package is referenced). This affects all Android builds but is required until #125 (use platform Run/Install targets) is implemented. The env var injection (DeviceRunnersTestRun) stays target-gated since _GenerateEnvironmentFiles is a target that runs after _DeviceRunnersMarkTestRun. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of setting EmbedAssembliesIntoApk=true and having the CLI do adb install (which breaks Fast Deployment), use the Android SDK's own Install target which handles both regular install and Fast Deployment. The CLI now receives --package instead of --app, so it just launches the already-installed app by package name and listens for TCP results. This eliminates the EmbedAssembliesIntoApk override entirely, keeping debug builds fast (Fast Deployment enabled) while dotnet test works. Verified locally: 44 tests (36 passed, 5 failed, 3 skipped) with Fast Deployment — no embedded assemblies needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Removed EmbedAssembliesIntoApk documentation and Fast Deployment troubleshooting. Replaced with explanation of the platform Install target approach where Fast Deployment works normally. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1. macOS docs: removed false claims about process handle retention, exit code capture, and stdout/stderr redirection (not implemented) 2. AzDO templates: added artifact cleanup (rm bin/obj) to Android, iOS, and macOS dotnet test templates (were uploading GB of output) 3. ci-pipeline.md: swapped Windows dotnet test support — loose MSIX is supported (✅), packaged MSIX is not (N/A) 4. WriteJson: replaced NRE-prone cast with explicit error message when a CommandResult subtype isn't registered in CliJsonContext 5. winapp.exe: only bundled for Windows RIDs, saving ~190MB from the NuGet package (was duplicated across all 6 RID folders) 6. Pack Exec: use $(DotNetHostPath) instead of bare 'dotnet' to work when dotnet is not on PATH Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DotNetHostPath is not always set (empty on Windows CI), causing pack to
fail with exit code 9009. Fall back to bare 'dotnet' when not available.
winapp.exe condition was too strict — only matched explicit win-* RIDs
but not the common case of dotnet build/tool install without a RID on
Windows. Added fallback for IsOSPlatform('windows') when no RID is set.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both changes were based on unvalidated review suggestions: - DotNetHostPath is not reliably set; bare dotnet works fine in CI - winapp.exe CopyToOutputDirectory is unconditional by design; the duplication in the NuGet package is from dotnet publish copying all output files to each RID folder, not from the ItemGroup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dotnet publish copies all output files (including winapp/) to every RID folder. winapp.exe is only used on Windows but was duplicated across all 6 RIDs, adding ~190MB to the package. RemoveDir after publish cleans the non-Windows folders. Package size: 169MB -> 90MB. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CLI csproj now conditions winapp.exe copy on RuntimeIdentifier: - No RID (dotnet tool install): both archs to winapp/<arch>/winapp.exe - win-x64: just win-x64 to winapp/winapp.exe - win-arm64: just win-arm64 to winapp/winapp.exe - Non-Windows RID: nothing copied WinAppService.GetWinAppPath() checks both layouts (direct then arch subfolder). Package size: 169MB -> 70MB (only 1 winapp.exe per Windows RID, none for others). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
No subfolder needed — winapp.exe sits directly alongside the CLI binary in tools/win-x64/ and tools/win-arm64/. GetWinAppPath() checks next to the exe first, then falls back to winapp/<arch>/ for dotnet tool installs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mattleibow
added a commit
that referenced
this pull request
May 14, 2026
Resolve 16 merge conflicts from main (which now includes PR #106). Fix CS0067 warning-as-error for CanExecuteChanged in ViewModelBindingTests. Fix reflection discoverer test count (PreEnumerateTheories=false yields 6 not 8). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ari4ka
approved these changes
May 14, 2026
mattleibow
added a commit
that referenced
this pull request
May 16, 2026
- Merge main (Testing.Targets #106, Blazor browser runner #110) - Add browser/WASM platform detection using UsingMicrosoftNETSdkBlazorWebAssembly - Add index.html.template for Blazor WebAssembly projects - Update BrowserTests sample to use Bootstrap for index.html generation - Fix platform detection to use GetTargetPlatformIdentifier() function - Condition icons/splash to MAUI-only (not browser) - Remove unused AndroidManifest property (SDK provides default + overlay) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds first-class
dotnet testsupport for device test apps on Android, iOS, macOS Catalyst, and Windows via a newDeviceRunners.Testing.TargetsNuGet package.Quick start
1. Add the package:
2. Configure your app (in
MauiProgram.cs):3. Run tests:
How it works
The package replaces the default
VSTesttarget for device TFMs and orchestrates: Build → Deploy → Launch → Collect results → Report.It ships self-contained, trimmed, single-file CLI binaries per RID (~20 MB each):
osx-arm64,osx-x64,win-x64,win-arm64,linux-x64,linux-arm64. The targets auto-detect the host OS and architecture at build time.Configuration injection
Each platform uses a different mechanism to pass configuration to the app:
_GenerateEnvironmentFilesSimCtl.LaunchAppAsync(AppleDev 0.8.10+)ProcessStartInfo(direct process launch)ProcessStartInfo(direct process launch)winapp.exe --args(env vars can't be forwarded to packaged apps)AddCliConfiguration()reads env vars first, then falls back to--device-runners-*CLI arguments. When neither source providesDEVICE_RUNNERS_AUTORUN, it's a no-op and the visual runner behaves normally.Windows support
.exein build output-p:WindowsPackageType=None)AppxManifest.xmlin build outputwinapp.exeloose-file deploy (no publish/cert needed)Output
Standard
dotnet testformat:Exit codes: 0 = pass, 1 = test failures, 2 = app crashed (TCP connection lost without end event).
Configuration
DeviceRunnersPort16384DeviceRunnersConnectionTimeout120DeviceRunnersDataTimeout30DeviceRunnersDeviceDeviceRunnersBinCI strategy
Three independent test paths are validated per platform:
device-runners test --app ...test-tcp-*test-dotnet-test-*(new)test-xharness-*What's new
DeviceRunners.Testing.Targets— MSBuild package with.props,.targets, and per-RID CLI binariesAddCliConfiguration()— Reads env vars first, falls back to CLI args for MSIX supportPublishTrimmed,TrimmerRootAssembly,JsonSerializerIsReflectionEnabledByDefaulttest-dotnet-test-*workflows for GH Actions and AzDO (5 each)using-dotnet-test.mdplus per-platform guides; updated README, TOC, architecture overview, CI docsReviewer notes
Suggested review order:
src/DeviceRunners.Testing.Targets/build/*.targetsand*.props— core MSBuild targetssrc/DeviceRunners.Testing.Targets/DeviceRunners.Testing.Targets.csproj— PublishCliTools target with item batchingsrc/DeviceRunners.Cli/DeviceRunners.Cli.csproj— trim properties,PublishForTargetsPackagesrc/DeviceRunners.VisualRunners/VisualTestRunnerConfigurationBuilderExtensions.cs—AddCliConfiguration()sample/test/DeviceTestingKitApp.DeviceTests/— MauiProgram.cs + csproj changestest-dotnet-test-*)docs/articles/dotnet-test-*.md,using-dotnet-test.md)