Skip to content

feat: Add WebAssembly (Blazor) browser test runner with dotnet test integration - #110

Merged
mattleibow merged 146 commits into
mainfrom
mattleibow/wasm-support
May 15, 2026
Merged

mattleibow merged 146 commits into
mainfrom
mattleibow/wasm-support

Conversation

@mattleibow

@mattleibow mattleibow commented May 11, 2026 •

Copy link
Copy Markdown
Owner

What this PR does

This PR adds first-class WebAssembly (Blazor) support to DeviceRunners — the same level of test infrastructure that already exists for Android, iOS, macOS, and Windows. After this PR, you can:

  1. Run tests in a real browser — your xunit tests execute inside Chrome via Blazor WebAssembly
  2. Use dotnet test — standard dotnet test workflow, same as every other platform
  3. Use the visual runner — interactive Blazor UI for running/filtering tests during development
  4. Use the CLI — device-runners wasm test for scripted/CI scenarios
  5. Get TRX results — standard test results for CI integration

No XHarness, Playwright, Selenium, or npm required. Just .NET and Chrome.


How it works

Tests run inside the browser as a Blazor WebAssembly app. The test app discovers and executes xunit tests via reflection (since XunitFrontController requires filesystem access, which is unavailable in WASM). Results stream as NDJSON through Console.Out.

On the host side, the CLI launches a local web server, opens headless Chrome via Chrome DevTools Protocol, captures console output, and converts it to TRX.

┌─────────────────────────────────────────────────────┐
│ Browser (Blazor WASM)                               │
│                                                     │
│  Tests → WasmXunitRunner → ConsoleResultChannel     │
│            → EventStreamFormatter → Console.Out     │
└──────────────────────┬──────────────────────────────┘
                       │ NDJSON via CDP console capture
┌──────────────────────▼──────────────────────────────┐
│ Host (DeviceRunners CLI)                            │
│                                                     │
│  HttpListener server → Chrome (CDP WebSocket)       │
│            → EventStreamService → TRX file          │
└─────────────────────────────────────────────────────┘

For dotnet test, MSBuild targets handle everything automatically — publish the WASM app, launch the CLI, capture results.


Quick start

Using dotnet test (recommended):

dotnet test MyBlazorTests.csproj --logger "trx"

Using the CLI directly:

dotnet publish MyBlazorTests.csproj -c release
device-runners wasm test --app ./artifacts/publish/.../wwwroot

Using the visual runner (development):
Just run the Blazor WASM app — it opens in your browser with an interactive test UI.


What was added

Blazor Visual Runner (DeviceRunners.VisualRunners.Blazor)

A Razor Class Library that provides the same interactive test runner UI as the MAUI visual runner, but rendered in Blazor. Reuses the existing HomeViewModel, TestAssemblyViewModel, etc. — same ViewModels, different UI layer.

WASM Xunit Integration (DeviceRunners.VisualRunners.Xunit/Wasm/)

Reflection-based test discoverer and runner designed for the single-threaded WASM environment:

  • WasmXunitDiscoverer — uses SynchronousMessageBus (no threads in WASM)
  • WasmXunitAssemblyRunner — cooperative execution via Task.Yield() to avoid blocking the UI

CLI Commands

  • device-runners wasm test — end-to-end: serve app → headless Chrome → capture results → TRX
  • device-runners wasm serve — serve a published WASM app for manual browser testing

CLI Services

  • BrowserService — launches headless Chrome, connects via CDP WebSocket, captures Runtime.consoleAPICalled events. Cleans up temp directories, handles stderr, 30s launch timeout. WebGL works out of the box (--headless=new + SwiftShader).
  • WasmWebServerService — HttpListener static file server with WASM MIME types (application/wasm) and cross-origin isolation headers (COOP/COEP) required for SharedArrayBuffer.

dotnet test Integration

MSBuild targets in DeviceRunners.Testing.Targets detect Blazor WASM projects via UsingMicrosoftNETSdkBlazorWebAssembly and wire up the publish → CLI → TRX pipeline. Forwards $(VSTestLogger) and $(VSTestResultsDirectory) from dotnet test to match standard behavior.

Platform-Specific AddCliConfiguration

Split the CLI configuration into platform-specific extensions:

  • VisualRunners.Maui → AddCliConfiguration() — reads env vars/CLI args → TCP result channel
  • VisualRunners.Blazor → AddCliConfiguration(string url) — reads URL query string → Console result channel
  • Shared VisualRunners — building blocks only (AddTcpResultChannel, AddConsoleResultChannel, etc.)

Sample Apps

  • sample/src/DeviceTestingKitApp.BlazorLibrary/ — Blazor component library (Counter, Login)
  • sample/test/DeviceTestingKitApp.BlazorLibrary.XunitTests/ — bUnit component tests
  • sample/test/DeviceTestingKitApp.BrowserTests/ — Blazor WASM visual test runner app

Documentation

  • WASM dotnet test guide, WASM CLI guide
  • Updated architecture overview and platform matrix for WASM parity
  • All existing platform docs updated with consistent naming and ordering

CI/CD

  • GitHub Actions: WASM Browser + dotnet test WASM Browser jobs
  • Azure DevOps: WASM_Browser_Tests + dotnet_test_WASM_Browser jobs
  • Also fixed AzDO dotnet-test templates to match TCP templates (disk space cleanup, env var handling, arch validation)

Design decisions

Decision Rationale
Reflection-based runner XunitFrontController requires filesystem — unavailable in browser WASM
Chrome DevTools Protocol Self-contained, no external tools. Chrome is pre-installed on CI runners
SynchronousMessageBus + Task.Yield() WASM is single-threaded — cooperative yielding keeps the UI responsive
$(VSTestLogger) forwarding Match standard dotnet test behavior instead of hardcoding TRX
No --disable-gpu --headless=new supports WebGL via SwiftShader automatically
HttpListener (not Kestrel) Lightweight, no framework dependency for the static file server
NDJSON via Console.Out Same protocol as existing TCP runners — reuses EventStreamService pipeline

WebGL support

Works out of the box. Headless Chrome with --headless=new uses SwiftShader for GPU emulation. The --disable-gpu flag was intentionally removed.

mattleibow and others added 30 commits May 8, 2026 18:27
….Targets

Introduces a new MSBuild targets NuGet package that makes dotnet test work
for Android and iOS MAUI test apps with a single command:

  dotnet test MyTests.csproj -f net10.0-android

New: DeviceRunners.Testing.Targets NuGet package
- build/DeviceRunners.Testing.Targets.props injects DEVICE_RUNNERS_AUTORUN,
  DEVICE_RUNNERS_PORT, DEVICE_RUNNERS_HOST_NAMES into the app bundle at build
  time via AndroidEnvironmentVariables / iOSEnvironmentVariables MSBuild items
- build/DeviceRunners.Testing.Targets.targets overrides the VSTest target for
  Android/iOS to invoke the bundled CLI tool; selects the correct self-contained
  binary for the host RID (win-x64, linux-x64, osx-x64, osx-arm64)
- PublishCliTools MSBuild target publishes the CLI self-contained + trimmed for
  all 4 RIDs before packing the NuGet package

New: UseTestRunnerEnvironment() extension method
Replaces the compile-time #if MODE_NON_INTERACTIVE_VISUAL block with a runtime
env var check. When DEVICE_RUNNERS_AUTORUN is set (injected by the .targets file),
the app auto-enables headless mode with TCP result streaming.

Updated: sample test project
- Removes #if MODE_NON_INTERACTIVE_VISUAL and TestingMode/MODE_NON_INTERACTIVE_VISUAL
- Calls UseTestRunnerEnvironment() instead (no-op when env var is absent)
- Imports DeviceRunners.Testing.Targets .props and .targets directly for in-repo use

Updated: CI workflows (test-tcp-android-linux, test-tcp-ios)
- Replaces 2-step dotnet publish + device-runners CLI with a single dotnet test call
- Adds PublishCliTools step to build bundled native binaries before dotnet test

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dotnet pack triggers PublishCliTools via BeforeTargets=GenerateNuspec, which
is the natural way to prepare the NuGet package and its bundled binaries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… support

- Replace non-existent AndroidEnvironmentVariables ItemGroup with correct
  _GeneratedAndroidEnvironment items in a BeforeTargets=_GenerateEnvironmentFiles target.
  The Android SDK compiles these into the native libxamarin-app.so so that Mono
  makes them available via Environment.GetEnvironmentVariable() at app startup.

- Use %3B to escape semicolons in HOST_NAMES value, preventing MSBuild from
  treating the list separator as an item separator.

- Update iOS LaunchAppAsync to accept environmentVariables parameter and inject
  them using the xcrun simctl SIMCTL_CHILD_* mechanism when provided.

- Update iOS TestCommand to pass DEVICE_RUNNERS_* env vars at app launch time
  (matching how Windows and macOS CLI commands already inject them).

- Add DeviceRunners.Cli FDD publish output to tools/ for in-repo testing.

Android dotnet test now works end-to-end: 44 tests, 36 pass, 5 intentional fail,
TRX written, correct non-zero exit code.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix inaccurate comment in MacOSService: 'open --env' does support env
  vars; direct launch is used for process handle retention and simplicity
- Read stderr on xcrun simctl launch failure for better CI diagnostics
- Make iOS CI dotnet pack use 'release' consistently with macOS CI
- Fix .app bundle discovery: MSBuild Include glob (*.app/) does not
  reliably match directories on macOS; use Directory.GetDirectories
  instead — verified working for both iOS and macOS Catalyst
- Remove committed binary DLLs from tools/ and add .gitignore so
  they are regenerated by dotnet pack (PublishCliTools target)
- Validated end-to-end: dotnet test works for macOS Catalyst (44 tests,
  TRX generated) and iOS simulator (44 tests, TRX generated)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rgets

Mirror the same simplification already applied to GitHub Actions:
- Replace 'dotnet tool install + dotnet publish + device-runners <plat> test'
  with 'dotnet pack Testing.Targets + dotnet test'
- Remove TestingMode=NonInteractiveVisual (replaced by env var auto-config)
- Remove manual .app/.apk/.exe discovery steps
- All four platforms updated: Android, iOS, macOS, Windows

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- global.json: pin SDK to 10.0.100 with rollForward=latestFeature
  so any 10.0.* SDK satisfies the requirement.
- Directory.Build.props: set ValidateXcodeVersion=false so minor
  Xcode version mismatches don't break builds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of the raw MSB3073 'command exited with code 1' error:
- All Exec tasks now use IgnoreExitCode=true and capture the exit code
- New _DeviceRunnersReportResults target parses the TRX via XmlPeek
  and emits a dotnet-test-style summary line, e.g.:
    Failed! - Failed: 5, Passed: 36, Skipped: 3, Total: 44
- On failure, emits a clean Error task message instead of MSB3073
- Fallback error for the case where CLI crashes before writing TRX
- Fix: count skipped via UnitTestResult[@outcome='NotExecuted'] elements
  because xunit/nunit skipped tests don't increment Counters/@notExecuted

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…gle Exec

Replace CallTarget with proper DependsOnTargets chains and eliminate
duplicate Exec blocks. New target hierarchy:

  VSTest
    → _DeviceRunnersCheckCli  (fail fast before building)
    → Build
    → _DeviceRunnersRunTests
        → _DeviceRunnersPrepareArgs
            → _DeviceRunnersCommonArgs   (TRX path, device flag, shared CLI flags)
            → _DeviceRunnersAndroidArgs  (Condition=android)
            → _DeviceRunnersiOSArgs      (Condition=ios)
            → _DeviceRunnersMacOSArgs    (Condition=maccatalyst)
            → _DeviceRunnersWindowsArgs  (Condition=windows)
        → _DeviceRunnersExecTests        (single Exec, consumes $(_DeviceRunnersCliArgs))
        → _DeviceRunnersReportResults    (XmlPeek TRX, emit summary)

Each platform arg target sets $(_DeviceRunnersCliArgs); the optional
$(_DeviceRunnersDeviceArg) is built once in _DeviceRunnersCommonArgs and
appended where the CLI supports it (Android, iOS). No duplicate Exec pairs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace CallTarget usage with proper DependsOnTargets chains
- VSTest -> _DeviceRunnersCheckCli;Build;_DeviceRunnersRunTests
- _DeviceRunnersRunTests -> _DeviceRunnersPrepareArgs;_DeviceRunnersExecTests;_DeviceRunnersReportResults
- _DeviceRunnersPrepareArgs -> _DeviceRunnersCommonArgs + 4 platform targets
- Introduce _DeviceRunnersPlatform and _DeviceRunnersIsSupportedPlatform properties
  so all conditions use property comparisons instead of repeated function calls
- Single _DeviceRunnersExecTests consumes fully-assembled $(_DeviceRunnersCliArgs)
- Single _DeviceRunnersDeviceArg with leading space for clean concatenation
- ASCII-only comments (no box-drawing chars, no -- inside XML comments)

Validated: dotnet test macOS Catalyst - 44 tests (36 pass, 5 fail, 3 skip)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Emit the standard dotnet test two-line format:
  Results File: /path/to/test-results.trx
  Test summary: total: 44, failed: 5, succeeded: 36, skipped: 3, duration: 9.3s

Changes:
- Output 'Results File:' line first (before summary, not after)
- Use 'succeeded' instead of 'passed' to match dotnet test terminology
- Read scalar TRX counters as PropertyName (not ItemName) for clean interpolation
- Compute duration from TRX Times/@start and Times/@finish attributes
- Keep bare 'Test run failed.' Error as the MSBuild failure signal

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds Code="TESTERROR" to the two test-failure Error tasks so the
output matches what dotnet test emits for failing tests:

  error TESTERROR: Test run failed.

Tools that parse dotnet test output keyed on the TESTERROR code will
now correctly recognise device test failures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dotnet CLI terminal logger (default in .NET 8+) filters MSBuild
Message tasks, so Results File and Test summary were invisible at
default verbosity. Fix by embedding the full summary in the TESTERROR
error message which is always surfaced:

  error TESTERROR: Test run failed: 5 of 44 tests failed
    (36 succeeded, 3 skipped, duration: 9.7s). Results File: /path/...

The Message tasks are kept for --tl:off / --verbosity normal users
where they render as the clean two-line dotnet test format:
  Results File: /path/to/test-results.trx
  Test summary: total: 44, failed: 5, succeeded: 36, skipped: 3, duration: 9.7s

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use %0A (MSBuild newline escape) to split the error into two lines
matching the standard dotnet test output format:

  error TESTERROR: Results File: /path/to/test-results.trx
  error TESTERROR: Test summary: total: 44, failed: 5, succeeded: 36, skipped: 3, duration: 18.5s

Both lines are always visible via the terminal logger.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Test classes are discovered via reflection at runtime. Without explicit
trimmer roots the iOS/Android linker strips the referenced test assemblies
(MauiLibrary.XunitTests, Library.NUnitTests), causing only tests in the
main DeviceTests assembly to run (~8 instead of ~44).

TrimmerRootAssembly tells the linker to keep the full assembly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The event stream protocol sends begin/end events to bracket the test run.
If the TCP connection closes after receiving results but without the end
event, the app crashed or was killed mid-run.

Previously this produced partial results with no indication of a crash.
Now the CLI explicitly reports:

  The application appears to have crashed during the test run.
  Only 8 test result(s) were received before the connection was lost.
  Check the device log for crash details.

And forces a non-zero exit code even if all received tests passed, so
CI pipelines correctly fail.

Changes:
- EventStreamService: add HasStarted and HasEnded tracking
- BaseTestCommand: check for begin-without-end after listener completes

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CLI exit codes:
  0 = all tests passed
  1 = test failures
  2 = app crashed (begin event received, results streamed, but no end event)

MSBuild targets check the exit code directly (no file reading needed):
  exit code 2 + TRX:    'Test summary: ... (incomplete: app crashed)'
  exit code 2 + no TRX: 'The application crashed before producing any test results'
  exit code 1 + TRX:    'Test summary: ...' (normal failure)

Validated:
  iOS:   8 tests, crash detected, '(incomplete: app crashed)' shown
  macOS: 44 tests, no crash, normal TESTERROR shown

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the (int testFailures, string? testResults) tuple with a
protected record TestListenerResult(int FailedCount, string? ResultsFile, bool Crashed).

This makes the crash detection explicit via a bool rather than a
sentinel value, and gives clear property names at all call sites.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Centralise exit code mapping and success logic on the record itself:
  - ToExitCode(): 0 = success, 1 = test failures, 2 = app crashed
  - Success: true when no failures and no crash

All callers now use listener.ToExitCode() and listener.Success instead
of duplicating the conditional logic.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MAUI 10.0.20 had a NullReferenceException in Button.LayoutButton when
the platform button was null during a layout pass on the UI thread.
This crashed the app mid-test-run on iOS, producing only partial results.

The fix (dotnet/maui#35284, fixing dotnet/maui#31048) adds a null guard
in LayoutButton. MAUI 10.0.60 includes this fix.

Results: iOS now completes all 44 tests (previously crashed at 8).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the existing AppleDev predicate parameter on GetLogsPlainAsync to
filter logs to just the app process:

  predicate: 'process == "DeviceTestingKitApp.DeviceTests"'

Results:
  Before: 17,000 lines of OS-level noise, no managed exceptions
  After:  190 lines with all PASS/FAIL output and full managed stack
          traces when the app crashes (NullReferenceException, etc.)

Also restores MAUI 10.0.60 (was temporarily reverted for testing).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AppleDev 0.8.10 adds SimCtl.LaunchAppAsync overload with env vars,
so we can replace the manual ProcessStartInfo + SIMCTL_CHILD_* workaround
with the native API call.

AndroidSdk 0.35.1 is the latest stable release.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Testing.Targets .props defaults WindowsPackageType=None (for dotnet test
unpackaged mode). The TCP Windows packaged and XHarness Windows workflows
need MSIX packaging, so pass -p:WindowsPackageType=MSIX explicitly.

Fixes: TCP Windows (GH+AzDO), XHarness Windows (GH+AzDO) CI failures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New articles:
- using-dotnet-test.md: Main guide covering setup, configuration,
  output format, crash detection, MSBuild properties, and how it works
- dotnet-test-android.md: Emulator setup, env var injection, APK config
- dotnet-test-ios.md: Simulator setup, device logs, crash troubleshooting
- dotnet-test-macos.md: Direct launch, no simulator needed
- dotnet-test-windows.md: Unpackaged EXE mode, WindowsPackageType

Updated:
- toc.yml: 'dotnet test (Recommended)' section at top, CLI renamed to
  'Advanced', XHarness renamed to 'Legacy' (from 'CLI Runners')
- index.md: Getting Started leads with dotnet test
- README.md: dotnet test as primary, links to docs site
- technical-architecture-overview.md: Testing.Targets package section
  with MSBuild target chain, exit code protocol, package structure
- ci-pipeline.md: dotnet test CI examples for GH Actions and AzDO

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove WindowsPackageType=None default from .props entirely. Instead,
the _DeviceRunnersWindowsArgs target detects what the build produced:

- If .exe exists (unpackaged): uses 'windows test --app path.exe'
- If .msix exists (packaged): uses 'windows test --app path.msix'
- If neither: clear error message

This means dotnet publish for MSIX packaging is no longer broken by the
Testing.Targets package. CI workflows no longer need -p:WindowsPackageType
overrides. Also search both artifacts/ and sample/ for .msix in CI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Create DeviceRunners.VisualRunners.Core (net9.0;net10.0 only) with the
cross-platform testing types that the CLI needs:

Moved from VisualRunners:
- Testing/ITestResultInfo, ITestCaseInfo, ITestAssemblyInfo,
  ITestAssemblyConfiguration, TestResultStatus
- Testing/Channels/IResultChannel, IResultChannelFormatter,
  FileResultChannel, FileResultChannelOptions, TestResultEvent,
  EventStreamFormatter, TextResultChannelFormatter,
  TrxResultChannelFormatter, TextWriterResultChannel

The CLI now references VisualRunners.Core instead of VisualRunners,
breaking the dependency on multi-target mobile TFMs. This unblocks
self-contained and AOT publish (no more Mono runtime pack errors).

VisualRunners references VisualRunners.Core transitively, so all
existing consumers are unaffected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add TrimmerRootAssembly for Spectre.Console.Cli and DeviceRunners.Cli
to preserve types discovered via reflection (command settings classes).
Spectre.Console.Cli is explicitly not trim-safe (spectreconsole/spectre.console#1690)
so the entire assembly must be rooted.

Suppress trim analysis warnings since Spectre.Console.Cli generates
unavoidable IL2026/IL2090 warnings that are covered by the rooting.

Result: 19MB self-contained single-file binary (down from 21MB multi-file).
  dotnet publish -r osx-arm64 --self-contained -p:PublishSingleFile=true

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CLI csproj now has SelfContained, PublishSingleFile, PublishTrimmed,
and RuntimeIdentifiers (osx-arm64;win-x64) set by default. Plain
'dotnet publish -r osx-arm64' produces a 19MB single-file binary.

The NuGet package PublishCliTools target overrides these back to
framework-dependent (SelfContained=false, PublishSingleFile=false,
PublishTrimmed=false) since the bundled tool runs via 'dotnet <dll>'.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NuGet package now ships self-contained, trimmed, single-file native
binaries for each platform in tools/<rid>/:

  tools/osx-arm64/DeviceRunners.Cli     (~20 MB)
  tools/win-x64/DeviceRunners.Cli.exe   (~20 MB)
  tools/linux-x64/DeviceRunners.Cli     (~20 MB)

The MSBuild targets detect the host OS and select the right binary.
Falls back to framework-dependent DLL at tools/DeviceRunners.Cli.dll
for in-repo dev or older packages.

Changes:
- CLI csproj: add JsonSerializerIsReflectionEnabledByDefault, linux-x64 RID
- Testing.Targets csproj: PublishCliTools publishes 3 RIDs (self-contained)
- .targets: detect host OS -> pick tools/<rid>/DeviceRunners.Cli[.exe]

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Only the per-RID single-file native binary is used now. The fallback
to tools/DeviceRunners.Cli.dll is removed since it will never exist.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds first-class WebAssembly (Blazor) browser test support to DeviceRunners. It introduces a Blazor visual runner, a reflection-based xunit discoverer/runner that can run threadlessly in WASM, and CLI commands (wasm test / wasm serve) that serve a published bundle from a built-in static-file server, drive headless Chrome via the Chrome DevTools Protocol, capture NDJSON console output, and convert it into TRX results. CI (GitHub Actions and Azure DevOps) and the DeviceRunners.Testing.Targets MSBuild integration are extended to cover the browser platform, and a new sample browser test project plus shared Blazor library demonstrate end-to-end usage.

Changes:

  • New DeviceRunners.VisualRunners.Blazor Razor Class Library + reflection-based xunit runner (XunitReflectionTestDiscoverer/XunitReflectionTestRunner plus XunitYielding…Runner cooperative-scheduling stack).
  • New CLI surface: wasm test/wasm serve commands with WasmWebServerService (HttpListener static server) and BrowserService (CDP over WebSocket), plus MSBuild targets/CI templates for the browser platform.
  • Sample apps reorganised: new DeviceTestingKitApp.BlazorLibrary, DeviceTestingKitApp.BrowserTests, DeviceTestingKitApp.BlazorLibrary.XunitTests; the MAUI library xunit test project is dropped from the solution.

Reviewed changes

Copilot reviewed 79 out of 79 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
src/DeviceRunners.VisualRunners.Xunit/Xunit{Reflection,Yielding…}*.cs, DiagnosticMessageSink.cs, EmptySourceInformationProvider.cs, XunitTestRunner.cs New reflection-based discoverer/runner stack and yielding runners for threadless xunit execution
src/DeviceRunners.VisualRunners.Xunit/VisualTestRunnerConfigurationBuilderExtensions.cs AddXunit(useReflection:) toggle to select the new runner
src/DeviceRunners.VisualRunners.Core/Testing/Channels/{Console,Debug}ResultChannel.cs, src/DeviceRunners.VisualRunners/Testing/Channels/ConsoleResultChannel.cs, …Builder/Extensions.cs Promote ConsoleResultChannel into Core, add formatter overloads, and new AddResultChannel extension overloads
src/DeviceRunners.VisualRunners.Blazor/** New Blazor Razor Class Library: pages/components, JS interop, builder extensions, app terminator, CSS
src/DeviceRunners.Cli/Program.cs, Commands/Wasm/{WasmTestCommand,WasmServeCommand}.cs Register wasm branch and implement test/serve commands
src/DeviceRunners.Cli/Services/{WasmWebServerService,BrowserService}.cs Static-file server and headless Chrome/CDP driver
src/DeviceRunners.Testing.Targets/build/*.{props,targets} Add browser platform support and _DeviceRunnersWasmArgs target
sample/src/DeviceTestingKitApp.BlazorLibrary/**, sample/src/DeviceTestingKitApp.Library/*.csproj, sample/src/DeviceTestingKitApp.MauiLibrary/Features/MauiSemanticAnnouncer.cs New Blazor sample library, library multi-targets net9/net10, null-arg validation tweak
sample/test/DeviceTestingKitApp.BrowserTests/, DeviceTestingKitApp.BlazorLibrary.XunitTests/, DeviceTestingKitApp.DeviceTests/Tests/, DeviceTestingKitApp.MauiLibrary.XunitTests/ New WASM/browser test project, new bUnit-based component tests, additional xunit tests
test/DeviceRunners.VisualRunners.Tests/Testing/{TestDiscovererTests,TestRunnerTests}.XunitReflection.cs Test fixtures for the new reflection discoverer/runner
DeviceRunners.slnx, Directory.Packages.props Solution/package additions (Blazor library, BrowserTests, bUnit, ASP.NET Core WASM); MAUI xunit test project removed from solution
.github/workflows/ci.yml, .github/workflows/test-wasm-browser/action.yml, .azure/azure-pipelines.yml, .azure/templates/test-wasm-browser.yml New WASM browser test job/stage in GitHub Actions and Azure DevOps

Comment thread DeviceRunners.slnx Resolved
mattleibow and others added 4 commits May 14, 2026 01:58
EmbedAssembliesIntoApk was always set to true when the Testing.Targets
package was referenced, disabling Fast Deployment for ALL Android builds.
This made normal debug builds from the IDE slow.

Now it's set in _DeviceRunnersMarkTestRun (which only runs during
dotnet test), so normal dotnet build / dotnet run / IDE builds keep
Fast Deployment enabled for fast iteration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace anonymous types and JsonDocument with typed models:
- ChromeDebugPage: Chrome /json endpoint response
- CdpCommand / CdpCommandWithParams<T>: commands we send
- CdpEvent / CdpEventParams / CdpConsoleArg: events we receive
- CdpNavigateParams: Page.navigate parameters
- BrowserJsonContext: STJ source generator context

Eliminates all IL2026 trim warnings without suppression attributes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New:
- cli-device-runner-for-wasm-using-devicerunners-cli.md
- dotnet-test-wasm.md

Updated:
- technical-architecture-overview.md: add Browser (WASM) platform
- ci-pipeline.md: add WASM Browser Tests section
- using-devicerunners-cli.md: add WASM commands
- index.md: add WASM entries
- visual-runner-in-the-ide.md: add Blazor visual runner

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Base automatically changed from mattleibow/investigate-dotnet-test-mobile to main May 14, 2026 08:09
Resolve 16 merge conflicts from main (which now includes PR #106).
Fix CS0067 warning-as-error for CanExecuteChanged in ViewModelBindingTests.
Fix reflection discoverer test count (PreEnumerateTheories=false yields 6 not 8).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mattleibow
mattleibow marked this pull request as ready for review May 14, 2026 09:02
mattleibow and others added 3 commits May 14, 2026 15:14
…nd docs

A. dotnet test end-to-end for WASM:
   - Fix platform detection: use UsingMicrosoftNETSdkBlazorWebAssembly
     for Blazor WASM projects (plain net9.0 has no -browser TFM)
   - Add Testing.Targets import to BrowserTests.csproj
   - Add dotnet-test-wasm composite action (GitHub Actions)
   - Add test-dotnet-test-wasm.yml template (Azure DevOps)
   - Add dotnet-test-wasm jobs to both CI pipelines

F. Browser console log capture:
   - WasmTestCommand writes all console messages to browser-console.log
     in the results directory (like logcat.txt for Android)

G. Expanded WASM CLI documentation:
   - Added Command Options tables for wasm test and wasm serve
   - Added dotnet test section with usage example
   - Added troubleshooting sections (Chrome, timeout, logs, headed mode)
   - Updated dotnet-test-wasm.md to reflect full dotnet test support
   - Updated architecture overview: WASM dotnet test status now ✅

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Restore MauiLibrary.XunitTests to solution (accidentally removed)
- Add BlazorLibrary.XunitTests to solution
- Fix TestAssemblyPage: move TestResultSelected subscription to
  OnInitialized to prevent duplicate handlers on re-render
- Fix path traversal in WasmWebServerService: append directory
  separator to root before StartsWith check
- Fix silent exception swallowing in WasmWebServerService: log to
  Debug.WriteLine and Console.Error
- Fix BrowserService: clean up temp Chrome user-data directory in
  DisposeAsync
- Fix BrowserService: capture stderr and exit code on Chrome launch
  failure for better diagnostics
- Fix WasmServeCommand: unsubscribe CancelKeyPress handler to prevent
  leak on repeated invocations
- Fix WasmTestCommand: catch OperationCanceledException instead of
  TaskCanceledException for robustness

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The MSBuild targets previously hardcoded --logger "trx;LogFileName=test-results.trx"
in the CLI args. Plain `dotnet test` does NOT produce TRX by default — only when
the user passes --logger. This change matches standard dotnet test behavior:

- Forward $(VSTestLogger) to the CLI --logger arg when set
- Forward $(VSTestResultsDirectory) to --results-directory when set
- Omit --logger entirely when not specified (no TRX produced)
- Report target already handles missing TRX gracefully

All CI dotnet-test workflows now explicitly pass --logger since they
use test result reporters (dorny/test-reporter, PublishTestResults)
that require TRX files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 105 out of 105 changed files in this pull request and generated 7 comments.

mattleibow and others added 6 commits May 15, 2026 14:45
The ReadDevToolsUrlAsync method's CancellationToken caused an
OperationCanceledException that bypassed the descriptive error message
with stderr capture. Now catches the exception and falls through to
build the diagnostic message. Also increased timeout from 15s to 30s
since CI runners can be slow to start Chrome.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove duplicate DeviceRunnersTestRun property assignment
- Use async CopyToAsync for file serving (avoids blocking on large WASM DLLs)
- Fix tab indentation in AdvancedTests.cs and DisposableTests.cs
- Improve TotalCount==0 error message with diagnostic guidance
- Remove --disable-gpu flag to enable WebGL in headless Chrome
  (--headless=new supports WebGL via SwiftShader without it)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix Windows XHarness status in architecture overview (❌ → ✅)
- Fix WASM troubleshooting advice in dotnet-test-wasm.md
- Add Browser (WASM) to all tables in using-dotnet-test.md
- Add DeviceRunnersWasmTimeout property to docs
- Standardize TFMs to net10.0 across all CLI docs
- Standardize platform ordering in index.md
- Use "Browser (WASM)" naming consistently in toc.yml
- Add iOS and WASM to CLI command patterns
- Add cross-links and clarify no -f flag for WASM
- Fix basic example to include --logger flag

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move the URL-based AddCliConfiguration(string currentUrl) overload from
DeviceRunners.VisualRunners.Blazor to DeviceRunners.VisualRunners where
it sits alongside the env-var/CLI-args overload. The method has no
Blazor dependencies — it's pure URL parsing + EnableAutoStart +
AddResultChannel. The Blazor project still calls it via the JS interop
in UseVisualTestRunner.

Also remove the duplicate WASM section from 'Additional Resources' in
using-devicerunners-cli.md — WASM is already a first-class platform in
the Supported Platforms table and Common Command Patterns section.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Each platform has a fundamentally different CLI configuration strategy:

- MAUI (MauiCliConfigurationExtensions): reads env vars (Android,
  iOS, macOS, Windows EXE) and CLI args (Windows MSIX) → adds TCP
  result channel for native platform communication
- Blazor (BlazorCliConfigurationExtensions): parses URL query string
  (?device-runners-autorun=1) → adds console result channel for
  browser communication via Chrome DevTools Protocol

These are platform concerns, not shared concerns. The shared
VisualRunners library keeps the building blocks (EnableAutoStart,
AddTcpResultChannel, AddConsoleResultChannel, AddResultChannel).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rm CLI config

Rename MauiCliConfigurationExtensions and BlazorCliConfigurationExtensions
to VisualTestRunnerConfigurationBuilderExtensions since they are extensions
for IVisualTestRunnerConfigurationBuilder.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 111 out of 112 changed files in this pull request and generated no new comments.

@mattleibow mattleibow changed the title feat: Add WebAssembly (WASM) browser test runner support feat: Add WebAssembly (Blazor) browser test runner with dotnet test integration May 15, 2026
mattleibow and others added 3 commits May 15, 2026 23:23
…late

The parameter was 'expected' but validate-arch.yml declares 'expectedArch'.
This mismatch caused AzDO to reject the entire pipeline at parse time,
preventing all CI jobs from running.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Android: add Free Disk Space step to prevent disk full failures
- Android: add Display Android Tool Logs step for debugging
- iOS: add architecture validation (validate-arch.yml)
- macOS: add architecture validation (validate-arch.yml)

These steps were present in the TCP templates but missing from the
dotnet-test templates, causing the Android job to fail with
'No connected devices found' due to 95% disk usage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Job-level variables with $(HOME) don't expand correctly for the
emulator — it couldn't find DotnetTestEmulator.ini in ANDROID_AVD_HOME.
Switch to ##vso[task.setvariable] at runtime (matching the TCP template)
so $HOME resolves to the actual home directory.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mattleibow
mattleibow merged commit 1367df5 into main May 15, 2026
50 checks passed
mattleibow added a commit that referenced this pull request May 16, 2026
- Merge main (Testing.Targets #106, Blazor browser runner #110)
- Add browser/WASM platform detection using UsingMicrosoftNETSdkBlazorWebAssembly
- Add index.html.template for Blazor WebAssembly projects
- Update BrowserTests sample to use Bootstrap for index.html generation
- Fix platform detection to use GetTargetPlatformIdentifier() function
- Condition icons/splash to MAUI-only (not browser)
- Remove unused AndroidManifest property (SDK provides default + overlay)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants