Skip to content

Fix WASM test runner to use connection/data timeouts like other platforms - #158

Open
mattleibow wants to merge 2 commits into
mainfrom
mattleibow-investigate-test-timeout
Open

mattleibow wants to merge 2 commits into
mainfrom
mattleibow-investigate-test-timeout

Conversation

@mattleibow

Copy link
Copy Markdown
Owner

Problem

When running large test suites on WASM, the run was killed at exactly 300s even though tests were actively passing — and it didn't fail the run, so it was easy to miss.

The WASM runner used a single total wall-clock cap from browser launch (--timeout, default 300s) that never reset. So a healthy suite that legitimately runs longer than 300s gets cancelled mid-stream, and because the aborted/never-run tests are simply not recorded, the results showed Failed=0 and looked green.

Root cause

Every other platform (android/ios/macos/windows) already inherits and honours two timeouts from the base settings:

  • --connection-timeout (120s) — wait for the first sign of life.
  • --data-timeout (30s) — an inactivity timeout that resets on every event, so a long healthy run keeps going as long as it keeps producing output.

WASM inherited both options but ignored them, using only its own total cap. It was just wrong.

Fix

Wire WASM up to the same two timeouts as everything else:

  • connection-timeout → wait for the first browser message (app booting / reporting).
  • data-timeout → inactivity timeout, reset on every browser console message.

A single resettable CancellationTokenSource models the connection-then-inactivity window. The old total --timeout / DeviceRunnersTimeout knob is removed entirely.

A run that hits either timeout is now treated as a crash → exit code 2, with a loud red Test execution TIMED OUT: <reason> message instead of a quiet yellow line followed by Failed=0. MSBuild's exit-2 report wording now mentions timeouts too.

Changes

  • WasmTestCommand — replace the 300s total cap with the connection + data (inactivity) model; loud failure message; disposal-race guard on the reset.
  • BaseTestCommand — reverted the exploratory overall---timeout option (kept the existing connection/data timeouts).
  • MSBuild props/targets — WASM invocation now passes --connection-timeout / --data-timeout; exit-2 error text says "crashed or timed out".
  • Docs updated: dotnet-test-wasm.md, WASM CLI doc, using-dotnet-test.md, CLI README.md.
  • New WasmTimeoutTests covering the connection vs. inactivity timeout messaging.

Testing

  • dotnet test DeviceRunners.Cli.Tests — 72 passed.
  • CLI and Testing.Targets projects build clean.

mattleibow and others added 2 commits July 22, 2026 16:57
…latforms

The WASM runner used a single 300s total wall-clock cap from browser launch
that never reset, so a healthy suite that ran longer than 300s was killed
mid-stream and (because the aborted tests were never recorded) looked green.

Every other platform (android/ios/macos/windows) already inherits and honours
`--connection-timeout` (wait for first sign of life) and `--data-timeout` (an
inactivity timeout that resets on every event). WASM inherited both but ignored
them. This wires WASM up to the same two timeouts:

- connection-timeout: wait for the first browser message (app booting/reporting)
- data-timeout: inactivity timeout, reset on every browser console message, so a
  long healthy run keeps going as long as it keeps producing output

A run that hits either timeout is now treated as a crash (exit code 2) with a
loud red "TEST RUN TIMED OUT" message, instead of a quiet yellow line + Failed=0.
MSBuild's exit-2 report wording is updated to mention timeouts.

Docs (dotnet-test-wasm, WASM CLI, using-dotnet-test, CLI README) updated and a
WasmTimeoutTests unit test added for the timeout-reason messaging.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a97a5ab5-e10b-41a0-bf6a-a85f37b465d3
… path

The device (TCP) runner just prints a single generic "timed out" line and has
no connection-vs-data distinction. DescribeWasmTimeout made WASM inconsistently
more verbose for little benefit, so remove it and print a plain line to match the
other platforms. Also drops the now-unused anyMessageReceived flag and the helper's
unit tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a97a5ab5-e10b-41a0-bf6a-a85f37b465d3

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant