Skip to content

Tests: network-fault tests for uploads and downloads (Playwright interception + Toxiproxy) #71

Open

Description

@Hiczownik

Summary

Add automated network-fault tests that deliberately break the network between the browser, the API, storage and Postgres. Then check the app fails safely:

  • no orphaned uploads or files;
  • no download counted for a failed save;
  • a clear message shown;
  • recovery once the network is back.

Two layers, cheapest first:

  1. Playwright request interception: no new infrastructure.
  2. Toxiproxy: real TCP-level faults between containers.

Why

Layer 1: Playwright request interception

Use page.route() / context.route() to fail, delay or cut requests from the browser. No compose changes needed.

Building blocks (helpers in e2e/faults.ts):

  • failNthRequest(pattern, n, status | 'abort'), e.g. the 3rd PUT to storage.
  • delayRequests(pattern, ms).
  • cutResponseAfter(pattern, bytes): fulfil with a truncated body, or abort mid-stream.
  • failServiceWorkerRegistration(): route /download-sw.js to a 500.
  • goOfflineFor(ms) via context.setOffline(true/false).

Matching:

  • Storage requests go to Storage:PublicBaseUrl (http://localhost:9000 in CI).
  • API requests go to http://localhost:3000/api/v1/files/....

Layer 2: Toxiproxy (TCP level)

Add a test-only compose override (docker-compose.faults.yml) with a toxiproxy container (Shopify Toxiproxy; HTTP control API on 8474), and proxies for:

  • API → storage: the API's Storage__ServiceUrl → http://toxiproxy:19000 → minio:9000.
  • API → Postgres: the connection string host → toxiproxy:15432 → postgres:5432.
  • Browser → storage: Storage__PublicBaseUrl → http://localhost:29000, published by Toxiproxy → minio:9000.
    • Toxiproxy is a plain TCP proxy, so the Host header the browser sends is passed through unchanged, and presigned-URL signatures (signed for that host) stay valid.
    • Check this with an upload before relying on it (README: 403 SignatureDoesNotMatch means the host changed).

Toxics to use: latency (with jitter), bandwidth, timeout (holds the connection), reset_peer (TCP RST), slow_close, limit_data (cut after N bytes).

Tests control the proxies through Toxiproxy's HTTP API with a small helper (e2e/toxiproxy.ts), and reset all toxics after each test.

Scenarios

# Fault Expected result Covers
1 In a 3-file upload, one file's 3rd multipart part fails permanently (L1: always 500) All other uploads stop. multipart/abort called for every started upload. A clear error, nothing confirmed #45
2 One part fails twice, then succeeds Retried with backoff, upload completes, the file is byte-identical after download multipartUploader.ts retries
3 Storage unreachable during presign after some multipart uploads were created (L2: reset_peer API→storage after N requests) 5xx with a clear message. No unfinished multipart uploads remain #38
4 Storage unreachable during confirm (L2) Error shown. No bundle created. The same upload can be confirmed once storage is back (within PendingUploadLifetime) #37
5 Postgres down for 10 s during confirm (L2: timeout API→Postgres) 503 / clear message, no partial rows. The API recovers on its own without a restart
6 Very slow storage during upload (L2: bandwidth 50 KB/s browser→storage) Progress keeps moving. No timeouts within the presigned URL's lifetime. No duplicate parts
7 Download stream cut halfway, single file (L1 cutResponseAfter, or L2 limit_data) No partial file saved as complete. The user sees an error. Re-downloading within the same session works without counting another download #25, #64
8 The service worker fails to register, then works on retry (L1) Falls back correctly. A later download on the same page tries the service worker again #41
9 Encrypted "Download All" on a slow connection (L2 bandwidth) for longer than an hour Exactly one download session opened (DownloadCount + 1). The ZIP is complete and opens #31, #64
10 A tampered stored file (existing scripts/tamper-stored-file.sh) under the throttled connection Refused, nothing saved
11 The browser goes offline for 20 s mid-upload (L1 context.setOffline) Retries after reconnecting, or fails clearly. Never a "success" with missing parts
12 Storage returns 503 SlowDown on part uploads (L1 fulfil 503 with an S3 error body) Backs off, then completes
13 A download session expires mid-download (shorten the lifetime through a test-only setting) A clear "session expired" message. No silent second counted download #31

Browser coverage: run on Chromium by default. The save-method scenarios (7–9) also run on Firefox and WebKit, since saving differs per engine (playwright.config.ts already defines chromium, firefox, webkit projects).

Checks after each scenario (helpers)

  • Storage: no unfinished multipart uploads (ListMultipartUploads on fileholder-bucket via a small API test endpoint, the AWS SDK from Node, or mc ls --incomplete). No unexpected objects for the scenario's bundle.
  • Database: FileGroups.DownloadCount, DownloadSessions count, PendingUploads and MultipartUploadSessions rows as expected, queried with docker compose exec postgres psql … from the test.
  • Browser: Playwright download events (no completed download for failed saves). The error text is visible, and the progress UI doesn't hang.
  • API: no unhandled exceptions in the API logs during the test (grep docker compose logs api for Unhandled).

Test-only settings

Some scenarios need shorter timings or switches. Add them as configuration that only the test override sets, never defaults:

  • a shorter SessionLifetime (scenario 13);
  • a shorter PendingUploadLifetime;
  • a faster purge interval.

This also requires moving those static readonly values into options. Keep the production values as the defaults.

Where and when it runs

  • A new Playwright project, faults, with its own testDir: e2e/faults.
  • The Layer 1 tests run in the existing e2e-tests CI job on every PR (they need no extra containers).
  • The Layer 2 tests run in a separate job that starts the stack with -f docker-compose.yml -f docker-compose.faults.yml. Run them on PRs touching src/fileholder-ui/src/features/**, src/FileHolder.Infrastructure/** or docker-compose*.yml, plus nightly.
  • Keep each scenario deterministic: no real timing races, only fixed delays and counts.

Acceptance

Related

#25, #31, #37, #38, #41, #42, #45, #64, PR #68. #69 (storage lifecycle safety net) (orphaned objects) and #72 (tracing) (traces make failing scenarios easy to diagnose) also relate.

Activity

  1. added
    enhancementNew feature or request
    frontendBrowser / React app
    backend.NET API and services
    uploadUploading files and bundles
    downloadDownloading and saving files
    storageObject storage, purge, multipart
    on Oct 21, 2026
  2. added a commit that references this issue on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backend.NET API and servicesdownloadDownloading and saving filesenhancementNew feature or requestfrontendBrowser / React appstorageObject storage, purge, multipartuploadUploading files and bundles

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions