Repository navigation
Description
Summary
Add automated network-fault tests that deliberately break the network between the browser, the API, storage and Postgres. Then check the app fails safely:
- no orphaned uploads or files;
- no download counted for a failed save;
- a clear message shown;
- recovery once the network is back.
Two layers, cheapest first:
- Playwright request interception: no new infrastructure.
- Toxiproxy: real TCP-level faults between containers.
Why
- Most of the upload and download code is failure handling: multipart retries with backoff (
multipartUploader.ts), aborts, save-method fallbacks (streamSaving.ts,serviceWorkerSaving.ts), download-session counting, OPFS clean-up. - CI only tests the happy path (
e2e/fileHolder.spec.ts), so these bugs have all been found by reading code:- Download can be counted, then fail, when a save method falls back to memory #25: a download counted, then failed, when the save method falls back.
- Encrypted 'download all' ZIP opens new download sessions after 15 minutes #31: long encrypted ZIPs opened extra sessions (fixed in fix(download): six-hour sessions, purge waits for running downloads (#31, #32) #64).
- Presign leaks S3 multipart uploads when a later step fails #38: presign leaking multipart uploads (PR fix(server): expiry before password, required key material, safe names, no leaked uploads (#38, #39, #40, #43, #52) #68).
- Service worker registration failure is cached for the whole page life #41: a service-worker registration failure cached for the page's life.
- Browser-storage (OPFS) temp files can be deleted while still in use #42: OPFS temp files deleted while in use.
- A failed file upload doesn't stop or abort the other uploads #45: a failed file upload doesn't stop or abort the others.
- Without fault tests, these fixes can silently regress.
Layer 1: Playwright request interception
Use page.route() / context.route() to fail, delay or cut requests from the browser. No compose changes needed.
Building blocks (helpers in e2e/faults.ts):
failNthRequest(pattern, n, status | 'abort'), e.g. the 3rdPUTto storage.delayRequests(pattern, ms).cutResponseAfter(pattern, bytes): fulfil with a truncated body, or abort mid-stream.failServiceWorkerRegistration(): route/download-sw.jsto a 500.goOfflineFor(ms)viacontext.setOffline(true/false).
Matching:
- Storage requests go to
Storage:PublicBaseUrl(http://localhost:9000in CI). - API requests go to
http://localhost:3000/api/v1/files/....
Layer 2: Toxiproxy (TCP level)
Add a test-only compose override (docker-compose.faults.yml) with a toxiproxy container (Shopify Toxiproxy; HTTP control API on 8474), and proxies for:
- API → storage: the API's
Storage__ServiceUrl→http://toxiproxy:19000→minio:9000. - API → Postgres: the connection string host →
toxiproxy:15432→postgres:5432. - Browser → storage:
Storage__PublicBaseUrl→http://localhost:29000, published by Toxiproxy →minio:9000.- Toxiproxy is a plain TCP proxy, so the
Hostheader the browser sends is passed through unchanged, and presigned-URL signatures (signed for that host) stay valid. - Check this with an upload before relying on it (README:
403 SignatureDoesNotMatchmeans the host changed).
- Toxiproxy is a plain TCP proxy, so the
Toxics to use: latency (with jitter), bandwidth, timeout (holds the connection), reset_peer (TCP RST), slow_close, limit_data (cut after N bytes).
Tests control the proxies through Toxiproxy's HTTP API with a small helper (e2e/toxiproxy.ts), and reset all toxics after each test.
Scenarios
| # | Fault | Expected result | Covers |
|---|---|---|---|
| 1 | In a 3-file upload, one file's 3rd multipart part fails permanently (L1: always 500) | All other uploads stop. multipart/abort called for every started upload. A clear error, nothing confirmed |
#45 |
| 2 | One part fails twice, then succeeds | Retried with backoff, upload completes, the file is byte-identical after download | multipartUploader.ts retries |
| 3 | Storage unreachable during presign after some multipart uploads were created (L2: reset_peer API→storage after N requests) |
5xx with a clear message. No unfinished multipart uploads remain | #38 |
| 4 | Storage unreachable during confirm (L2) |
Error shown. No bundle created. The same upload can be confirmed once storage is back (within PendingUploadLifetime) |
#37 |
| 5 | Postgres down for 10 s during confirm (L2: timeout API→Postgres) |
503 / clear message, no partial rows. The API recovers on its own without a restart | |
| 6 | Very slow storage during upload (L2: bandwidth 50 KB/s browser→storage) |
Progress keeps moving. No timeouts within the presigned URL's lifetime. No duplicate parts | |
| 7 | Download stream cut halfway, single file (L1 cutResponseAfter, or L2 limit_data) |
No partial file saved as complete. The user sees an error. Re-downloading within the same session works without counting another download | #25, #64 |
| 8 | The service worker fails to register, then works on retry (L1) | Falls back correctly. A later download on the same page tries the service worker again | #41 |
| 9 | Encrypted "Download All" on a slow connection (L2 bandwidth) for longer than an hour |
Exactly one download session opened (DownloadCount + 1). The ZIP is complete and opens |
#31, #64 |
| 10 | A tampered stored file (existing scripts/tamper-stored-file.sh) under the throttled connection |
Refused, nothing saved | |
| 11 | The browser goes offline for 20 s mid-upload (L1 context.setOffline) |
Retries after reconnecting, or fails clearly. Never a "success" with missing parts | |
| 12 | Storage returns 503 SlowDown on part uploads (L1 fulfil 503 with an S3 error body) | Backs off, then completes | |
| 13 | A download session expires mid-download (shorten the lifetime through a test-only setting) | A clear "session expired" message. No silent second counted download | #31 |
Browser coverage: run on Chromium by default. The save-method scenarios (7–9) also run on Firefox and WebKit, since saving differs per engine (playwright.config.ts already defines chromium, firefox, webkit projects).
Checks after each scenario (helpers)
- Storage: no unfinished multipart uploads (
ListMultipartUploadsonfileholder-bucketvia a small API test endpoint, the AWS SDK from Node, ormc ls --incomplete). No unexpected objects for the scenario's bundle. - Database:
FileGroups.DownloadCount,DownloadSessionscount,PendingUploadsandMultipartUploadSessionsrows as expected, queried withdocker compose exec postgres psql …from the test. - Browser: Playwright
downloadevents (no completed download for failed saves). The error text is visible, and the progress UI doesn't hang. - API: no unhandled exceptions in the API logs during the test (grep
docker compose logs apiforUnhandled).
Test-only settings
Some scenarios need shorter timings or switches. Add them as configuration that only the test override sets, never defaults:
- a shorter
SessionLifetime(scenario 13); - a shorter
PendingUploadLifetime; - a faster purge interval.
This also requires moving those static readonly values into options. Keep the production values as the defaults.
Where and when it runs
- A new Playwright project,
faults, with its owntestDir: e2e/faults. - The Layer 1 tests run in the existing
e2e-testsCI job on every PR (they need no extra containers). - The Layer 2 tests run in a separate job that starts the stack with
-f docker-compose.yml -f docker-compose.faults.yml. Run them on PRs touchingsrc/fileholder-ui/src/features/**,src/FileHolder.Infrastructure/**ordocker-compose*.yml, plus nightly. - Keep each scenario deterministic: no real timing races, only fixed delays and counts.
Acceptance
- Helpers for both layers.
- Scenarios 1–13 implemented, or marked
test.fixmeand linked to the open bug they reproduce until it's fixed (Download can be counted, then fail, when a save method falls back to memory #25, Service worker registration failure is cached for the whole page life #41, Browser-storage (OPFS) temp files can be deleted while still in use #42, A failed file upload doesn't stop or abort the other uploads #45 are open). - The CI jobs described above.
- A short
e2e/faults/README.md: how to run locally (docker compose -f docker-compose.yml -f docker-compose.faults.yml up -d, thennpx playwright test --project=faults).
Related
#25, #31, #37, #38, #41, #42, #45, #64, PR #68. #69 (storage lifecycle safety net) (orphaned objects) and #72 (tracing) (traces make failing scenarios easy to diagnose) also relate.
Activity
Merge pull request #104 from Hiczownik/test/fault-interception