fix(load-test): tear down the app-side data a load run leaves behind

[philmerrell]

Why

scripts/load-test/teardown.sh deleted the Cognito user and the quota override, but nothing the application writes once a load-test user signs in and chats. Production (checked 2026-09-22) still has 620 @load.invalid rows in <prefix>-users from the 2026-09-10/11 runs, 310 of them with September user-cost-summary rows. The admin dashboards count them as real users: 1,829 this month, of which 1,519 are real.

What changed

teardown.sh --manifest now also deletes per-user app data, keyed on the manifest's user_id (sub). Everything is read first and printed as the plan. The plan also shows the load-test spend that stays in the rollups. A new app-data.sh holds the inventory and delete code.

Deleted (a load run writes it)	Left alone (documented in the README)
users (read first, deleted last), sessions-metadata (all USER#<sub> rows: S#, C#, D#, F#, TSUM#, APPCARD#, UIRES#, LEASE#), user-cost-summary, system-cost-rollup ACTIVE# markers, quota-events, overrides via UserOverrideIndex, user-file-uploads + user-files/<sub>/ and compaction-offload/<sub>/, user-artifacts + <sub>/ objects, AgentCore Memory long-term records	Memory events (expire after 90 days), bff-sessions (TTL + logout), user-settings / tool and skill preferences (explicit PUT only), api-keys, oauth-user-tokens, memory spaces, shares, assistants, fine-tuning, announcement acks, audit log

New safety rails, on top of the existing ones (the loadtest- prefix check across the whole manifest, overrides deleted before users, fail loudly and keep the manifest, re-runnable):

• Every user_id in the manifest must be a well-formed sub, checked across the whole file before anything is deleted. An empty sub would otherwise make user-files/<sub>/ a prefix covering every user.
• Owner check. Each sub's <prefix>-users row must carry <username>@${LOAD_TEST_EMAIL_DOMAIN} before any of its app data is touched. One mismatch refuses the whole run.
• The users row is deleted last, and only after every other step for that user succeeded, so a re-run can always verify the user again.

system-cost-rollup decision. The population counters are corrected and the cost totals are kept:

• Each ACTIVE# marker is deleted in the same TransactWriteItems that decrements the activeUsers / uniqueUsers counter it once incremented. The dashboard's "active users" reads ROLLUP#MONTHLY.activeUsers directly, nothing in the codebase recomputes it, and the table has no TTL.
• The transaction is conditional on the marker existing, so a re-run can't decrement twice. Conflicts with live cost writes are retried.
• totalCost and the token/request totals are kept. That was real Bedrock spend, and removing it would put the rollups out of step with Cost Explorer and the PLATFORM# rows. The plan prints the spend kept per period, and the README explains the gap.

--orphans handles runs whose manifest is already gone:

• It is a dry run unless --apply is given.
• It finds @load.invalid users through EmailDomainIndex, requires a loadtest-…@load.invalid email, and skips any sub that still has a Cognito account. Each remaining user then goes through the same owner check, inventory and delete as a manifest entry. --limit N processes a batch at a time.
• This PR does not run it against prod. The operator runs it, dry run first.

Per-user work runs as a bounded job pool (--jobs, default 8). The 5-user dry run took 47s serially and 21s with the pool. At the default, the README estimates roughly 15–30 minutes per 600 users.

Testing (dev only, dev-ai / dev-boisestateai-v2)

The repo has no shell-test pattern, so testing was against dev, in a GNU userland (bash 5.2, aws-cli 2.34).

• Orphans: dev's one real orphan (from a run on 2026-09-03) was removed, including 10 session rows and 5 memory records. Every store was checked empty afterwards. The daily, monthly and model counters each dropped by exactly 1, and totalCost was unchanged. Then two users were turned into orphans (Cognito deleted, app data kept) and cleared with --limit 1 followed by a second pass.
• Manifest: one freshly provisioned user that never signed in, plus leftover users with real data, run in parallel. Counters dropped by exactly the number of users with data.
• Re-run: the same manifest run a second time made no changes and left the counters unchanged. Both transaction-cancel branches were exercised: marker already gone, and rollup row missing.
• Rails: each of these refused with exit 1 and deleted nothing:
  • a load-test username paired with a real user's sub
  • two load-test users' subs swapped with each other
  • an empty sub
  • a non-loadtest- username
• Failure path: an injected BatchWriteItem failure on sessions-metadata gave exit 1 with the manifest and users row kept. A plain re-run then deleted only what was left.
• bash 3.2 (macOS) is refused up front with a clear message, since wait -n needs 4.3.

Side effect in dev: the 2026-09-03 run was never torn down. Six of its users were used as test subjects and are now fully removed. The other 44 are still in dev's Cognito, with expired overrides still in the quota table.

🤖 Generated with Claude Code