ci: add Aspire-backed system smoke testing and evaluate pipeline orchestration #571

Context

#565 / PR #567 added a .NET Aspire 13.5 AppHost for local PostgreSQL, Redis, API, Vite, health ordering, service discovery, and observability. PR #568 surfaced the workflow in the README/runbook.

The AppHost model is covered by Cluckwork.AppHost.Tests, but CI does not currently boot the assembled Aspire stack. The existing test layers have different purposes:

• Cluckwork.Api.IntegrationTests uses WebApplicationFactory + Testcontainers for fast, hermetic, in-process API/EF tests and DI access.
• Redis contract tests use a dedicated Testcontainers Redis fixture.
• the Playwright and k6 simulation workflows use Docker Compose to exercise the built production image, Production boot configuration, one-shot commands, and production-like topology.
• image build, Trivy, digest publication, attestation, and release promotion validate the exact artifact that ships.

Aspire can add closed-box assembled-stack coverage and remove some future shell orchestration, but it must not collapse those boundaries or turn the Development AppHost into an unreviewed deployment system.

Official references:

• https://learn.microsoft.com/en-us/dotnet/aspire/testing/overview
• https://learn.microsoft.com/en-us/dotnet/aspire/testing/accessing-resources
• https://aspire.dev/reference/cli/commands/aspire-wait/
• https://aspire.dev/reference/cli/commands/aspire-run/

Goal

Add a small, deterministic Aspire-backed system-smoke layer to CI, then produce an evidence-backed decision for each other GitHub Actions workflow and for any future deployment use.

Aspire should own AppHost lifecycle, dynamic endpoints, dependency ordering, and cleanup where that is the behavior under test. Docker/Testcontainers/Compose remain where they provide isolation or validate the production artifact.

Proposed work

1. Make the AppHost safe for automated tests

• Add an explicit CI/system-test mode without changing the normal local-development contract.
• Use ephemeral or run-unique PostgreSQL storage in tests; never reuse the fixed developer volume cluckwork-apphost-postgres-pg18.
• Do not load developer user-secrets. Generate test-only parameters/credentials at runtime and never print, persist, hash, or upload them.
• Keep randomized host ports and support concurrent/isolated runs.
• Preserve the local AppHost's Development migrations, persistent volume, pinned images, health checks, references, and Vite proxy behavior outside test mode.
• Add model guards proving the local and CI storage/configuration modes cannot be accidentally swapped.

2. Add a closed-box Aspire system-test project

Create a focused project such as tests/Cluckwork.SystemTests using the locked Aspire.Hosting.Testing 13.5 package.

Start one DistributedApplication fixture for the suite, wait for resources through Aspire's health/resource APIs, and dispose it once. Cover only assembled-stack contracts that the existing in-process suite cannot prove:

• exactly one postgres, redis, api, and web resource becomes healthy;
• a client resolved from the dynamic web endpoint reaches Vite and proxies /api to the API;
• a fresh-database unknown-user login returns exactly 401, application/problem+json, and Auth.NoOwnerProvisioned;
• a database-backed request proves API → PostgreSQL wiring;
• a rate-limit/shared-state request proves API → Redis wiring;
• health and startup ordering fail closed when a required dependency cannot become ready;
• teardown leaves no test-owned AppHost process, container, or volume.

Share the expensive AppHost once per test collection. Do not port the full API integration suite into separate-process Aspire tests.

3. Add a dedicated GitHub Actions smoke job

• Install the repository's pinned .NET, Node, and Aspire CLI/package versions; restore NuGet in locked mode and npm with npm ci.
• Run the new system-test project on a Docker-capable runner with a hard timeout.
• Measure and record cold/warm duration before making it a required PR gate; target a bounded smoke suite, not another full E2E suite.
• Use always() cleanup and verify ownership before stopping/removing anything.
• On failure, retain only sanitized resource state and bounded logs. Never upload dashboard URLs/tokens, raw environment/configuration, request bodies, user-secret values, or credential-bearing traces/reports.
• Mutation-prove that breaking a reference, wait/health relationship, Vite proxy target, PostgreSQL wiring, or Redis wiring makes the job fail.

Prefer DistributedApplicationTestingBuilder for the test-owned lifecycle. If a workflow-level CLI probe is still useful, use the pinned CLI with aspire run --isolated --detach --non-interactive, explicit aspire wait --apphost ..., and a trap/finally-owned aspire stop; do not select or stop an unrelated AppHost.

4. Evaluate existing workflows one by one

Record a keep/adopt/defer decision with measured cost and lost/gained coverage for:

• ci.yml build-and-test and schema-doc generation;
• e2e-smoke.yml Playwright simulation;
• k6-baseline.yml load/rehearsal workflow;
• CodeQL/security/dependency workflows;
• image build + Trivy;
• publish, attestation, release promotion, and repair dispatch;
• failure diagnostics/log collection.

Expected starting position:

• Keep Testcontainers for WebApplicationFactory, EF/migration/schema, and Redis contract tests. They need hermetic resources, in-process DI/service access, or only one dependency—not the whole stack.
• Keep Docker Compose for Playwright/k6 production-image rehearsal. The local AppHost runs Development source projects with a separate Vite process and is not equivalent to the shipped single-container image or Production boot configuration.
• Keep Docker/buildx and registry tooling for image construction, Trivy, digest publication, attestation, promotion, and release repair. Aspire orchestration cannot substitute for validating/promoting the exact shipped bytes.
• Consider Aspire only where the AppHost graph itself or a development-style assembled stack is the subject under test.

5. Research deployment support without implementing it implicitly

Evaluate Aspire publish/deploy capabilities as a separate decision, not as a side effect of the smoke-test work.

The decision must preserve these repository invariants:

• the repository remains host/provider agnostic; provider manifests, IaC, concrete environment values, DNS/CDN/edge configuration, and secret-store wiring stay in the deployment/ops repository;
• CI continues to build, scan, attest, and publish the existing digest-addressed production image;
• release promotion remains a server-side retag of the already-tested digest, never an Aspire-triggered rebuild;
• deployment consumes a verified digest and preserves the existing migrate-before-serve ordering, health probes, one-shot verbs, production boot guards, tzdata/ICU, proxy/TLS requirements, and rollback story;
• no preview Aspire deployment API becomes production-critical without an ADR/design, threat review, rollback proof, and a separately approved implementation issue.

Conclude with one of: do not adopt, use only for portable artifact export, or open a separately scoped deployment integration issue. Document why.

Non-goals

• Replacing WebApplicationFactory or the existing API/EF integration suite.
• Replacing Testcontainers where a test needs one hermetic dependency.
• Replacing the Compose-based production-image Playwright/k6 rehearsal.
• Replacing Docker/buildx, Trivy, registry promotion, attestations, or release-please.
• Adding provider-specific deployment configuration to this repository.
• Treating the Development AppHost as the production topology.

Acceptance criteria

• CI-safe AppHost mode is isolated from local persistent development state and developer user-secrets.
• Model guards cover both local and CI AppHost modes and are mutation-proven.
• A closed-box Aspire fixture starts the complete stack once and verifies web proxy, API, PostgreSQL, Redis, health, and startup-order contracts.
• Three sequential local runs pass with no leftover AppHost process, container, or volume; relevant failure mutations go red.
• A bounded GitHub Actions job runs the Aspire smoke suite and records cold/warm runtime before being made required.
• Failure diagnostics are useful but contain no credential values, dashboard tokens, request bodies, or raw secret-bearing artifacts.
• Existing workflows have a documented keep/adopt/defer matrix grounded in measured coverage and cost.
• Production-image, security-scan, attestation, digest-promotion, and Compose rehearsal gates remain intact unless a separate reviewed change proves equivalent or stronger coverage.
• Deployment evaluation ends in an explicit decision and creates a separate issue before any provider/deployment implementation.
• README.md, docs/runbooks/README.md, and the Aspire runbook document the CI/system-test workflow if adopted.

Dependency

Follow-up to #565. The local AppHost foundation is merged; this issue should not reopen or redesign that local workflow except for a narrow, explicitly guarded test-mode seam.