Add Copilot-driven adversarial stress and security testing #102

Summary

Add a separate Copilot-driven adversarial test harness for NVX. Copilot CLI should act as an adaptive strategist that selects or generates guest-side stress and security probes, while a deterministic broker exclusively controls NVX, enforces resource limits, and judges containment.

The tests must require Copilot CLI to already be installed and authenticated on the controller. They must not install Copilot CLI or initiate login during a run. The existing .github/workflows/copilot-setup-steps.yml installs gh-aw and does not satisfy this prerequisite.

Goals

Exercise three boundaries:

1. Workload-to-agent or workload-to-guest isolation escape.
2. Guest-kernel-to-OpenVMM or guest-to-host VM escape.
3. Availability failures such as OpenVMM crashes, hangs, leaks, incomplete teardown, or inability to launch a clean VM afterward.

Expected workload termination and deliberate guest kernel failure may be valid test actions. OpenVMM failure, host failure, policy bypass, host-canary modification, unexpected network access, or leaked resources must fail the campaign.

Proposed architecture

• Copilot controller: Runs Copilot CLI non-interactively with fixed action, time, and AI-credit budgets.
• Typed broker: Exposes only allowlisted NVX operations and validates every argument. Copilot must not receive unrestricted host shell access.
• NVX executor: Reuses OpenvmmProcess, ControlSession, sandbox_lifecycle, console markers, and structured outcome reports.
• Independent watchdog: Monitors host health, OpenVMM state, resource usage, filesystem/network canaries, and teardown without relying on Copilot.
• Disposable target: Runs either an unprivileged managed sandbox workload or a privileged test guest, depending on the campaign.

Prefer a two-node topology: a Copilot controller and a disposable backend executor. This is required to reliably classify a target-host crash.

Execution modes

Workload adversary

Use the managed lifecycle:

sandbox provision -> sandbox start -> repeated sandbox exec -> sandbox stop

Exercise workload identity, namespaces, capabilities, no_new_privs, cgroups, overlay storage, network policy, and control-channel isolation.

Guest adversary

Use a test-only root guest and the existing microVM process harness to exercise guest-controlled PMIO, virtio queues, console traffic, network packets, block I/O, virtio-fs requests, and abnormal guest shutdown paths.

Snapshot adversary

Exercise capture and restore during CPU, block, console, filesystem, and network activity. Use the low-level OpenVMM lifecycle harness because the public nvx sandbox command does not currently orchestrate sandbox snapshots.

Initial campaign families

• CPU, memory, process, disk, inode, and output exhaustion.
• Signal storms, process/thread churn, repeated managed exec, timeout, cancellation, and reconnect behavior.
• Mount, namespace, cgroup, ptrace, capability, device, and initramfs-access probes.
• Filesystem traversal, symlink/junction races, aliasing, concurrent rename/open, and read-only-layer mutation attempts.
• Host-loopback, local canary service, port/protocol, ingress, and egress policy probes.
• Malformed or high-rate virtio descriptors, packets, FUSE operations, and PMIO sequences.
• Guest panic, abnormal exit, reset, snapshot-under-load, repeated restore, and cross-instance state isolation.

Start with deterministic prebuilt primitives. Model-generated scripts may be added later, but they must execute only inside the target guest or a separate disposable builder.

Adaptive loop

1. Run scripts/nvx.py verify and validate backend access and artifact provenance.
2. Require copilot --version and a non-interactive authenticated smoke prompt to succeed.
3. Create a unique run directory and plant synthetic filesystem and network canaries.
4. Boot and stop a baseline VM.
5. Give Copilot the available primitives, normalized observations, coverage map, and remaining budget.
6. Require exactly one schema-validated action.
7. Execute it through the broker and run deterministic oracles.
8. Stop on budget exhaustion, coverage convergence, or an anomaly.
9. Boot and stop a fresh canary VM after the campaign.

Treat all guest output as untrusted data. Length-limit and encode it before returning it to Copilot so guest text cannot become controlling prompt instructions.

Proposed CLI and layout

python3 scripts/nvx.py test-adversarial \
  --backend kvm \
  --campaign workload-isolation \
  --budget-seconds 900 \
  --seed 1234 \
  --output-dir build/test-results/adversarial-kvm

scripts/nvx_tools/adversarial.py
scripts/nvx_tools/adversarial_broker.py
scripts/nvx_tools/adversarial_oracles.py
scripts/nvx_tools/adversarial_cases/*.json
.github/agents/nvx-adversary.md
.github/workflows/adversarial.yml
doc/adversarial-testing.md

Provide --replay actions.jsonl so a campaign can be reproduced without Copilot. Minimize failures by replaying progressively shorter action prefixes on fresh targets, then promote the minimized case into the deterministic test-microvm suite.

Required oracles and artifacts

Record and verify:

• NVX/OpenVMM commits, dirty state, OS, backend, host type, Copilot version/model, seed, and budgets.
• OpenVMM PID, exit status, readiness, control ping, action timeout, and outcome category.
• Every structured teardown field and absence of leftover processes, sockets, mounts, endpoints, or storage.
• Filesystem canary hashes and network-canary access logs.
• Host resource usage and external executor heartbeat.
• A successful clean NVX boot following the campaign.
• Copilot responses, validated actions, guest/OpenVMM/watchdog logs, outcome reports, and a replay manifest.

Potential escape payloads and complete exploit-like transcripts must use access-controlled security storage rather than normal public PR artifacts.

CI and containment

Do not run escape attempts on the existing persistent self-hosted runner fleet unless each machine is reimaged after every job. Use dedicated ephemeral executors with no production or GitHub credentials, no production network access, bounded resources, synthetic targets only, and an external quarantine/reimage mechanism.

Initially run the suite through trusted manual dispatch and scheduled dev jobs only. Never run fork code. Adaptive campaigns should not gate pull requests directly; only minimized deterministic regressions should become normal PR gates.

Acceptance criteria

• Missing or unauthenticated Copilot CLI fails preflight rather than skipping tests.
• Copilot cannot execute arbitrary host commands or alter budgets and oracles.
• Every accepted action is schema-validated, recorded, and replayable.
• Injected test-only policy violations and availability faults are detected correctly.
• All cleanup and post-campaign canary-boot checks are deterministic.
• KVM, MSHV, and WHP can each run a bounded campaign on disposable infrastructure.
• A suspected escape automatically quarantines the executor and prevents reuse.