Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions doc/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,24 @@ guest resume behavior:
python3 scripts/nvx.py benchmark --suite snapshot-restore-vcpu --backend mshv --processors 8 --memory-mib 128 --warmups 1 --runs 5 --snapshot-profile --skip-build --output-dir data/runs/restore-vcpu-mshv-capacity-8
```

Measure the same activation from one platform snapshot, the clone tier that
a fleet restores with a different size each time:

```console
python3 scripts/nvx.py benchmark --suite platform-snapshot-restore-vcpu --backend kvm --processors 8 --memory-mib 512 --warmups 1 --runs 5 --skip-build --output-dir data/runs/platform-restore-vcpu-kvm
```

A platform command line carries no tenant configuration, so the suite cannot
use the pre-banner capture: the capture boots with `maxcpus=1` as its only
command-line option, a distro layer, and a scratch disk, and the guest captures
from its shell with `--snapshot-tier platform`. Each restore reattaches the
blocks and reaches its marker after the guest finishes the platform repair,
which brings the target prefix online, refreshes the clone's identity, and
acknowledges the restore gate. Its latency is therefore comparable between
targets and backends, not with `snapshot-restore-vcpu`. `--snapshot-profile`
adds the lifecycle phases of each target, and the results stay out of the
performance CSVs.

Measure restore-time memory activation from one immutable 512 MiB snapshot
with 2 GiB of ABI-reserved capacity:

Expand Down Expand Up @@ -260,6 +278,7 @@ python3 scripts/nvx.py performance collect --platform linux-kvm-baremetal --comm
| Shell snapshot | `benchmark --suite shell-snapshot` | Compares cold boot with lifecycle-aligned snapshot restore at 128, 256, and 512 MiB. |
| Shell snapshot restore | `benchmark --suite shell-snapshot-restore` | Captures an unmeasured lifecycle-aligned snapshot and measures only restore latency for the selected memory sizes. |
| Restore-time vCPU activation | `benchmark --suite snapshot-restore-vcpu --processors 8` | Restores one boot-online-1, capacity-8 snapshot at online targets 1/2/4/8 and reports latency plus peak RSS. |
| Platform restore-time vCPU activation | `benchmark --suite platform-snapshot-restore-vcpu --processors 8` | Restores one boot-online-1, capacity-8 platform snapshot at online targets 1/2/4/8, through the platform repair, and reports latency plus peak RSS. |
| Network snapshot | `benchmark --suite network-snapshot` | Compares a network-ready cold boot with snapshot restore and verifies gateway connectivity. |
| Snapshot lifecycle profile | `benchmark --suite snapshot-profile` | Retains raw capture and restore phase records and summarizes 128/256/512/1024 MiB warm/cold restores. |
| Virtio device restore profile | `benchmark --suite device-restore-profile` | Verifies active and driver-unbound deferred restore for console, network, and virtio-fs; emits standalone diagnostic JSON and raw logs. |
Expand Down
4 changes: 4 additions & 0 deletions doc/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,10 @@ it once, and the first snapshot a second time to prove that a restore leaves it
reusable. `restore-processors` captures one boot-online CPU with capacity 8 and
probes the CPUs that each restore target activated. After the 1/2/4/8-CPU
restores, it restores the same snapshot once without `--restore-processors`.
`platform-restore-processors` does the same for a platform snapshot of
capacity 4, with the requested counts up to 4: its capture boots with
`maxcpus=1` and sandbox blocks, each restore must also report the platform
repair, and OpenVMM must refuse the counts 0, 3, and 8 before the guest runs.
After the warp probe, every `smp-snapshot` and `restore-processors` restore
runs `/sbin/nvx-time status`, which waits for the restore's deferred checks;
it must exit 0 with a passing `NVX-TIME-ABI ... phase=restore` line that
Expand Down
48 changes: 28 additions & 20 deletions doc/openvmm-upstream-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,14 @@
[Upstream status](#upstream-status).
- Mapped fork tip: `a0556d7d85ce1fff89f5b45a98f15c5c79cbb6fb`, position 157. The graph, PR
summary, and commit mapping below cover positions 001-157 and the fixups at positions
334-338.
- NVX's pinned OpenVMM revision: `8a0d945edca13b301a5152f225702057337b2a56`, position 338,
which microsoft/openvmm#4734 proposes for `feature/microvm/dev`. It is one commit on
`79a501b63`, the head of the `feature/microvm/dev` and `feature/microvm/release/v2.3`
branches of microsoft/openvmm, and 181 commits past the mapped tip on the same first-parent
chain. Positions 158-333 are not yet assigned to PRs; see
[Commits after the mapped tip](#commits-after-the-mapped-tip).
334-340.
- NVX's pinned OpenVMM revision: `2b681564dd63483f2f9c9f9bba3b98a4c00e9680`, position 340.
Positions 339 and 340, which microsoft/openvmm#4735 proposes for `feature/microvm/dev`, are
two commits on position 338, `8a0d945ed`, the head of `feature/microvm/dev` since
microsoft/openvmm#4734 merged it. Position 338 is one commit on `79a501b63`, the head of the
`feature/microvm/release/v2.3` branch of microsoft/openvmm, and the pin is 183 commits past
the mapped tip on the same first-parent chain. Positions 158-333 are not yet assigned to PRs;
see [Commits after the mapped tip](#commits-after-the-mapped-tip).
- Upstream main: `22c0485be597f87a76771e1db03eaa823a685ffc` (2026-10-22), the fork base.

## Upstream status
Expand Down Expand Up @@ -202,10 +203,10 @@ flowchart TB
P17["PR17<br/>Capture + authoritative restore<br/>P - 11 commits"]
P18["PR18<br/>Host attachments<br/>P - 11 commits + split tip"]
P19["PR19<br/>Block roles + tiers<br/>N/P - 3 commits"]
P20["PR20<br/>Restore activation<br/>P - 8 commits + split tip"]
P20["PR20<br/>Restore activation<br/>P - 9 commits + split tip"]
P21["PR21<br/>Network egress policy<br/>P/N - 6 commits"]
P22["PR22<br/>Filesystem deny policy<br/>P - 2 commits"]
P23["PR23<br/>Authenticated control console<br/>N - 9 commits + split tip"]
P23["PR23<br/>Authenticated control console<br/>N - 10 commits + split tip"]
P24["PR24<br/>Managed workload/reporting<br/>N - 5 commits"]
P25["PR25<br/>Management plane<br/>P - 3 commits + split tip"]
P26["PR26<br/>Profiling benchmark<br/>U - 1 commit"]
Expand Down Expand Up @@ -302,7 +303,8 @@ The physical branch is one first-parent chain:
-> unassigned [158-333]
-> PR10 fixup [334] -> PR05/PR20/unassigned split [335]
-> PR14 fixup [336] -> PR09 fixup [337]
-> PR02 fixup [338] -> 8a0d945ed (NVX pin)
-> PR02 fixup [338] -> PR20 fixup [339] -> PR23 fixup [340]
-> 2b681564d (NVX pin)
```

For upstream review, PR01-PR08 and PR14 can be rebased independently onto current upstream main,
Expand All @@ -318,14 +320,16 @@ the commits that [Rebase conflicts and fixups](#rebase-conflicts-and-fixups) nam
into PR10, position 336 into PR14, position 337 into PR09, and position 335 into PR05, the PR20
fixup at position 151, and the unassigned positions 163 and 197. Fold position 338 into PR02's
position 010, which microsoft/openvmm#4558 carries. PR27 can be prepared once PR22 lands; its
generic confinement commit can be submitted independently if preferred.
generic confinement commit can be submitted independently if preferred. Fold position 339 into
PR20's position 120 and position 340 into PR23's position 136.

## Commits after the mapped tip

The NVX pin extends the chain past position 157 without merges. Of the 181 later commits,
The NVX pin extends the chain past position 157 without merges. Of the 183 later commits,
positions 158-333 are not yet classified or assigned to PRs, positions 334-337 are the fixups
of the `feature/microvm/release/v2.3` rebase, and position 338 is a PR02 fixup. This table
groups them by topic in chain order; ranges are inclusive, as in the [PR summary](#pr-summary).
of the `feature/microvm/release/v2.3` rebase, position 338 is a PR02 fixup, and positions 339
and 340 are PR20 and PR23 fixups. This table groups them by topic in chain order; ranges are
inclusive, as in the [PR summary](#pr-summary).

| Positions | Commit range | Count | Topic |
| ---: | --- | ---: | --- |
Expand Down Expand Up @@ -361,6 +365,8 @@ groups them by topic in chain order; ranges are inclusive, as in the [PR summary
| 333 | `fcb4ea1c6` | 1 | Petri and the VMM tests fall back to the host CPU profile where `auto`'s is unsupported |
| 334-337 | `9bc90f55f..79a501b63` | 4 | Fixups of the `feature/microvm/release/v2.3` rebase; see [Rebase conflicts and fixups](#rebase-conflicts-and-fixups) |
| 338 | `8a0d945ed` | 1 | Windows mapped-file flushes mark every page of a writable view modified and write the range in one ascending pass, so NTFS no longer zero-fills ranges that are then written again (microsoft/openvmm#4734); fold into 010 |
| 339 | `2f51e8bab` | 1 | Platform snapshots opt into restore-time vCPU activation with one canonical `maxcpus=` token, so each clone of a platform capture can bring its own VP prefix online; fold into 120 |
| 340 | `2b681564d` | 1 | The control-console authentication timeout counts only while the broker reads host input, so a gated restore no longer detaches a host that authenticated at launch; fold into 136 |

Some of these commits change mapped work rather than add to it. Positions 246-250 remove the
legacy microVM clock paths and change files that PR10 and PR17 also change. Position 252 makes
Expand Down Expand Up @@ -393,10 +399,10 @@ as whole physical commits here but should be folded into their target commits be
| PR17 | P | microVM snapshot capture and authoritative restore | 093-103 | `693d64ca8..d0cb54c9d` | 11 | PR16 |
| PR18 | P | microVM host attachments and device persistence | 104-113; 153 (split); 154 (fixup) | `9611571b9..b7816551d`; `9638a8f42` (split); `e1586485a` | 11 + split | PR17 |
| PR19 | N/P | Sandbox block roles and snapshot tiers | 114-116 | `cec1a5510..59cb42a01` | 3 | PR18 |
| PR20 | P | Restore-time processor and memory activation | 117-123; 151-152; 335 (split) | `9f2d6eeee..764d668d1`; `73a324390..9be096e1b`; `d2e83bdfd` (split) | 8 + split | PR19 |
| PR20 | P | Restore-time processor and memory activation | 117-123; 151-152; 335 (split); 339 (fixup) | `9f2d6eeee..764d668d1`; `73a324390..9be096e1b`; `d2e83bdfd` (split); `2f51e8bab` | 9 + split | PR19 |
| PR21 | P/N | Network egress policy | 124-129 | `44ec6b08d..b6e8dc870` | 6 | PR20; core can start after PR07 + PR08 + PR13 |
| PR22 | P | Filesystem deny policy | 130-131 | `07eb316dc..7894a3b50` | 2 | PR21 |
| PR23 | N | Authenticated control console | 132-140; 153 (split) | `2d649d42c..ed9d48548`; `9638a8f42` (split) | 9 + split | PR22; protocol/broker can start after PR03 + PR04 + PR13 |
| PR23 | N | Authenticated control console | 132-140; 153 (split); 340 (fixup) | `2d649d42c..ed9d48548`; `9638a8f42` (split); `2b681564d` | 10 + split | PR22; protocol/broker can start after PR03 + PR04 + PR13 |
| PR24 | N | Managed workload and outcome reporting | 141-145 | `ddaa1810b..ad0b68265` | 5 | PR23 |
| PR25 | P | Management plane | 146-147; 152 (split); 155 (fixup) | `941f8f720..a1820db81`; `9be096e1b` (split); `e9f23af5d` | 3 + split | PR24 |
| PR26 | U | Profiling benchmark | 148 | `d90539d5a` | 1 | PR25 in the current stack |
Expand All @@ -407,7 +413,7 @@ as whole physical commits here but should be folded into their target commits be

Positions 152 and 153 intentionally appear under two PRs because their commits must be split when
forming submission branches. Positions 154 and 155 are fixups that should be folded into positions
104 and 146, respectively. Tip positions 151-155 and the fixups at positions 334-338 are listed
104 and 146, respectively. Tip positions 151-155 and the fixups at positions 334-340 are listed
under their destination PRs rather than in numerical order, and position 335 appears under both
PRs that it changes. The counts distinguish whole commits from selected split hunks.
Struck-through entries landed upstream before the fork base. The `microvm/v2.2` and
Expand Down Expand Up @@ -438,7 +444,7 @@ Partial upstream overlap is called out without striking the remaining fork work,
- 008 `8d72321f5` - mesh_protobuf: clone type-erased messages and expose encoded length (**partly landed upstream**: microsoft/openvmm#4556, merged as `96647c331`, derives `Clone`; this commit now adds only `ProtobufAny::value_len` and a clone check)
- 009 `15a895dd8` - sparse_mmap: add private copy-on-write file views (**open upstream**: microsoft/openvmm#4557)
- 010 `a95f8ef33` - sparse_mmap: flush file-backed ranges with bounded concurrency (**open upstream**: microsoft/openvmm#4558)
- 338 `8a0d945ed` - sparse_mmap: flush writable file views whole, in one ascending pass (**fold into 010**; proposed for `feature/microvm/dev` in microsoft/openvmm#4734)
- 338 `8a0d945ed` - sparse_mmap: flush writable file views whole, in one ascending pass (**fold into 010**; merged into `feature/microvm/dev` by microsoft/openvmm#4734)

</details>

Expand Down Expand Up @@ -679,7 +685,7 @@ that upstream lacks.
</details>

<details>
<summary><strong>PR20 - Restore-time processor and memory activation (P)</strong> - 8 commits + split tip</summary>
<summary><strong>PR20 - Restore-time processor and memory activation (P)</strong> - 9 commits + split tip</summary>

- 117 `9f2d6eeee` - chipset: expose a stable generation ID and restore entropy on portb
- 118 `310b3e8f5` - openvmm: gate external input until post-restore repair completes
Expand All @@ -691,6 +697,7 @@ that upstream lacks.
- 151 `73a324390` - virt_mshv: synchronize restored TSCs only on created VPs
- 152 `9be096e1b` - openvmm_core: instantiate only the requested VP prefix on MSHV restore (**split with PR25**: worker and backend changes belong in PR20; the gRPC reference hunk belongs in PR25)
- 335 `d2e83bdfd` - virt_mshv: adapt the microVM partition to partition scrub (**split fixup with PR05**: the hunk for position 151 belongs in PR20)
- 339 `2f51e8bab` - microVM: let platform snapshots opt into restore-time vCPU activation (**fold into 120**)

</details>

Expand All @@ -715,7 +722,7 @@ that upstream lacks.
</details>

<details>
<summary><strong>PR23 - Authenticated control console (N)</strong> - 9 commits + split tip</summary>
<summary><strong>PR23 - Authenticated control console (N)</strong> - 10 commits + split tip</summary>

- 132 `2d649d42c` - serial_core: expose local peer identity and active-peer eviction
- 133 `cba534b9d` - serial_socket: report Unix and named-pipe peer identities
Expand All @@ -727,6 +734,7 @@ that upstream lacks.
- 139 `6c1043d08` - openvmm_helpers: record the control console in the microVM machine contract
- 140 `ed9d48548` - openvmm_entry: add authenticated microVM control-console endpoints
- 153 `9638a8f42` - Restore listener endpoint rebinding (**split with PR18**: authenticated control-listener portions belong in PR23; boot-listener and shared attachment-contract portions belong in PR18)
- 340 `2b681564d` - virtio_console: time host authentication only while host input is read (**fold into 136**)

</details>

Expand Down
4 changes: 3 additions & 1 deletion doc/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -534,7 +534,9 @@ complete while the restored vCPU remains stopped. With
the source must have captured a canonical `maxcpus` boot-online prefix, and the
guest onlines exactly the requested prefix before readiness and host input
release. Targets are limited to 1, 2, 4, or 8 and cannot be below the captured
boot-online count or above capacity. Legacy snapshots reject the option. The
boot-online count or above capacity. Platform snapshots opt in the same way, so
each clone of one platform capture can bring its own prefix online. Legacy
snapshots reject the option. The
peer must accept and read while startup is in progress; Windows flush
completion waits for the named-pipe peer to consume the frame. Failure to
write the complete event aborts and tears down the restore.
Expand Down
8 changes: 5 additions & 3 deletions doc/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -814,7 +814,8 @@ and `sandbox-proxy-only` drive the public `sandbox` lifecycle commands (see
Ubuntu layer and scratch template, which packages omit. Alpine remains the
default. Ubuntu and Azure Linux cannot act as sandbox control, so they reject
the Alpine-control-only `sandbox-blocks` and `scratch-snapshot` scenarios and
the sandbox-control-dependent `snapshot-tiers`, `managed-exec-config`,
the sandbox-control-dependent `snapshot-tiers`, `platform-restore-processors`,
`managed-exec-config`,
`sandbox-lifecycle`, and `sandbox-proxy-only` scenarios. Ubuntu also rejects
the Alpine-prompt-specific `console-snapshot` scenario. `--debug-kernel` boots
the CI debug kernel, `build/vmlinux-debug`, whose soft-lockup and hung-task
Expand Down Expand Up @@ -881,7 +882,8 @@ characters, and otherwise `null`. A `scenario` record also has the
`scenario`, the `guest`, `backend`, and `kernel`, and its `processors`: the
`--processors` counts that the execution covers, such as `[4]` for `smp` on 4
vCPUs, the largest count for `time-abi-conformance`, all of them for
`smp-snapshot` and `restore-processors`, or `null` for a scenario that doesn't
`smp-snapshot` and `restore-processors`, those up to 4 for
`platform-restore-processors`, or `null` for a scenario that doesn't
take them. A failed run ends with the failed scenario's record and the
`teardown`; a run that is killed keeps every record that it completed and has
no `teardown`.
Expand Down Expand Up @@ -1433,7 +1435,7 @@ python3 scripts/nvx.py benchmark [OPTIONS]

| Option | Default | Description |
| --- | --- | --- |
| `--suite {boot,snapshot,restore,e2e,phase2,snapshot-profile,all,cold-start,device-io,device-restore-profile,network-snapshot,performance,shell-snapshot,shell-snapshot-restore,snapshot-restore-memory,snapshot-restore-vcpu,virtfs}` | `boot` | Select an acceptance, diagnostic, or workload suite. |
| `--suite {boot,snapshot,restore,e2e,phase2,snapshot-profile,all,cold-start,device-io,device-restore-profile,network-snapshot,performance,platform-snapshot-restore-vcpu,shell-snapshot,shell-snapshot-restore,snapshot-restore-memory,snapshot-restore-vcpu,virtfs}` | `boot` | Select an acceptance, diagnostic, or workload suite. |
| `--backend {whp,kvm,mshv,both}` | `both` on Windows; `kvm` elsewhere | Select the hypervisor backend. |
| `--platform NAME` | inferred OS/backend | Record the host-typed performance series. |
| `--openvmm-dir PATH` | `openvmm/` | Select the OpenVMM repository. |
Expand Down
Loading
Loading