Skip to content

id(world): wire manifest apply — module install + schedules over the wire (#807) - #1820

Merged
dezren39 merged 10 commits into
mainfrom
queue/2661-807-apply-wiring
Oct 30, 2026
Merged

dezren39 merged 10 commits into
mainfrom
queue/2661-807-apply-wiring

Conversation

@dezren39

@dezren39 dezren39 commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Closes the apply-wiring gap in #807: id world apply reconciled grants, ceiling and profile over the wire, but module and schedule steps were only reported, and diff/plan compared schedules and the install seed against fabricated empty state.

What changed

  • apply_steps applies every step (pkgs/id/src/commands/world.rs): InstallModule compiles a manifest source on the host, or downloads a known module hash and re-installs its bytes under the manifest seed; ScheduleAt/CancelSchedule call the new admin client operations. step_is_applied now counts every kind — the id(world-manifest): PR #801 — 'id world apply' prints 'applied N step(s)' for steps it explicitly did not apply, and diff/plan run against a fabricated seed=0 / empty-schedules state #818 guard stays explicit per kind so a future reported-only step cannot silently claim credit.
  • New admin wire operations (world_session): set_schedule / cancel_schedule frames, admin-only, journaled in the actor before the reply. Re-setting a pending id journals a cancel before the replacement set (journal replay refuses two pending sets for one id). Cancelling an absent id is a no-op.
  • Real live state (world_state frame): current_world_state now reads the pending schedules and the install seed from the server instead of BTreeMap::new() / 0. The seed is returned only to admin readers — a capability-only read (e.g. diff) gets the schedules and a null seed, so the seed never enters a participant projection (Install seed is readable by every joiner #240). Apply always reads with its admin token, so install steps use the real current seed.
  • Shell contract: additive revision 2 — client frames set_schedule, cancel_schedule, world_state; reply frames schedules, world_state.
  • CLI doc for apply updated to match.

Tests

  • Session service tests: schedule set is admin-only, refused changes change nothing, re-set replaces the fire time, cancel is a no-op when absent.
  • Protocol round-trips: set_schedule/cancel_schedule replies carry the full pending map; world_state hides the seed from a capability reader (null) and returns the real seed (4242, from a test install) to the admin reader; no credential is refused.
  • commands::world: applied-count test now asserts every step kind counts; new test for manifest source reading (directory main + .roc siblings, single file).
  • Existing world/world-manifest test suites run in CI (id-check).

Second identical apply is a no-op by construction: plan compares against the now-real live state (module hash, schedules, seed), so a converged world plans zero steps.

Queue task 2661 (from triage task 2247).


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

apply_steps now applies every step over the wire: InstallModule compiles
a manifest source on the host or downloads a known hash and re-installs
its bytes under the manifest seed; ScheduleAt/CancelSchedule go through
new admin frames (set_schedule/cancel_schedule) that journal before
replying, with replace-by-cancel-then-set semantics in the actor.

current_world_state reads real live state through a new world_state
frame: pending schedules for capability or admin readers, and the
install seed only for admin readers — the seed never enters a
participant projection (#240). Shell contract gains the three client
frames and two reply frames (additive, revision 2). step_is_applied now
counts every kind (#818 semantics preserved: only wired steps count).

Tests: session-level admin-only/replace/no-op-cancel coverage, protocol
round-trips incl. seed hidden from capability readers, compile-source
reading, updated applied-count test.
@dezren39
dezren39 requested a review from a team as a code owner October 28, 2026 20:48
@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 18 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: deec8b36-c8b4-4933-8044-7524872ec8b8
Commits

Reviewing files that changed from the base of the PR and between c58507a and 569abd2.

Files selected for processing: 13
  • pkgs/id/contract/shell-contract.v1.json
  • pkgs/id/src/cli.rs
  • pkgs/id/src/commands/world.rs
  • pkgs/id/src/meta_client.rs
  • pkgs/id/src/shell_contract/tests.rs
  • pkgs/id/src/world/actor.rs
  • pkgs/id/src/world/core.rs
  • pkgs/id/src/world/handle.rs
  • pkgs/id/src/world/tests.rs
  • pkgs/id/src/world_net.rs
  • pkgs/id/src/world_session/mod.rs
  • pkgs/id/src/world_session/tests.rs
  • pkgs/id/src/world_session/types.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T14:47:34.463584Z 569abd2 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9e97c7357

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkgs/id/src/world/actor.rs Resolved
Comment thread pkgs/id/src/commands/world.rs Resolved
Comment thread pkgs/id/src/commands/world.rs Resolved
The apply-wiring change added ServerFrame::Schedules and
ServerFrame::WorldState and registered both in shell-contract.v1.json,
but the exhaustive server_frame_type test helper was not updated, so
cargo-check failed with E0004. Add both arms and both
SERVER_FRAME_TYPES entries.
@dezren39

Copy link
Copy Markdown
Member Author

CI settled on the previous head and I triaged all three failures:

  • test-unit — PR-caused, fixed in 6f9451c: the_contract_check_reports_conformance_or_every_violation expected "3 frames conform to shell contract v1.1", but this PR bumps the contract to revision 2 (v1.2) when adding the set_schedule/cancel_schedule/world_state frames. Updated the stale expectation in world.rs.
  • clippy-lint — pre-existing on main: both errors are expect() on Option at src/tags.rs:1484,1499, which this PR does not touch; tracked as id(tags): clippy-lint fails on main — expect() on Option in parse_search_query (tags.rs:1484,1499) breaks the flake check for every PR #2008.
  • crash-gate-ingest — infra: the build failed fetching config.sub from git.savannah.gnu.org (curl connect timeout after 132s), unrelated to this change.

All other checks passed on the previous head (including test-shard-0/1/2/3, cargo-check, fmt). Parking the queue task until CI on 6f9451c lands; merge follows if test-unit passes with only the pre-existing/infra failures remaining.

@dezren39

Copy link
Copy Markdown
Member Author

Resumed by queue task 2661 (lane C). CI classification on head 6f9451c:

Fix applied: merged current main (c58507a) into the branch — clean merge, new head e4b094b; meta_client.rs in the tree is now main's formatted version verbatim. Pushed and waiting on CI re-run; merge follows if the only remaining failure is the pre-existing clippy #2008.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e4b094bbcf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkgs/id/src/commands/world.rs Resolved
@dezren39

Copy link
Copy Markdown
Member Author

Not merged: four unresolved P2 findings in this PR's new apply path (queue task 2661)

Resumed by queue task 2661 (openchamber scheduled lane). The CI classification from the previous lane still holds — test-unit, all test shards and cargo-check pass; cargo-fmt-check / rustfmt-check / treefmt-check fail only on pkgs/id/src/meta_client.rs:53 (a file this PR does not touch — main itself is not rustfmt-clean there), and clippy-lint only on src/tags.rs:1484,1499 (#2008).

But a Codex review landed on the current head e4b094bb (2026-10-29T07:59Z), after the previous lane's merge plan, with four P2 findings against this PR's own new code. Two bear directly on #807's acceptance ("installs (or compiles) the manifest module and applies schedule set/cancel") and I confirmed both in the tree:

  1. commands/world.rs:983 — the InstallModule { hash } path cannot switch module hash. CONFIRMED. download_module_session returns "requested module is not the active world module" (world_session/mod.rs:1609), and the new path calls client.download_module(capability, hash) for a manifest hash that differs from the active one. So a manifest selecting a different hash cannot install it — this is the core "install the manifest module" acceptance. Needs an admin-authorized stored-module retrieval/activation operation rather than the active-module-only participant download.
  2. commands/world.rs:989 — schedule reconciliation fails at capacity. CONFIRMED. MAX_PENDING_SCHEDULES = 64 and WorldCore::check_schedule rejects an add at capacity (world/core.rs:597); the plan emits ScheduleAt before CancelSchedule, so a manifest that replaces an ID at capacity fails the add and the ? skips the cancel — a manifest whose final map fits the limit cannot be reconciled. Execute cancels for undeclared IDs before additions.
  3. commands/world.rs:978 — profile applied too late. The plan places InstallModule before SetProfile, so a module + profile change instantiates the guest under the old profile (set_execution_profile only affects the next install/restart).
  4. world/actor.rs:1319 — replacement beyond MAX_SCHEDULE_HORIZON_MS loses the existing schedule. The code durably cancels/removes the original before check_schedule rejects the replacement; the caller gets an error but the previously valid schedule is gone, including after journal replay. Validate the replacement before persisting the cancellation.

Not merging until at least (1) and (2) are resolved, since they contradict the acceptance criteria. Parking the queue task (2661) with these as the resume brief.

— repo-worker-code (openchamber scheduled lane), queue task 2661

…ncels and profile first (#807)

Three Codex P2 findings on PR #1820:

(4) AdminScheduleAt persisted the ScheduleCancelled journal entry and removed the pending schedule before check_schedule validated the replacement, so a replacement past the horizon destroyed a valid schedule in memory and on journal replay. check_schedule now has a replacement-aware form (the replaced entry is neither a duplicate nor a use of the capacity) and the actor validates with it before persisting anything; the cancel-then-set journal order itself is unchanged (replay refuses two pending sets for one id). New core test covers duplicate/capacity/horizon under replacement.

(2) apply_steps ran steps in plan order, which emits ScheduleAt before CancelSchedule, so a manifest replacing ids at MAX_PENDING_SCHEDULES failed the add and skipped the cancel. Cancels now execute before additions.

(3) The plan places InstallModule before SetProfile, so a module+profile apply instantiated the guest under the old profile. SetProfile now executes before the remaining steps (a profile applies on the next install).
…#807)

Codex P2 finding (1) on PR #1820: the InstallModule { hash } apply path fetched bytes through the participant download_module, which serves only the ACTIVE module, so a manifest naming a different hash could never install it.

Add an admin-authorized stored-module retrieval: new download_module_admin client frame (shell contract revision 3) - WorldService::stored_module_bytes reads the blob store every install pins into, by hash, active or not, with the same size and hash re-verification as the participant path; commands/world.rs apply now downloads via the admin token and installs the bytes under the manifest seed as before. A hash the host never installed is refused with a clear message - those bytes cannot be conjured.

The participant download is unchanged (active module only, capability-auth). Its streaming tail is shared with the admin session (send_module_bytes), and the client receive path is shared (download_module_with).

Test: peers_install_then_download_the_active_module_by_hash extended - after a second install makes the first module non-active, the participant download refuses it, the admin download returns its exact bytes, a wrong admin token is refused, and an unknown hash is refused.
@dezren39

Copy link
Copy Markdown
Member Author

All four P2 findings addressed — new head 514b9778 (queue task 2661, lane A)

CI classification from the previous heads still stands (test-unit/shards/cargo-check pass; fmt failures only on meta_client.rs:53, clippy only on tags.rs #2008 — both pre-existing, files this PR does not touch). The four Codex P2 findings are fixed in two commits:

  1. Module hash switch (e21a3200+514b9778) — new admin-authorized stored-module retrieval: download_module_admin client frame (shell contract revision 3). WorldService::stored_module_bytes reads the blob store every install pins into, by hash, active or not, with the same size/hash re-verification as the participant path; apply_steps now fetches the manifest's hash with the admin token and installs the bytes under the manifest's seed. A hash the host never installed is refused with a clear message. The participant download is unchanged. Extended test in world_net: after a second install makes the first module non-active, participant download refuses it, admin download returns its exact bytes, wrong token and unknown hash are refused.
  2. Schedule reconciliation at capacity (e21a3200) — apply_steps now executes CancelSchedule steps before ScheduleAt additions, so a manifest replacing ids at MAX_PENDING_SCHEDULES frees its own slots first.
  3. Profile ordering (e21a3200) — SetProfile executes before the remaining steps (in particular InstallModule), so a module installed in the same apply is instantiated under the manifest's profile.
  4. Cancel-before-validate (e21a3200) — WorldCore::check_schedule_replacing validates a replacement as a replacement (the replaced entry is neither a duplicate nor a capacity use), and AdminScheduleAt validates BEFORE persisting the ScheduleCancelled entry; a refused replacement (e.g. past the horizon) now leaves the existing schedule standing in memory and on replay. The cancel-then-set journal order is unchanged. New core test covers duplicate/capacity/horizon under replacement.

Not merged yet: waiting on CI for 514b9778. The queue task is parked on it; merge follows if the only failures are the pre-existing ones above. (One unrelated local note: the worktree holds an uncommitted rustfmt fix for meta_client.rs:53 from an earlier lane's local verification — deliberately left uncommitted, as that file is not this PR's to change.)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 514b97783b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkgs/id/src/commands/world.rs Resolved
Comment thread pkgs/id/src/commands/world.rs Resolved
Comment thread pkgs/id/src/world_session/mod.rs Resolved
…ixes (#807)

CI classification on PR #1820 (run 40853234653): test-unit, crash-gate-sampled and crash-gate-sweep all failed with E0004 - the DownloadModuleAdmin variant added by this PR was handled in the first-frame match (world_session/mod.rs:1252) but not in handle_client_frame for joined participants. It joins its sibling DownloadModule in the already-joined group. cargo-fmt-check diffs in world_net.rs / world_session/mod.rs applied verbatim from the CI log; the meta_client.rs fmt fix (exact CI diff at :53) was already in the worktree uncommitted and is included.
…operation, grants/ceiling before install, native module retrieval

Dispositions on PR #1820 (details on the PR threads): (1) schedule replacement validation already landed in actor.rs - validate-before-cancel via check_schedule_replacing; (2) inactive-module retrieval already landed as DownloadModuleAdmin/stored_module_bytes; (3) profile-before-install and (4) cancels-before-additions already landed in apply_steps ordering. Still live and fixed here: (5) the apply sequence reused one stream per operation - admin frames are served one operation per stream, so download->install (and every later step, and the current_world_state reads) wrote to finished sessions; WorldClient::fresh_session now opens a per-operation session on the same connection, used by current_world_state and every apply step. (6) grants/ceiling could still run after a compile in plan order; ordered_steps (extracted, unit-tested) now orders cancels, profile, grants, ceiling, install, schedule additions. (7) stored_module_bytes only searched the blob store; native installs live only in the module directory, which it now falls back to via ModuleDir::load. Tests: ordered_steps order pinned; wire test sequencing install/download/install on fresh sessions over one connection; stored_module_bytes module-dir fallback.
@dezren39

Copy link
Copy Markdown
Member Author

Parked again — CI starved on head 5990d9b1 (queue task 2661, openchamber scheduled lane)

id workflow run 40868118355 was queued from 11:47Z; its jobs only began at 12:24Z and at 12:46Z 23 of 28 were still queued behind other runs (4 completed: guard, shellcheck, statix, plus the skipped main-tier job). nix ci run 40868118252 is success on the same head, and no new Codex round has landed since 5990d9b.

Nothing about the diff is unresolved — this is a runner-queue wait. Resume: poll run 40868118355; merge when green if the only failures are the two pre-existing ones (meta_client.rs:53 fmt, tags.rs #2008 clippy), otherwise park again.

— repo-worker-code (openchamber scheduled lane), queue task 2661

apply_steps and its reachability-order test call ordered_steps, but the helper was lost from world.rs, so the lib did not compile (CI clippy-lint E0425 at world.rs:976 plus cascade type errors). Restore it as a stable sort by reachability rank: cancel schedules, set profile, set grants, set ceiling, install module, schedule additions.
basic: nix flake check (cargo-fmt-check) failed on dd872c3 with
rustfmt diffs in world.rs (apply_steps call, capability report) and
world_session tests; format-only change, no behaviour difference.
@dezren39
dezren39 added this pull request to the merge queue Oct 29, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to Branch Protection failures Oct 29, 2026
You're not authorized to push to this branch. Visit "About protected branches" for more information.
@dezren39
dezren39 added this pull request to the merge queue Oct 29, 2026
@dezren39 dezren39 added work/assigned Queue worker lifecycle state queue/parked Queue task is parked (waiting on CI, time, or an event) status/parked Deliberately paused with a recorded resume condition labels Oct 29, 2026
@dezren39

Copy link
Copy Markdown
Member Author

🤖 Agent note · picked up (resume)

  • Agent: openchamber scheduled lane (queue worker repo-worker-code)
  • Session / ID: queue task 2661 (small_fix) — attempt 1, resumed from checkpoint stage awaiting_ci
  • Found via: queue claim
  • Task: land F12.1 apply wiring (id: F12.1 apply wiring — module install + schedules reconciliation #807) — this PR is the implementation; remaining step is merging it.
  • Revalidated at 17:50Z: PR OPEN, MERGEABLE; every check green except clippy-lint.

🤖 Agent note · parked

  • Agent: openchamber scheduled lane (queue worker repo-worker-code)
  • Session / ID: queue task 2661 (small_fix) — parked 20 min
  • What I found: the one failing check is not a code failure. basic: nix flake check (clippy-lint) in run 40894646360 died in the nix cache fetch: HTTP error 418 from the local cache proxy plus GitHub API error: Twirp error ... ResourceExhausted "rate limit exceeded" / GitHub Actions Cache throttled Magic Nix Cache, before any rust lint ran. That also failed the aggregate nix flake check job.
  • Action taken: gh run rerun 40894646360 --failed at 17:52Z, so clippy-lint is re-queued on head 569abd29. No code change was needed or made.
  • Why parked: the merge queue is stalled repo-wide (ci: Actions check runs stuck in queued across PRs — no runner pickup since ~05:00Z 2026-10-28 #1599) — positions 1-7 are all AWAITING_CHECKS and nothing has merged since ~10:57Z, so this PR shows UNSTABLE and cannot land regardless of the rerun.
  • Resume condition: if clippy-lint comes back green the PR is fully green (the other 28 checks already pass); merge when the queue moves. If it fails with real rust lints, fix them inside this diff. If it fails again on cache throttling, the blocker is infrastructure, not the PR.

@dezren39

Copy link
Copy Markdown
Member Author

Agent note — parked: pre-existing clippy-lint break blocks merge

  • Queue task: 2661 (openchamber scheduled lane, repo-worker-code), resumed from awaiting_ci.
  • Fresh state of PR id(world): wire manifest apply — module install + schedules over the wire (#807) #1820: OPEN, mergeStateStatus: UNSTABLE. Every check passes except basic: nix flake check (clippy-lint) (run 40894646360 job 123626426997), and that failure is not from this branch:
    • the two errors are clippy::expect_used at pkgs/id/src/tags.rs:1484 and :1499 (input[i..].chars().next().expect(...));
    • pkgs/id/src/tags.rs is not among this PR's 13 changed files;
    • the same two lines exist verbatim on origin/main, so any app/id PR is red right now.
  • The earlier classification on this task ("nix cache throttling, HTTP 418") was only half right: the 418/Twirp ResourceExhausted was a transient cache-fetch failure on an earlier attempt; the rerun got through to clippy and found these real errors.
  • Filed id: clippy-lint red on main — expect_used at src/tags.rs:1484,1499 (clippy 1.99) #2401 to fix it (let-else instead of expect(), plus how to verify with the clippy-lint check rather than cargo check).
  • Parked task 2661 for 30 min. Resume: once id: clippy-lint red on main — expect_used at src/tags.rs:1484,1499 (clippy 1.99) #2401 is merged, merge main into this branch (or just re-run clippy-lint — no conflict expected, different file), confirm the full check set green, then merge and complete task 2661.

@dezren39 dezren39 added queue/claimed Queue worker lifecycle state status/in-progress Queue worker lifecycle state queue/parked Queue task is parked (waiting on CI, time, or an event) status/parked Deliberately paused with a recorded resume condition and removed status/parked Deliberately paused with a recorded resume condition queue/parked Queue task is parked (waiting on CI, time, or an event) status/in-progress Queue worker lifecycle state queue/claimed Queue worker lifecycle state labels Oct 29, 2026
@dezren39

Copy link
Copy Markdown
Member Author

🤖 Agent note · parked

@dezren39 dezren39 added queue/claimed Queue worker lifecycle state status/in-progress Queue worker lifecycle state queue/parked Queue task is parked (waiting on CI, time, or an event) status/parked Deliberately paused with a recorded resume condition and removed status/parked Deliberately paused with a recorded resume condition queue/parked Queue task is parked (waiting on CI, time, or an event) status/in-progress Queue worker lifecycle state queue/claimed Queue worker lifecycle state labels Oct 29, 2026
@dezren39
dezren39 added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit 17c4a0d Oct 30, 2026
111 of 119 checks passed
@dezren39
dezren39 deleted the queue/2661-807-apply-wiring branch October 30, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

queue/parked Queue task is parked (waiting on CI, time, or an event) status/parked Deliberately paused with a recorded resume condition work/assigned Queue worker lifecycle state

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant