Repository navigation
id(world-replication): a journal whose Checkpoint line exceeds 256 KiB can never be replicated — split_entry_payload refuses the single entry and send_snapshot dies before the first frame #4630
Description
Summary
A world whose journal contains a single entry larger than MAX_REPLICATION_PAYLOAD_BYTES (256 KiB) can never be replicated over the new /id-world-replication/1 feed (PR #4589, phase 2 of #806). send_snapshot splits the journal image with split_entry_payload, which refuses any single entry over the cap — and the one journal entry that routinely grows past it is the compaction Checkpoint: it carries the program's own state snapshot (uncapped) plus all retained events on one line. Once such a checkpoint exists, every replication stream for that world dies before the first frame, deterministically, on every reconnect; live subscribers are dropped the same way when the post-compaction Reset snapshot is sent.
Evidence (origin/main 51535c2b4)
pkgs/id/src/world/replication.rs:79—MAX_REPLICATION_PAYLOAD_BYTES = 256 * 1024.pkgs/id/src/world/replication.rs:164-192(split_entry_payload): chunks are split only at entry (newline) boundaries; at:179-181a single entry withentry_end - start > MAXreturnsReplicationError::PayloadTooLarge.pkgs/id/src/world_replication_net.rs:197(send_snapshot):let chunks = split_entry_payload(&snapshot.bytes).context("split journal snapshot")?;— the error ends the stream before any frame is written. The same function serves theResetpath, so a live stream dies at the reset after compaction too.pkgs/id/src/world/mod.rs:874-908—JournalEntry::Checkpoint { .., snapshot: String, events: Vec<WorldEvent> }: the program's state serialization and the retained events ride in this one entry.pkgs/id/src/world/core.rs:731-743(checkpoint_entries): the checkpoint'seventsisself.events.iter().cloned().collect()— all retained events (WorldLimits::retained_eventsdefaults to 1024,world/mod.rs:103), with no byte budget applied to the entry (thesnapshot_bytesbudget atworld/mod.rs:80-83bounds participant snapshot frames, not this journal entry).pkgs/id/src/world/actor.rs:261-268(compact_journal):program.snapshot()is journaled with no size check. Compaction is routine:checkpoint_everydefaults to 1000 commits (world/mod.rs:106), andcompact_if_duepublishes the reset afterwards (world/actor.rs:1618,:1630).pkgs/id/src/world_store.rs:242-245—FileJournal::compactwrites each entry as exactly oneserde_jsonline, so the checkpoint is a single line in the journal file;replication_snapshot(:206-218) returns those exact bytes tosend_snapshot.
So both growth routes land on the same line: a program whose serialized state exceeds ~256 KiB, or ~1024 retained events whose serializations sum past it. Neither is exotic for a long-lived world, and compaction is the normal state of such a world — precisely the worlds read replicas (#806) exist for.
Reproduced this run (laptop probe)
Probe ~/git/fleet/probe-2026-11-02-laneB-replcheckpoint/ on the laptop: split_entry_payload + the cap constant extracted verbatim from origin/main 51535c2b4 (snippet md5 5dda768b; source replication.rs md5 99ceb4bf), fed journal byte images in the exact on-disk shape (one Created line + one Checkpoint line):
- PROBE1 PASS: checkpoint line with a 300,000-byte program snapshot →
Err(PayloadTooLarge)— the snapshot can never be sent. - PROBE2 PASS (control): 100,000-byte snapshot →
Ok(1 chunk), chunks reassemble to the exact journal bytes. - PROBE3 PASS (boundary): a single entry of exactly 262,144 bytes (incl. newline) →
Ok; 262,145 bytes →Err(PayloadTooLarge).
Why it matters
The failure is silent from the replica's side (the stream just ends, with the error only in the server's debug log), total (no snapshot, no tail, no reset — the world is unreplicable, not degraded), and permanent (the checkpoint stays in the journal; retrying reproduces the identical refusal). It strikes exactly the worlds that have lived long enough to compact with substantial state — the ones whose writer most needs a surviving replica per #806's acceptance ("Replicas serve spectators while the writer is offline").
Acceptance criteria
- A world whose compacted journal contains a
Checkpointentry larger than 256 KiB replicates end to end: a fresh subscriber receives the full snapshot and reaches the terminal cursor, and a live subscriber survives the post-compaction reset. - The chosen mechanism (e.g. sub-entry payload splitting with the chain/cursor contract updated, a larger single-entry allowance within the 1 MiB transport frame bound, or a bounded checkpoint) is applied consistently to the snapshot, tail, and tracker-accept paths, and the replication module docs state the resulting per-entry guarantee.
- Regression test: a journal image whose checkpoint line exceeds
MAX_REPLICATION_PAYLOAD_BYTES( program-snapshot route and/or retained-events route) round-trips through the feed split/send path and is accepted byReplicationTrackerwith a cursor that matches a recomputation from the journal bytes. - Existing replication tests (
cargo test -p id --lib replicationand the world_store replication feed tests) still pass.
Related: #806 (the replication epic this feed implements), #4592/#4593 (replication admission contract — receiver side; this issue is the sender side).
Activity
dezren39 commentedon Nov 2, 2026
🤖 Agent note · state: filed
dezren39 commentedon Nov 2, 2026
🤖 Agent note · state: picked-up
35 remaining items
dezren39 commentedon Nov 3, 2026
agent-note:v1
Transition: picked up
Worker: repo-worker-code (lane 7b47536b-c162-444c-be8a-3a9538cc4cb5, OpenChamber scheduled lane)
Task: task 6557 (queue repo-health-code-pilot)
Scope: design-only. Choose the oversized-entry replication mechanism for #4630 — fragmentation/continuation frames vs larger in-frame allowance vs bounded checkpoint — and land the design note by PR; implementation issues filed from it. The fix itself is out of scope; #4630 stays open until implementation merges.
Fresh state check 2026-11-03T12:00Z: issue OPEN, no
held/user/owned/do-not-mergelabels, no competing PR and noqueue/*4630*branch on origin.dezren39 commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 6557 (small_fix, queued, priority 30) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
dezren39 commentedon Nov 3, 2026
🤖 Agent note · handed off
claimed— the next claimant resumes from that checkpoint, it does not restart from zero.claimed.docs(id): design note for #4630 — bound the checkpoint at write time
Merge pull request #4962 from developing-today/queue/6557-4630-oversi…
dezren39 commentedon Nov 3, 2026
agent-note:v1
Transition: design complete (task 6557, queue repo-health-code-pilot)
Worker: repo-worker-code (lane 335a5d1f-bfc1-4831-99ea-850c8939a727, OpenChamber scheduled lane)
Design delivered — #4630 mechanism chosen
Design note merged in #4962 (merge
f3d150bb5):doc/2026-11-03T14-55-00Z_design_id4630_oversized_replication_entries.md.Chosen: bound the checkpoint at write time (
WorldCore::checkpoint_entries) by emitting bounded part entries,MAX_CHECKPOINT_PART_BYTES = 192 KiB— below the existing 256 KiBMAX_REPLICATION_PAYLOAD_BYTES. No wire change, no envelope-version bump, noReplicationTracker::acceptchange, no receiver upgrade;JOURNAL_VERSION6 for the new part kinds. The v1 invariant #4592 relies on (payloads are whole entries; the cursor is entries absorbed) is preserved exactly.Evaluated and recorded in the note:
acceptrefuses any other envelope version) plus a permanent per-fragment receiver-buffering surface to solve a writer-controlled entry kind.actor.rs:275), so the residual is large; and it is a receiver-side constant, so raising it silently compatibility-splits v1 receivers.Per-entry guarantee is stated in the note: every writer-emitted entry ≤ 192 KiB, hence ≤ one v1 payload frame. The one residual case (a
Checkpointalready stored by an older binary, not since re-compacted) has its own follow-up.Implementation follow-ups filed
JOURNAL_VERSION6, part bound, restore assembly), with the id(world-replication): a journal whose Checkpoint line exceeds 256 KiB can never be replicated — split_entry_payload refuses the single entry and send_snapshot dies before the first frame #4630 oversized-Checkpoint round-trip regression and the named suites (cargo test -p id --lib world::replication,world_storereplication feed tests, journal-version/restore tests).Checkpointgets force-recompaction or an actionable diagnostic.This issue stays OPEN — the implementation in #4966 closes it. Labels cleared to match the handoff (no in-flight
queue/*,status/*,work/*).