Repository navigation
id: investigate: ephemeral live data vs the chained event graph — retention and intermittent signing #2034
Description
LIST 9a. ID may need to distinguish a LIVE data view from a chained graph of every event that ever occurred. Does every keystroke (typing, backspace, arrows, B, A) need a signature and timestamp retained forever? Probably not: candidates are very low retention for keystroke-class streams, or intermittent signing — chained off an initial signature and a final signature. Investigate: stream classes and their retention/signing needs, what an intermittent-signature chain proves and what it gives up, and how live views are reconstructed. Deliverable: data-class table + signing-chain proposal.
Program: Drew's 2026-10-28 directive (full text: hatch-personal memory/topics/id-program-2026-10-28.md).
Working pattern: parallel investigations, then the named synthesis issue merges them into a full design. Route guidance: Workhorse by default; Specialist/Summit for synthesis and ramification-heavy work. A result that says "done" is not the endpoint — the orchestrator reviews it and may commission another round. Deliverables: a markdown design/investigation doc (mermaid/SVG diagrams welcome) linked from this issue, plus follow-up issues for anything buildable. Build a proof of concept where the design calls for one, but do NOT build the permanent feature on top of an unreviewed PoC.
Activity
developing-today commentedon Oct 29, 2026
CODE-955
dezren39 commentedon Oct 29, 2026
Parent epic: #2056 — ID platform expansion (2026-10-28 directive).
dezren39 commentedon Oct 29, 2026
Top priority — ID program wave-1. Drew directed this be picked up in the next runs (2026-10-29): enqueued directly as triage task 3348 in repo-health-code-pilot at queue priority 20 (only breakage-class work at priority 10 outranks it). Wave-1 is the ungated investigations and designs; syntheses and PoCs stay gated on their named inputs. Investigation route: Workhorse.
dezren39 commentedon Oct 29, 2026
🤖 Agent note · picked up
dezren39 commentedon Oct 29, 2026
🤖 Agent note · investigation complete — two-tier model, doc in review, follow-on routed
Deliverable
doc/2026-10-29T10-45-00Z_investigation_ephemeral_live_data_retention/2026-10-29T10-45-00Z_investigation_ephemeral_live_data_retention.md— PR #2239 (docs-only, basemain). Every claim is labelled grounded (withfile:line) or proposed.Premise: partly false, in a useful direction
"Does every keystroke need a signature and permanent timestamp?" — the codebase does not do that today.
web/collab.rscarries steps and cursors as unsigned MessagePack frames over a WebSocket; only the saved revision becomes a signeddocument.putevent (local_events.rs:1-18). The world journal is already bounded (WorldLimits::retained_events = 1024,world/mod.rs:59-99), cursors drop 5 min after disconnect, doc state 1 h after the last client leaves. What is actually missing is the signed checkpoint that makes a low-retention stream auditable after the fact — and any retention path at all inevent_store(no TTL/compaction/GC exists;event_store/mod.rs:4-23keeps every verified event forever).Verdict
Two-tier model. Keep the per-author signed hash chain (
ChainExt, one Ed25519 signature per event,chain.rs:282-309) for durable facts; move keystroke/cursor/input traffic onto an ephemeral per-session stream authenticated once by a session-header signature plus a per-session MAC key, closed by intermittent segment checkpoints (every N events or T seconds, plus on commit/blur) — never a signature per event."Initial + final signature" as written in the issue is too weak: a bare final signature proves ordering inside the segment but not timing, and lets the relay truncate the tail before the final signature arrives. Segment checkpoints close the truncation gap at a cost proportional to segments, not events.
What a checkpoint proves: the signer attests "this segment's ordered delta from state Sₙ₋₁ produced Sₙ; here is its root." What it gives up: per-event third-party non-repudiation, provable exact timing per event, and any guarantee that an unseen intermediate checkpoint exists. Biggest risk: relay equivocation / selective withholding — an intermittent chain cannot prove two recipients saw the same live stream between checkpoints, so money, match results and disputable chat keep per-event signing or add peer countersignatures (#2037/#2038 consume this).
Live views are rebuilt from the latest signed checkpoint plus the ephemeral stream since it; late joiner loads + rebases, reconnect reconciles at the next boundary, partition heals at a checkpoint.
The doc also carries the requested data-class table (9 classes: identity/key material, durable documents, currency, match records/audits, chat, presence/typing/cursor/keystroke, media blobs, relay/sync metadata) with retention, signing granularity, storage and loss consequence per class, plus a retention policy proposal and mermaid diagrams.
Follow-on routed this pass
id: design: session header + segment checkpoint event kind— the design input; decides graph-event vs parallel artifact, MAC key scheme, segment defaults, fork-proof/waiting-pool interaction.id: PoC: intermittent-signature live stream for collaborative editing— gated PoC (blocked by id: design: session header + segment checkpoint event kind #2240), explicitly not to be merged into the permanent feature before a review round.event_store; mutual two-sided checkpoints investigation for match results/chat disputes.Open questions left for the synthesis issues (#2036, #2037, #2038)
Checkpoint as graph event vs parallel signed artifact · per-class segment defaults and world-configurability · MAC key in header vs derived, rotation mid-session · which chat channels are durable vs live and who changes that · interaction with fork proofs and the waiting pool · whether peer countersignature is mandatory for any class by default (and its relay/online implications) · role of a witness/notary server for high-value checkpoints vs "servers accept all signed objects" (#2030).
Review before building on: the checkpoint wire format and MAC scheme (proposed, security-sensitive), the retention numbers (proposed, policy), and the graph-event-vs-artifact decision.