feat(files): stream copilot edits into the collaborative doc smoothly - #6122
Conversation
- apply the agent stream client-side into the live Yjs binding as minimal updateYFragment diffs (like main's setContent, but incremental) so it renders smoothly AND broadcasts to every peer via CRDT — a collaborator on /files sees the stream for free - gate the apply on collabReady so diffs never land on an unseeded doc; keep the read-only placeholder visible until the seed swaps in - run streamed ops under a dedicated tx origin so they stay out of the user's undo stack - delete the throttled server-side streaming merge and the baseVersion ordering machinery it needed (relay + notify + session contract); the durable final write still reconciles open editors and seeds late joiners
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
PR SummaryHigh Risk Overview Realtime relay: New Removed: Server-side mid-stream Client: Reviewed by Cursor Bugbot for commit ca81494. Configure here. |
Greptile SummaryThis PR moves copilot file-preview streaming into the collaborative Yjs document while retaining a durable final reconciliation.
Confidence Score: 4/5The PR should not merge until the remaining simultaneous-start path can no longer place duplicated copilot content into the shared collaborative document. The reply states that the startup race is bounded, and the revised relay correctly prevents its output from being persisted, but simultaneous viewers can still elect themselves from stale local awareness and apply duplicate frames that are immediately visible to collaborators. Files Needing Attention: apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/rich-markdown-editor.tsx; apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/collaboration/agent-stream-leader.ts
|
| Filename | Overview |
|---|---|
| apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/rich-markdown-editor.tsx | Integrates collaborative stream ticks, leader transitions, settle handling, and shadow-session cleanup; the acknowledged simultaneous-start dual-writer window remains. |
| apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/collaboration/apply-streamed-markdown.ts | Introduces a private Yjs shadow that captures agent-only deltas and relays them into the live collaborative document. |
| apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/collaboration/agent-stream-leader.ts | Elects the lowest announced awareness client as stream writer, with eventual-consistency behavior during simultaneous startup. |
| apps/realtime/src/handlers/file-doc.ts | Adds non-persisting agent sync handling and supports multiple awareness client IDs per shared socket. |
| apps/realtime/src/handlers/file-doc-store.ts | Tags agent stream entries across Redis replication and compaction so transient frames do not mark documents durable. |
| apps/sim/lib/copilot/tools/server/files/file-preview.ts | Makes append previews fail closed when existing file content is unavailable, preventing fragment-only snapshots. |
Sequence Diagram
sequenceDiagram
participant Agent as Copilot stream
participant Leader as Elected editor
participant Shadow as Private Yjs shadow
participant Shared as Shared Yjs document
participant Relay as Realtime relay
participant Peer as Collaborator
Agent->>Leader: Growing markdown snapshot
Leader->>Shadow: Reconcile snapshot
Shadow-->>Leader: Minimal agent delta
Leader->>Shared: Apply agent-origin update
Shared->>Relay: SYNC_NO_PERSIST
Relay->>Peer: Broadcast Yjs update
Agent->>Relay: Durable final write
Relay->>Shared: Reconcile final content
Reviews (14): Last reviewed commit: "fix(files): own presence per client id, ..." | Re-trigger Greptile
…snapshots Review round 1 (Greptile P1s): - apply the stream against a private shadow replica (seeded from the live doc at stream start) and relay only the agent's own delta into the shared doc, so a concurrent peer edit to a region the agent snapshot didn't include is no longer reverted (previously the whole-body reconcile deleted it) - gate append snapshots on "must extend the base": a base-less append fragment (emitted before the base loads) can no longer reconcile the seeded doc to a wipe; patch still legitimately replaces a mid-region - gate the apply on collabReady so diffs never land on an unseeded doc; keep the placeholder visible until the seed swaps in - plumb streamOperation through the preview surfaces to drive the append gate - add a peer-edit-preservation test (fails under whole-body reconcile) and refresh the undo-isolation + broadcast tests for the session API
|
@cursor review |
Cursor round 1 (Low): endAgentStream ran inside runOffRender, whose microtask is dropped when a rapid follow-up stream bumps the run token — leaking the shadow Y.Doc. Split it out into an unguarded microtask queued after the (droppable) final apply, so the shadow is always destroyed.
Cursor round 1 (High): client-applied stream frames broadcast over the sync channel, so the relay stamped a socket origin and ran schedulePersist — durably writing partial agent content mid-stream, attributed to the watching user (the old server-merge applied with no origin and never did). Restore that behavior: - new FILE_DOC_MESSAGE_TYPE.SYNC_NO_PERSIST wire tag; the provider tags AGENT_STREAM_ORIGIN updates with it (normal user edits stay SYNC) - the relay applies it under an AgentSyncOrigin (carries the socket id for broadcast exclusion, but is not a plain string) so originSocketId() is null → no edited/schedulePersist/lastEditorUserId; excludeSocketId() still excludes the sender, and the update still publishes to the stream so peers converge - the copilot's final edit_content write remains the authoritative durable persist - tests: relay applies+fans-out but never persists a SYNC_NO_PERSIST frame (verified it fails if applied as a socket edit); provider tags agent edits
|
@cursor review |
Cursor round 2: - High (settle skips apply without session): the stream shadow is now opened on the first ready frame, BEFORE the extend gate — so an `update` rewrite (whose every frame is gated out until settle) and a stream that finishes before seed still get a session, and settle applies the final body via the reused-or-on-demand shadow instead of leaving the doc stale until the durable reconcile. - Medium (peer edits stall the stream): the extend gate now reads a private `lastStreamedBodyRef` (the agent's own last frame), snapshotted at stream start, not `lastSyncedBodyRef` which `onUpdate` clobbers on peer edits — so a collaborator typing can't make the growing snapshot stop prefixing the shown body and freeze it. - Medium (multi-replica over-persist): pre-existing, documented "safe over-persist" (a peer task tails the frame as REDIS_ORIGIN and marks edited) — refreshed the stale comment to describe the SYNC_NO_PERSIST source; copilot's edit_content write remains the authoritative durable persist.
|
@cursor review |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Cursor/Greptile round 3 (High + Medium) — remove the fragile string-prefix "extend gate", which was the root of both findings: - Server: `buildFilePreviewText` now fails closed for an `append` whose base content hasn't loaded (returns undefined, like patch/update), so a base-less fragment never reaches the client. This eliminates the base-less wipe at settle (Greptile P1) at the source; an empty file (existingContent === '') still previews normally. - Client: the collab streaming tick no longer string-prefixes the raw preview against the editor's canonical markdown (the '*' vs '-' / emphasis mismatch that froze every append frame — Cursor). The mid-stream hold is now purely operation-based: `update` waits for settle; append/patch/create apply each frame via the (peer-safe) shadow reconcile. lastStreamedBodyRef is now a plain dedup guard, not a prefix baseline. Keeps the shadow, durable write, and SYNC_NO_PERSIST unchanged.
|
@cursor review |
Cursor round 4 (High): with the stream applied client-side, two tabs/windows on the same chat could each derive streamingContent (the reconnect/resume path re-consumes preview events) and each independently insert the stream under a different Yjs clientID, duplicating content until the durable reconcile. Fix — single-writer election via the file-doc awareness (new agent-stream-leader): - a client applying an agent stream announces `agentApplying` on its own awareness - only the leader (min clientID among announcers) applies mid-stream AND at settle; a non-leader renders the leader's ops via Yjs and does not apply (a non-leader applying the final body would re-insert the whole doc as a duplicate) - re-checked each frame, so it converges to one writer the moment awareness propagates; the sub-frame startup race is reconciled by the durable write - single-client (the common case) is unaffected: it is the only announcer, so it always leads
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit a49c118. Configure here.
…lings) Cursor round 7 (Medium): SYNC_NO_PERSIST frames applied under an origin carrying the sender socket id, and excludeSocketId dropped that whole socket from the relay fan-out. A second FileDocProvider on the same socket (chat preview + Files editor) then missed all mid-stream ops and stayed stale until the durable reconcile — a regression from the old no-origin server merge, which reached both. Fix: the agent origin is now a plain AGENT_SYNC_ORIGIN symbol, and agent frames broadcast to the WHOLE room (no socket excluded), matching the old behavior — so a same-socket sibling provider stays live; the emitting provider no-ops on its own echo (the ops are already applied locally). originSocketId still returns null for the symbol, so it keeps skipping edited/schedulePersist. Removed excludeSocketId and the socket-carrying origin object. Updated the relay test to assert the whole-room broadcast (verified it fails if the sender is excluded).
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit f3f7b7a. Configure here.
A peer task tailing an agent-streamed preview frame previously applied it as REDIS_ORIGIN, marking the seeded room edited and making a transient startup-race duplicate eligible for that task's last-disconnect flush. Mark agent frames with a stream field so peers apply them as REDIS_AGENT_ORIGIN, excluded from the edited/persist gate. The copilot's durable edit_content write stays the sole authority over file bytes.
|
Fixed the multi-replica persistence gap in 888496f. Agent-streamed frames are now tagged no-persist across replicas, not just on the originating task:
So a peer task now tails an agent frame as `REDIS_AGENT_ORIGIN` (previously `REDIS_ORIGIN`) and never marks the seeded room `edited` — a transient startup-race duplicate between two stream leaders can no longer become eligible for a peer's last-disconnect flush. The copilot's durable `edit_content` write stays the sole authority over file bytes. Covered by a falsification-verified store test (`file-doc-store.test.ts`): a normal publish tails as `REDIS_ORIGIN`, an agent publish tails as `REDIS_AGENT_ORIGIN`, and only the agent entry carries the marker. |
|
@cursor review |
Two multi-writer edge cases surfaced in review: - rich-markdown-editor: a client that led, lost leadership, then regained it reused its stale shadow (which never saw the interim leader's ops), re-emitting ops for content already present. Tear the shadow down when a client observes it is not the leader, so a regain rebuilds fresh from the current doc. - file-doc-store: compaction always stamped its snapshot REDIS_SNAPSHOT_ORIGIN (marks peers edited). A long agent-only stream crossing the threshold could fold preview content into a persist-eligible snapshot. Track whether a room integrated any real edit and stamp an agent-only snapshot REDIS_AGENT_ORIGIN so it stays no-persist. Both covered by falsification-verified tests.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 96fd287. Configure here.
Independent audit surfaced two real gaps: - file-doc-store: realEdited was latched AFTER appendUpdate's awaits, but the edit already sits in room.doc synchronously. A concurrent agent-frame compaction could read realEdited=false, snapshot that real content, and stamp it a no-persist agent frame — a lost edit. Latch it synchronously (same tick as the doc mutation) before any await. Deterministic falsifiable test added. - rich-markdown-editor: at settle every tab applied the final body, and a non-leader's local microtask runs before the leader's final propagates, so both insert the tail (Yjs keeps both) -> duplicated tail. Elect a single settle writer (reliable — awareness is long converged by settle), reading leadership before clearing the announcement. Corrects the overclaiming idempotency comment and the handoff pick-up comment. Adds a y-tiptap internals upgrade-guardrail test.
|
@cursor review |
The shared workspace socket hosts one collaborative provider per mounted view, so the chat file preview and the standalone Files editor for the same file each bind their own Yjs client id over ONE socket. The relay owned a single client id per socket, so the later JOIN overwrote the earlier and dropped its awareness — which silently broke the single-writer agent-stream election (a peer stopped seeing the streaming provider's announcement and could self-elect, duplicating streamed text for the whole stream). Track ownership per (socket, client id): a socket owns a set of client ids; the awareness gate accepts a frame only if every id it carries is owned; cleanup drops all of a socket's ids; the roster stays one-entry-per-session. Reclaim and the same-user reconnect path evict just the reclaimed id, dropping the old socket only if it empties. Falsification-verified test added.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit ca81494. Configure here.
1 participant
Summary
main, without breaking collaboration.updateYFragmentdiffs (the same primitive TipTap runs per keystroke) instead of a full-docsetContent. Each diff renders smoothly locally and broadcasts to every peer via CRDT — so a collaborator watching/filessees the smooth stream for free.collabReady(synced + seeded), so diffs never land on an unseeded doc; the read-only placeholder stays visible until the seed swaps in.baseVersionversion-ordering machinery it needed (relayfile-doc.ts/http.ts,notify.ts, session contract). The durable final write still reconciles open editors and seeds late joiners.Why
On
mainfile views are non-collaborative and stream via localsetContent; on this branch the editor is collaborative (Yjs), and that smooth path was disabled becausesetContentwould replace the shared doc and wipe peers. Content arrived only via throttled full-snapshot server merges — visibly chunky. Applying the stream as incremental CRDT diffs is smooth and collaborative.Type of Change
Testing
apply-streamed-markdown.test.ts(headless collaborative editor): incremental apply + broadcast, undo isolation (verified it fails when the tracked origin is used), concurrent-peer no-clobber merge.file-preview-adapter.test.ts/notify.test.ts/ relayfile-doc*.test.tsfor the surviving contract.bunx tscclean (apps/sim + apps/realtime); affected vitest suites,check:api-validation, lint, monorepo-boundary and realtime-prune checks all pass.Checklist