Repository navigation
fix(studio-server): history sweeps stop holding the host's event loop on large projects - #7491
Conversation
There was a problem hiding this comment.
Approving at 1b05a119. I found no blockers.
- Fix-PR rule: met. Two "fails without the fix:" lines name tests in
projectHistory.walk.test.ts. The slow-disk test mocks only the synclstatSync/readdirSync, so a blocking walk of its 420 entries holds the event loop for at least 420 ms (limit 200 ms); the async walk isn't slowed. The inode test checks thatvideo-state.jsonis left alone on an unchanged sweep and rewritten after a change. Both target exactly what changed. CI on this head is green. I didn't rerun the before/after myself (no bun in my sandbox). - Async listing: all five former sync call sites (
start,takeInUnnamedLedger,firstOpen,sweep,fileHashes) now await. They all still run insidequeue(), so #7480's one-consistent-listing guarantee holds. The slice-based relative path is safe: the collector builds every path withresolve(normalizedProjectDir, …)and never a realpath, and theendsWith(sep)guard covers a root dir. owesVideoStategate:writeHistoryStatereads onlylog.entries,manifest()(fromtracked) andmanifestAt(log). Every log change at head goes throughpersistLog(commit, baseline, fold, pin).trackedis never reassigned or mutated in place (onlyset/delete, whichTrackedFilesintercepts). The flag clears before the await and is set again on a failed write, so a change that lands during the write isn't lost.
Non-blocking: before this PR, every queued op rewrote video-state.json. If that file is deleted or emptied outside history, or if a render changes renders.retainedRevisions (which dropArtifactsOfGoneRevisions reads), the fix-up now waits for the next history change instead of the next sweep. That seems acceptable. Setting owesVideoState when readState finds the file missing would restore self-healing if that matters.
|
On the non-blocking note: keeping it as is. A deleted or emptied |
…nce watermark Seventeen commits, PRs 7469-7493, issues 7473 and 7486 and twelve pin moves are recorded with their evidence in the continued ledger, DECISIONS, UPSTREAM and ROADMAP. Adopted: heygen-com#7466, heygen-com#7479, heygen-com#7482, heygen-com#7490 and the heygen-com#7441 and heygen-com#7440 docs. Deferred: heygen-com#7469, heygen-com#7471, heygen-com#7476, heygen-com#7480, heygen-com#7468, heygen-com#7487, heygen-com#7491. Not applicable: heygen-com#7470, heygen-com#7351, heygen-com#7475, heygen-com#7483. The watermark moves to 61e720b, PR heygen-com#7493, issue heygen-com#7486; thirteen new PRs are pinned. FORK.md records the subComposition test differences. The core packed-size roadmap row trigger is not met. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
What
Two blocking passes per history sweep, both proportional to the project's file count, are gone:
readdirSync+lstatSync.start,firstOpen,takeInUnnamedLedger,sweepand the video state's file hashes now await the walk, so a host that serves requests on the same thread keeps answering while a large project is listed..hyperframes/video-state.jsonis rewritten only when history changed. Before, every queued operation, including a sweep that found nothing, read and parsed the file, rebuilt one record per project file and wrote it back synchronously.The sync
listProjectFileshad no caller left (it was never exported from the package), so it is removed; its tests now coverlistProjectFilesAsync.Why
Every history sweep listed the whole project with blocking calls. On a drive where each stat is slow (large projects, scanned or network drives), the host's event loop stood still for the full walk, once per sweep.
Related work
Refs #7480: same file. Its guarantee (undo and restore see one consistent file list when a save lands mid-copy) holds, because history operations still run one at a time through
queue(): a sweep's walk can now yield to file writes on disk (as its per-file reads already did), never to another undo, restore or window.Overlaps #7484 in
videoState.ts(one line here); whichever merges second rebases.How
helpers/projectSignature.ts:listProjectFilesAsyncmapscollectProjectFilesAsync's entries to project-relative paths. The synclistProjectFilesis deleted.history/projectHistory.ts:historyFilesawaits it; the four call sites awaithistoryFiles.history/videoState.ts:fileHashesawaits it.history/projectHistory.ts: anowesStateflag, set bypersistLogand by any change to the tracked files (a smallMapsubclass), gateswriteHistoryStateinqueue()'sfinally; a failed write sets it again. Every log change already goes throughpersistLog.path.relativeper file.Test plan
New
history/projectHistory.walk.test.ts:projectHistory.walk.test.ts"a sweep of a project on a slow disk lets the host keep answering while it lists the files" (400 files; each blockinglstatSync/readdirSyncholds the thread 1 ms, the async calls are untouched; asetImmediateloop records the event loop's longest hold). WithhistoryFilesback on the sync listing it fails with "held the loop 482.5 ms: expected 482.45 to be less than 200"; with the fix the hold is a few ms.projectHistory.walk.test.ts"a sweep that finds nothing leaves the video state file alone, and a change rewrites it" (the file is replaced atomically, so a rewrite shows as a new inode). With the write ungated it fails with "expected 284347 to be 284348".src/historyplusprojectSignatureandrenderOutputstests: 16 files, 286 passed, 2 skipped.tsc --noEmitclean;oxfmt --checkandoxlintclean on the changed files.Measured on a 20,000-file fixture project (200 folders x 100 files) on a local Linux disk, longest event-loop hold per sweep, 5 sweeps each:
A CPU profile of the "async listing only" sweeps put the remaining hold in
updateState(videoState.ts): about 95 ms per sweep of synchronous read, rebuild and write ofvideo-state.json, plus about 60 ms ofpath.relative/path.resolvemapping paths. In this PR's profileupdateStateno longer runs on an unchanged sweep; garbage collection is 37 ms across the 5 sweeps. The remaining ~40 ms hold is short synchronous work spread between the walk's awaits; the walk itself is the nativelstatcalls, off the event loop.