Skip to content

fix(studio-server): history sweeps stop holding the host's event loop on large projects - #7491

Merged
miguel-heygen merged 4 commits into
mainfrom
dperf/history-async-walk
Nov 10, 2026
Merged

miguel-heygen merged 4 commits into
mainfrom
dperf/history-async-walk

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

What

Two blocking passes per history sweep, both proportional to the project's file count, are gone:

  1. Project history lists the project's files with the async collector instead of readdirSync + lstatSync. start, firstOpen, takeInUnnamedLedger, sweep and the video state's file hashes now await the walk, so a host that serves requests on the same thread keeps answering while a large project is listed.
  2. .hyperframes/video-state.json is rewritten only when history changed. Before, every queued operation, including a sweep that found nothing, read and parsed the file, rebuilt one record per project file and wrote it back synchronously.

The sync listProjectFiles had no caller left (it was never exported from the package), so it is removed; its tests now cover listProjectFilesAsync.

Why

Every history sweep listed the whole project with blocking calls. On a drive where each stat is slow (large projects, scanned or network drives), the host's event loop stood still for the full walk, once per sweep.

Related work

Refs #7480: same file. Its guarantee (undo and restore see one consistent file list when a save lands mid-copy) holds, because history operations still run one at a time through queue(): a sweep's walk can now yield to file writes on disk (as its per-file reads already did), never to another undo, restore or window.

Overlaps #7484 in videoState.ts (one line here); whichever merges second rebases.

How

  • helpers/projectSignature.ts: listProjectFilesAsync maps collectProjectFilesAsync's entries to project-relative paths. The sync listProjectFiles is deleted.
  • history/projectHistory.ts: historyFiles awaits it; the four call sites await historyFiles.
  • history/videoState.ts: fileHashes awaits it.
  • history/projectHistory.ts: an owesState flag, set by persistLog and by any change to the tracked files (a small Map subclass), gates writeHistoryState in queue()'s finally; a failed write sets it again. Every log change already goes through persistLog.
  • The listing takes each relative path by slicing the project folder off, not path.relative per file.

Test plan

  • Unit tests added/updated
  • Manual testing performed
  • Documentation updated (if applicable)
  • Comments follow CONTRIBUTING.md "Comments"

New history/projectHistory.walk.test.ts:

  • fails without the fix: projectHistory.walk.test.ts "a sweep of a project on a slow disk lets the host keep answering while it lists the files" (400 files; each blocking lstatSync / readdirSync holds the thread 1 ms, the async calls are untouched; a setImmediate loop records the event loop's longest hold). With historyFiles back on the sync listing it fails with "held the loop 482.5 ms: expected 482.45 to be less than 200"; with the fix the hold is a few ms.
  • fails without the fix: projectHistory.walk.test.ts "a sweep that finds nothing leaves the video state file alone, and a change rewrites it" (the file is replaced atomically, so a rewrite shows as a new inode). With the write ungated it fails with "expected 284347 to be 284348".
  • Both 3 runs in a row green. src/history plus projectSignature and renderOutputs tests: 16 files, 286 passed, 2 skipped. tsc --noEmit clean; oxfmt --check and oxlint clean on the changed files.

Measured on a 20,000-file fixture project (200 folders x 100 files) on a local Linux disk, longest event-loop hold per sweep, 5 sweeps each:

longest hold per sweep (ms)
before (main) 564, 338, 354, 345, 352
async listing only 204, 169, 163, 182, 157
this PR (async listing + state written only on change) 42, 40, 39, 47, 43

A CPU profile of the "async listing only" sweeps put the remaining hold in updateState (videoState.ts): about 95 ms per sweep of synchronous read, rebuild and write of video-state.json, plus about 60 ms of path.relative / path.resolve mapping paths. In this PR's profile updateState no longer runs on an unchanged sweep; garbage collection is 37 ms across the 5 sweeps. The remaining ~40 ms hold is short synchronous work spread between the walk's awaits; the walk itself is the native lstat calls, off the event loop.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 267 (base branch 267), smooth 113 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (8)

@miguel-heygen miguel-heygen changed the title fix(studio-server): project history lists files without holding the host's event loop fix(studio-server): history sweeps stop holding the host's event loop on large projects Nov 10, 2026

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 1b05a119. I found no blockers.

  • Fix-PR rule: met. Two "fails without the fix:" lines name tests in projectHistory.walk.test.ts. The slow-disk test mocks only the sync lstatSync/readdirSync, so a blocking walk of its 420 entries holds the event loop for at least 420 ms (limit 200 ms); the async walk isn't slowed. The inode test checks that video-state.json is left alone on an unchanged sweep and rewritten after a change. Both target exactly what changed. CI on this head is green. I didn't rerun the before/after myself (no bun in my sandbox).
  • Async listing: all five former sync call sites (start, takeInUnnamedLedger, firstOpen, sweep, fileHashes) now await. They all still run inside queue(), so #7480's one-consistent-listing guarantee holds. The slice-based relative path is safe: the collector builds every path with resolve(normalizedProjectDir, …) and never a realpath, and the endsWith(sep) guard covers a root dir.
  • owesVideoState gate: writeHistoryState reads only log.entries, manifest() (from tracked) and manifestAt(log). Every log change at head goes through persistLog (commit, baseline, fold, pin). tracked is never reassigned or mutated in place (only set/delete, which TrackedFiles intercepts). The flag clears before the await and is set again on a failed write, so a change that lands during the write isn't lost.

Non-blocking: before this PR, every queued op rewrote video-state.json. If that file is deleted or emptied outside history, or if a render changes renders.retainedRevisions (which dropArtifactsOfGoneRevisions reads), the fix-up now waits for the next history change instead of the next sweep. That seems acceptable. Setting owesVideoState when readState finds the file missing would restore self-healing if that matters.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 10, 2026
@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

On the non-blocking note: keeping it as is. A deleted or emptied video-state.json is rewritten on the next history change, and readers already treat a missing file as empty (readState returns emptyState()). Self-healing on every sweep would cost a stat of the file per sweep for a case nothing in the repo causes; easy to add if one appears.

Merged via the queue into main with commit 61e720b Nov 10, 2026
82 checks passed
@miguel-heygen
miguel-heygen deleted the dperf/history-async-walk branch November 10, 2026 10:44
@miguel-heygen miguel-heygen mentioned this pull request Nov 10, 2026
3 of 4 tasks
SanHsien added a commit to SanHsien/hyperframes that referenced this pull request Nov 10, 2026
…nce watermark

Seventeen commits, PRs 7469-7493, issues 7473 and 7486 and twelve pin moves
are recorded with their evidence in the continued ledger, DECISIONS,
UPSTREAM and ROADMAP. Adopted: heygen-com#7466, heygen-com#7479, heygen-com#7482, heygen-com#7490 and the heygen-com#7441 and
heygen-com#7440 docs. Deferred: heygen-com#7469, heygen-com#7471, heygen-com#7476, heygen-com#7480, heygen-com#7468, heygen-com#7487, heygen-com#7491.
Not applicable: heygen-com#7470, heygen-com#7351, heygen-com#7475, heygen-com#7483. The watermark moves to
61e720b, PR heygen-com#7493, issue heygen-com#7486; thirteen new PRs are pinned. FORK.md
records the subComposition test differences. The core packed-size roadmap
row trigger is not met.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant