Skip to content

fix(core): a stuck seek in check or snapshot fails named before the protocol timeout - #6596

Merged
miguel-heygen merged 3 commits into
mainfrom
fix/seek-barrier-bounded
Oct 31, 2026
Merged

miguel-heygen merged 3 commits into
mainfrom
fix/seek-barrier-bounded

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

A seek waits for everything it started (video frames, frame sources, scene windows, motion blur, effects, image decodes) before a frame is captured. If one of those holds never settled, check and snapshot sat until Puppeteer's protocol timeout (180 s) fired, with a message that said nothing about why.

Now the caller can bound the wait, and the error names what is still pending:

seek did not settle within 170000 ms: still waiting on frame-source
  • __hfWaitForSeekCompletion({ timeoutMs }) ends at the caller's limit. Without timeoutMs it waits as before, so export renders keep the limits they already had: protocolTimeout, scaled by canvas size and settable by the user.
  • check (the overlap grid) and snapshot / settleCompositionSeek pass SEEK_SETTLES_BEFORE_PROTOCOL_TIMEOUT_MS, which is 170 s, 10 s under the protocol timeout their browser runs with. A seek that used to finish still finishes; one that used to die at 180 s with an opaque timeout now fails at 170 s, named. The grid runs all its seeks in one page call, so it spends one 170 s budget across all of them.
  • Every hold registers with its kind (registerSeekCompletion(promise, hold)), so the error can say which one is stuck.
  • Holds registered while the barrier waits share the same deadline; a stream of new holds can't extend it.

What it does not fix

  • It does not make a slow seek faster.
  • It does not fix the many-clips snapshot hang. That is a renderer crash, handled separately.

Tests

seek-dispatch.test.ts:

  • A hold that never settles ends at the caller's deadline and is named.
  • Holds registered while waiting end at the same deadline, in under 1 s.
  • With no timeoutMs, a seek waits as long as its holds take (fake timers, 30 min).
  • Holds that settle in time resolve as before.

fails without the fix: packages/core/src/runtime/adapters/seek-dispatch.test.ts "a seek with a hold that never settles ends at its deadline, naming what it still waits on" and "holds registered while the barrier waits still end at the same deadline": on main the wait has no deadline, so both run into vitest's 5 s timeout; on this branch they pass, 3 runs in a row. The CLI capture tests now expect the limit to be passed.

Results on Linux:

  • Core seek tests: 3 runs in a row, 14/14 each time.
  • Full CLI unit project: 418 files, 5810 tests.
  • Fallow audit: clean.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES · 1 blocker · 0 important · 0 nits · head c74bb142

The 30 s default also applies to export renders, not just check/snapshot. That turns renders that are slow but valid into failures. Details inline.

Scope and checks
  • Read all 7 files. Also checked every caller of __hfWaitForSeekCompletion: engine frameCapture/drawElementService/riskFrames/videoFrameInjector, cli captureCompositionFrame/checkBrowser/motionShot, and studio useShadowPreviewReload. None of them pass timeoutMs.
  • The producer's virtual-time shim freezes Date.now while the barrier waits, so deadline - Date.now() still works out to 30 s of real time. That part is fine.
  • Ran locally on the head: seek-dispatch, init.seekOrder and frameSources tests (35 pass); core tsc is clean. CI was still running when I posted.

(completion) => options?.videoFrames !== false || !_videoFrameCompletions.has(completion),
([, hold]) => options?.videoFrames !== false || hold !== "video-frame",
);
const timeoutMs = options?.timeoutMs ?? SEEK_SETTLE_CAP_MS;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: every engine/producer render goes through this default. waitForPendingSeekCompletion (engine frameCapture.ts:3190) calls __hfWaitForSeekCompletion() with no options. The 180 s in the PR body is the CLI's Puppeteer default (captureTimeout.ts). Renders run under protocolTimeout, which is 300 s by default, scales with canvas area up to 30 min (scaleProtocolTimeoutForComposition exists because slow-but-valid seek/paint calls on large canvases were being killed), and can be raised by the user.

So with this change, a render that used to finish now fails after 30 s, with no way to raise the limit. Example: a frame source whose ready loads a heavy model or scene for 45 s on a cold worker. docs/guides/frame-sources.md says "ready holds initialization and export capture" and documents no limit. The same goes for a slow software-GL vfx/motion-blur paint at 4K.

Fix: have the engine pass timeoutMs derived from its effective protocol timeout, minus a margin so the named error still wins the race. Keep 30 s, or the CLI's budget, for check/snapshot. If 30 s is meant to be the new contract for renders too, document it in the frame-sources guide and make it configurable.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in ce1d96b. The core wait no longer has a default limit: without timeoutMs it waits as before, so renders keep their scaled protocolTimeout. Only the CLI's check grid and settleCompositionSeek (snapshot) pass a limit, 170 s, 10 s under the 180 s protocol timeout their browser runs with. Anything that finished before still finishes; a seek that hit the opaque 180 s timeout now fails at 170 s and names the stuck hold.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed in ce1d96b. The core wait has no default limit anymore, so renders keep their scaled protocolTimeout. Only check and snapshot pass 170 s.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 143 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (2)

  • seqplainkeys-none-pct-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true
  • seqrotatenudge-none-pct-r0-nested-z100: tracking 0.02, pressJump 0, drop 0, reload 3.03, render 0.06, renderKey -, undo false, teleport true / tracking 0.02, pressJump 0, drop 0, reload 0, render 0.06, renderKey -, undo true, teleport true / tracking 0.02, pressJump 0, drop 0, reload 0, render 0.06, renderKey -, undo true, teleport true

@miguel-heygen miguel-heygen changed the title fix(core): a seek that never settles ends after 30 s and names what it waits on fix(core): a stuck seek in check or snapshot fails named before the protocol timeout Oct 27, 2026

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 1 resolved, 0 open, 0 new · c74bb14 → ce1d96b. CI is green; the first CI run lost edit-accuracy shard 4 to a stalled Chrome download, and the second run on this head passed.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 31, 2026
Merged via the queue into main with commit 1e5413f Oct 31, 2026
183 of 185 checks passed
@miguel-heygen
miguel-heygen deleted the fix/seek-barrier-bounded branch October 31, 2026 17:04
SanHsien added a commit to SanHsien/hyperframes that referenced this pull request Nov 9, 2026
The 1a9bb89 port of heygen-com#7402 and heygen-com#7378 brought in releaseMedia(el, flag)
with window.__HF_RELEASE_PASSED_MEDIA (heygen-com#6929, needs the heygen-com#6785 buffered
guard) and registerSeekCompletion(landed, "image-decode") (the SeekHold
union from heygen-com#6596). Both upstream commits stay deferred, so typecheck:runtime
failed in the Windows render verification workflow. The fork's
releaseMedia already frees every clip, so the call keeps its behaviour.

Add tsconfig.runtime.json to the pre-commit typecheck, which only covered
tsconfig.json (it excludes src/runtime), and pin it in the hook test.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant