Skip to content

fix(producer): a laptop that sleeps mid-render no longer counts the sleep as a capture stall - #7096

Merged
miguel-heygen merged 2 commits into
mainfrom
fix/capture-watchdog-sleep-gap
Nov 3, 2026
Merged

miguel-heygen merged 2 commits into
mainfrom
fix/capture-watchdog-sleep-gap

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

What

A laptop that sleeps in the middle of a render no longer has the sleep counted as a capture hang. On wake the render carries on instead of ending with "capture stalled: no frame progress".

Why

The three no-progress watchdogs (the single-worker capture race, the parallel capture timer, and the render-stage rule) measure silence against the wall clock or against a timer that keeps counting through sleep. After a sleep longer than the 60 s budget, the first check on wake sees "no frame for hours" and fails the render, though nothing was hung.

Clock facts, which decide the shape of the fix:

  • Windows: QueryPerformanceCounter, which libuv's uv_hrtime uses, returns ticks since boot "including the time when the machine was in a sleep state such as standby, hibernate, or connected standby" (Microsoft Learn, Acquiring high-resolution time stamps). Node timers and performance.now() count the sleep.
  • macOS: libuv's uv__hrtime calls mach_continuous_time(), which counts sleep (libuv src/unix/darwin.c, v1.x branch). I confirmed this from the source, not on a sleeping Mac, and did not pin the libuv release that introduced it.
  • Linux: CLOCK_MONOTONIC excludes suspend (clock_gettime(2)), so there a monotonic clock would exclude the sleep.

Because two of the three platforms count sleep in every clock Node offers, swapping Date.now() for a monotonic clock is not the fix.

How

stallClock.ts owns one rule: a gap between two looks at the silence that is more than 15 s past when the look was due means the process was suspended, and the whole gap is paused time. All three watchdogs read their silence from it (raceAgainstStall, the parallel streaming timer and the pipelined drain's remaining budget, and watchStall in observability.ts). A process that keeps running, with looks on time and no frame, still trips at the same budget as before.

A render that is only slow, or whose event loop is blocked for under 15 s, is judged exactly as before.

Not changed

A stall still restarts the capture from frame 0 on retry. That is separate and not part of this change.

Test plan

  • Unit tests added/updated
  • Manual testing performed: not on a real sleep; the tests move the clock 10 hours between two looks while timers keep running, which is what a wake looks like to these watchdogs.
  • Documentation updated (not applicable)
  • Comments follow CONTRIBUTING.md "Comments"

Fails without the fix: at the merge-base the wake tests fail (observability, the sequential race, the parallel timer) and the on-time stall tests pass. Each test also fails when its own code is broken: no forgiveness in the clock, the clock always idle 0, the race never tripping, the parallel timer not forgiving, the parallel timer never tripping, the stage rule not forgiving, the stage rule never tripping.

Passes 3 runs in a row: stallClock.test.ts and observability.test.ts (24 tests) under vitest, captureStreamingStage.test.ts (33 tests) under bun. oxfmt --check, oxlint, tsc --noEmit and the comment checks pass.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at e325f3a2. I found no blockers.

Fails-without-the-fix check, run locally: I kept the new tests and put back the merge-base (fc006f71) versions of observability.ts and captureStreamingStage.ts. These tests then fail:

  • does not read a ten-hour sleep as no progress (vitest, 1 failed / 21 passed)
  • does not trip a sequential capture the moment the machine wakes
  • does not trip the parallel watchdog the moment the machine wakes (bun, 2 failed / 31 passed)

The on-time stall tests still pass at the merge-base. At this head, stallClock + observability pass 24/24 and captureStreamingStage passes 33/33.

Real hangs still trip:

  • stallClock.idle only skips the time when a check arrives more than 15 s after it was due.
  • When checks arrive on time, watchStall and the parallel setInterval count silence exactly as before.
  • raceAgainstStall now checks every ≤5 s against its own clock, so a capture call that never returns still fails at its deadline.
  • A 0 ms deadline still fails right away: the timer fires at 0 and idle(0) >= 0.

Non-blocking note, the drain loop's quiet.idle(0): there's no due time for this check, so any gap over 15 s since the last check or reset is skipped as if the machine had slept. That includes a capture call that was merely slow (say 50 s, under the 60 s budget) and wasn't followed by a frame write. The only such case is the first batch, where prevBatch is empty, so the next call gets the full budget instead of what was left of it. That allows silence of up to about 2× the budget once per render, not an unbounded hang, because every later iteration writes a frame and resets the clock. If you want it tight, a follow-up could pass the time the drain actually expected to wait, or skip only time spent outside a raceAgainstStall. It doesn't need to block this PR.

CI: Comments, PR captures, preview-regression and Player perf are green. CI, regression, Windows render verification and CodeQL were still running when I reviewed.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 267 (base branch 267), smooth 108 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (8)

Unstable (1)

  • seqrepeat-none-center-r0-root-z100: tracking 0.05, pressJump 0, drop 0, reload 0, render 0.02, renderKey -, undo true, teleport true / tracking 0.05, pressJump 0, drop 0, reload 0, render 0.02, renderKey -, undo true, teleport true / tracking 0.05, pressJump 0, drop 0, reload 0, render 0.02, renderKey -, undo true, teleport true

@miguel-heygen
miguel-heygen force-pushed the fix/capture-watchdog-sleep-gap branch from e325f3a to c66b013 Compare November 3, 2026 22:56

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at c66b013a. This is my approval of e325f3a2 carried over to the rebase onto #7104. I confirmed the rebase changed nothing in this PR's own changes, and that the two features fit together.

  • This PR's own changes are identical: I compared the +/- lines of the old PR (fc006f71...e325f3a2) with this head's PR diff, and they match line for line. Only the code around them changed, because #7104 landed underneath.
  • How it fits with #7104:
    • The parallel watchdog's quiet clock is created inside the resume loop, so each resumed attempt gets a fresh clock and resumeFrom reporting is unchanged.
    • The sequential streaming loop (lastProgressAt = Date.now(), around line 1090) still uses the wall clock. That code was already there before this PR and is safe: it resets right after each frame is written, and a sleep while a capture is in flight is handled by raceAgainstStall's own sleep-aware clock. So the only exposure is the sub-millisecond gap between the reset and the next race.
  • Tests at this head: stallClock + observability pass 24/24, captureStreamingStage (including #7104's resume tests) passes 43/43 under bun, and tsc --noEmit is clean.
  • Fails without the fix, against the new merge-base 3dc82f18: I kept the new tests and put back main's observability.ts and captureStreamingStage.ts. These tests then fail:
    • does not read a ten-hour sleep as no progress (1 failed / 21 passed)
    • does not trip a sequential capture the moment the machine wakes
    • does not trip the parallel watchdog the moment the machine wakes (2 failed / 41 passed)

My earlier non-blocking note about the drain loop's quiet.idle(0) still applies as written. No blockers.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit e2edc93 Nov 3, 2026
91 checks passed
@miguel-heygen
miguel-heygen deleted the fix/capture-watchdog-sleep-gap branch November 3, 2026 23:28
This was referenced Nov 4, 2026
SanHsien added a commit to SanHsien/hyperframes that referenced this pull request Nov 4, 2026
Adopted with tests that fail on the old code (except the link change):
- offload: rewrite media inside <template>, keep same-named files from
  other folders (heygen-com#6842, offload part only)
- autoUpdate: no background upgrade across a 0.x minor; Homebrew installs
  only notify (heygen-com#7013, autoUpdate part only)
- engine: script load failures carry origin/code; aborted scripts are not
  load failures (heygen-com#7088, heygen-com#7093)
- community Discord invite links (heygen-com#7097)
- studio: write media duration to the store once per file (heygen-com#6745)
- check: restart the idle watchdog on each driver call during analysis
  (heygen-com#6919, adapted to this line's fixed 120s idle window)
- check: default wait raised from 10s to 45s, matching snapshot/render
  (heygen-com#6947)
- image cancellations no longer ignored by extension (heygen-com#6928)
- capture: system sleep no longer counts as a stall (heygen-com#7096, hand-ported;
  the sequential branch also uses the shared stall clock)

Commit watermark advances to 183614b (28 consecutive commits with
file:line evidence); PR and issue watermarks are unchanged. The rest of
the review is in docs/UPSTREAM_REVIEW_2026-11-04_CONTINUED.md.

Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
Co-authored-by: WaterrrForever <miao.yang@heygen.com>
Co-authored-by: Wenbo Zhu <wenbo_zhu@berkeley.edu>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant