Repository navigation
fix(producer): a laptop that sleeps mid-render no longer counts the sleep as a capture stall - #7096
Conversation
There was a problem hiding this comment.
Approving at e325f3a2. I found no blockers.
Fails-without-the-fix check, run locally: I kept the new tests and put back the merge-base (fc006f71) versions of observability.ts and captureStreamingStage.ts. These tests then fail:
does not read a ten-hour sleep as no progress(vitest, 1 failed / 21 passed)does not trip a sequential capture the moment the machine wakesdoes not trip the parallel watchdog the moment the machine wakes(bun, 2 failed / 31 passed)
The on-time stall tests still pass at the merge-base. At this head, stallClock + observability pass 24/24 and captureStreamingStage passes 33/33.
Real hangs still trip:
stallClock.idleonly skips the time when a check arrives more than 15 s after it was due.- When checks arrive on time,
watchStalland the parallelsetIntervalcount silence exactly as before. raceAgainstStallnow checks every ≤5 s against its own clock, so a capture call that never returns still fails at its deadline.- A 0 ms deadline still fails right away: the timer fires at 0 and
idle(0) >= 0.
Non-blocking note, the drain loop's quiet.idle(0): there's no due time for this check, so any gap over 15 s since the last check or reset is skipped as if the machine had slept. That includes a capture call that was merely slow (say 50 s, under the 60 s budget) and wasn't followed by a frame write. The only such case is the first batch, where prevBatch is empty, so the next call gets the full budget instead of what was left of it. That allows silence of up to about 2× the budget once per render, not an unbounded hang, because every later iteration writes a frame and resets the clock. If you want it tight, a follow-up could pass the time the drain actually expected to wait, or skip only time spent outside a raceAgainstStall. It doesn't need to block this PR.
CI: Comments, PR captures, preview-regression and Player perf are green. CI, regression, Windows render verification and CodeQL were still running when I reviewed.
Edit accuracy: accurate 267 (base branch 267), smooth 108 of thoseThe gate passes. Quarantined, measured but not gated (8)
Unstable (1)
|
e325f3a to
c66b013
Compare
There was a problem hiding this comment.
Approving at c66b013a. This is my approval of e325f3a2 carried over to the rebase onto #7104. I confirmed the rebase changed nothing in this PR's own changes, and that the two features fit together.
- This PR's own changes are identical: I compared the
+/-lines of the old PR (fc006f71...e325f3a2) with this head's PR diff, and they match line for line. Only the code around them changed, because #7104 landed underneath. - How it fits with #7104:
- The parallel watchdog's
quietclock is created inside the resume loop, so each resumed attempt gets a fresh clock andresumeFromreporting is unchanged. - The sequential streaming loop (
lastProgressAt = Date.now(), around line 1090) still uses the wall clock. That code was already there before this PR and is safe: it resets right after each frame is written, and a sleep while a capture is in flight is handled byraceAgainstStall's own sleep-aware clock. So the only exposure is the sub-millisecond gap between the reset and the next race.
- The parallel watchdog's
- Tests at this head:
stallClock+observabilitypass 24/24,captureStreamingStage(including #7104's resume tests) passes 43/43 under bun, andtsc --noEmitis clean. - Fails without the fix, against the new merge-base
3dc82f18: I kept the new tests and put back main'sobservability.tsandcaptureStreamingStage.ts. These tests then fail:does not read a ten-hour sleep as no progress(1 failed / 21 passed)does not trip a sequential capture the moment the machine wakesdoes not trip the parallel watchdog the moment the machine wakes(2 failed / 41 passed)
My earlier non-blocking note about the drain loop's quiet.idle(0) still applies as written. No blockers.
Adopted with tests that fail on the old code (except the link change): - offload: rewrite media inside <template>, keep same-named files from other folders (heygen-com#6842, offload part only) - autoUpdate: no background upgrade across a 0.x minor; Homebrew installs only notify (heygen-com#7013, autoUpdate part only) - engine: script load failures carry origin/code; aborted scripts are not load failures (heygen-com#7088, heygen-com#7093) - community Discord invite links (heygen-com#7097) - studio: write media duration to the store once per file (heygen-com#6745) - check: restart the idle watchdog on each driver call during analysis (heygen-com#6919, adapted to this line's fixed 120s idle window) - check: default wait raised from 10s to 45s, matching snapshot/render (heygen-com#6947) - image cancellations no longer ignored by extension (heygen-com#6928) - capture: system sleep no longer counts as a stall (heygen-com#7096, hand-ported; the sequential branch also uses the shared stall clock) Commit watermark advances to 183614b (28 consecutive commits with file:line evidence); PR and issue watermarks are unchanged. The rest of the review is in docs/UPSTREAM_REVIEW_2026-11-04_CONTINUED.md. Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com> Co-authored-by: WaterrrForever <miao.yang@heygen.com> Co-authored-by: Wenbo Zhu <wenbo_zhu@berkeley.edu> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What
A laptop that sleeps in the middle of a render no longer has the sleep counted as a capture hang. On wake the render carries on instead of ending with "capture stalled: no frame progress".
Why
The three no-progress watchdogs (the single-worker capture race, the parallel capture timer, and the render-stage rule) measure silence against the wall clock or against a timer that keeps counting through sleep. After a sleep longer than the 60 s budget, the first check on wake sees "no frame for hours" and fails the render, though nothing was hung.
Clock facts, which decide the shape of the fix:
QueryPerformanceCounter, which libuv'suv_hrtimeuses, returns ticks since boot "including the time when the machine was in a sleep state such as standby, hibernate, or connected standby" (Microsoft Learn, Acquiring high-resolution time stamps). Node timers andperformance.now()count the sleep.uv__hrtimecallsmach_continuous_time(), which counts sleep (libuvsrc/unix/darwin.c, v1.x branch). I confirmed this from the source, not on a sleeping Mac, and did not pin the libuv release that introduced it.CLOCK_MONOTONICexcludes suspend (clock_gettime(2)), so there a monotonic clock would exclude the sleep.Because two of the three platforms count sleep in every clock Node offers, swapping
Date.now()for a monotonic clock is not the fix.How
stallClock.tsowns one rule: a gap between two looks at the silence that is more than 15 s past when the look was due means the process was suspended, and the whole gap is paused time. All three watchdogs read their silence from it (raceAgainstStall, the parallel streaming timer and the pipelined drain's remaining budget, andwatchStallinobservability.ts). A process that keeps running, with looks on time and no frame, still trips at the same budget as before.A render that is only slow, or whose event loop is blocked for under 15 s, is judged exactly as before.
Not changed
A stall still restarts the capture from frame 0 on retry. That is separate and not part of this change.
Test plan
Fails without the fix: at the merge-base the wake tests fail (observability, the sequential race, the parallel timer) and the on-time stall tests pass. Each test also fails when its own code is broken: no forgiveness in the clock, the clock always idle 0, the race never tripping, the parallel timer not forgiving, the parallel timer never tripping, the stage rule not forgiving, the stage rule never tripping.
Passes 3 runs in a row:
stallClock.test.tsandobservability.test.ts(24 tests) under vitest,captureStreamingStage.test.ts(33 tests) under bun.oxfmt --check,oxlint,tsc --noEmitand the comment checks pass.