Repository navigation
test(engine): nightly motion blur run over generated compositions - #7437
Conversation
A seeded generator of text-free compositions combines 19 features (nested canvas, overflow, clip path, scroll container, callbacks, a last-child rule) and checks the two pixel invariants on auto: outside the trail at two holds and at a moving time, inside the trail at a moving time. The runtime lane runs seeds 1 to 3; motion-blur-nightly.yml runs 200 a day on the Chrome 153 pin and on stable, and opens one issue per failing seed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nces, caps its issues A crash, a timeout or a launch failure is recorded before the job goes red; a failing check whose blur-off reference disagrees with itself is reported as unstable instead of failing; the report job moves into a tested script that fetches open titles once, caps first, and opens one systemic issue above 5 seeds. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…eps a crashed leg's record Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![heygengenesis[bot]](https://avatars.githubusercontent.com/in/2913576?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verdict: APPROVE · 0 blocker · 2 important · 2 nits · head aebfb99
The trust-boundary handling holds up. I found no injection path in the report job, so the two important findings are about signal quality. ↓ inline
Scope
Audited: scripts/ci/motion-blur-nightly-report.mjs (end to end), .github/workflows/motion-blur-nightly.yml, captureMotionBlur.generated.browser.test.ts, the differentialPage.ts diff. Trusting: generate.ts (read parseSeeds only), generate.test.ts, the report test (ran it locally: 0 fail).
Security checks that passed:
workflowCommandmatches the toolkit's escaping (data% CR LF, properties also: ,), so failure text can't start a new command line.inlinecode spans can't be closed because backticks are replaced.fenceduses a fence longer than the longest backtick run, so it can't be closed early.- Seed is checked with
isSafeInteger,datemust match^\d{4}-\d{2}-\d{2}$, and flavour comes from the artifact name the workflow sets. @/#text only appears inside code, so it can't ping anyone or link an issue.inputs.seedsreaches the job throughenvandjq --arg, with no${{ }}inrun:.issues: writeis granted onreportonly, and that checkout usespersist-credentials: false.
CI: all workflows green on this head.
No earlier reviews or comments on the PR.
— Genesis
| ): Promise<void> { | ||
| if (first.d.bad.length === 0) return; | ||
| const disagreement = await blurOffDisagreement(chrome(), f, region, t, first); | ||
| if (disagreement === undefined) { |
There was a problem hiding this comment.
important — An unstable reference can hide a real defect. blurOffDisagreement compares fresh blur-off recaptures only with first.reference, which comes from the reused referencePage. It never checks first.onFrame. If the reused page and a fresh page differ, which is plausible for once-guard, tl.call or onUpdate seeds after earlier seeks, every failure on that seed becomes "unstable" and goes to the summary only, however far off the blur-on frame is. Suggested fix: in the recapture loop, also compare first.onFrame with each fresh again under first.mask. Treat the result as noise only if blur-on agrees with at least one fresh blur-off. Otherwise keep it in problems.
| }; | ||
| } | ||
|
|
||
| const seedTitle = (seed) => `motion blur nightly: seed ${seed}`; |
There was a problem hiding this comment.
important — The title dedupe never matches on scheduled runs. Each day's seeds are days*200…+199, so seed N titles never repeat. A persistent defect that hits 1–5 seeds a night opens 1–5 new issues every night. systemic failure <date> also opens a new issue daily. Suggested fix: dedupe on the sorted feature set, or keep one open tracking issue per title family and comment on it instead of creating new issues.
| openIssue(gh, titles, systemicTitle(date), systemicBody(failing, date, runUrl), log); | ||
| return; | ||
| } | ||
| const entries = failing.slice(0, maxIssues); |
There was a problem hiding this comment.
nit — The default maxIssues (25) can't take effect. This branch runs only when failing.length <= systemicAbove (5), so the cap and its warning on L206–207 only work when someone overrides --systemic-above.
| flavour: [pin, stable] | ||
| chunk: ${{ fromJSON(needs.plan.outputs.chunks) }} | ||
| steps: | ||
| - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 |
There was a problem hiding this comment.
nit — This checkout also runs third-party code (bunx vitest and @puppeteer/browsers). Add persist-credentials: false here as well, so the contents: read token isn't left in .git/config while the job runs. The repo is public, so the risk is low, but this would match the report job.
…a fixpoint Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Verdict: APPROVE — 0 resolved, 4 open (non-blocking), 0 new · aebfb99 → cdba44b. The delta only addresses the CodeQL alerts: inline() now escapes backslashes first, and a new p\|q table row covers it. The test helper strips tags until nothing changes. CodeQL and all other CI checks are green on this head.
…ve more; advance watermark Twelve commits, PRs 7453-7468, issues 7457-7465 and eleven pin moves are recorded with their evidence in the continued ledger, DECISIONS and UPSTREAM. Adopted: heygen-com#7338, heygen-com#7459, heygen-com#7463, heygen-com#7454, heygen-com#7456 and the heygen-com#7439 docs; heygen-com#6732 was already ported. Deferred: heygen-com#7455, heygen-com#7437. Not applicable: heygen-com#7453, heygen-com#7269, heygen-com#7464. The watermark moves to 9e346f9, PR heygen-com#7468, issue heygen-com#7465; four new PRs are pinned and the eleven moved pins re-pinned at their current heads. FORK.md records the fork-only differences in the heygen-com#7338 port; the roadmap tracks the standalone scripts that keep a bare mkdtemp and issue heygen-com#7458. Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
What
Tests and CI only; no runtime change. A nightly run that renders generated compositions with motion blur on and checks the picture against blur off, so combinations of features nobody wrote a fixture for get looked at. It is one of the gates for flipping blur default-on in Desktop.
How it works
generate.ts:generateComposition(seed)builds a 320x180, 2 s, 30 fps composition from 1 to 4 of 19 features (transform, left/top mover, rotation, overflow parent, scroll container, canvas2d, webgl, svg clip-path, mask, onUpdate mover,tl.call, once-guard, CSS keyframes, nesting, pseudo-elements, panel in front, blend mode on a non-mover, fixed child,:last-child). Deterministic per seed. All extra motion sits in 0..0.4 s and 1.6..2.0 s, so every checkpoint is 3 frames from any DOM change. Sweep defects E and O are excluded from the generator (EXCLUDED_FEATURES) because they are known and pinned elsewhere. No tier pin, soauto(the filter tier since feat(core): auto motion blur draws with the filter tier #7288) is what gets tested.captureMotionBlur.generated.browser.test.ts:HF_BLUR_SEEDS(list ora-b, default1,2,3). Per seed: outside the trail at two hold frames and at a moving frame, inside the trail at the moving frame, with the same invariants as the differential suite. A failure printsseed 17 [rotation, panelInFront]: rerun with HF_BLUR_SEEDS=17.HF_BLUR_REQUIRED=1makes a missing Chrome throw instead of skip, so the nightly cannot pass by skipping everything..github/workflows/motion-blur-nightly.yml: daily at 08:41 UTC and on dispatch with aseedsinput. 200 seeds a day as 4 chunks of 50, on Chrome 153 pinned and on stable,fail-fast: false. Areportjob opens one issue per failing seed (deduped by exact title, capped at 25 a run), or onesystemic failure <date>issue when more than 5 seeds fail.scripts/ci/motion-blur-nightly-report.mjsholds the report logic (with a test run bytest:scripts).Proof it catches things
With #7293 reverted locally (
document.documentElement.append(defs)back toroot.append(defs)), 28 of 28 seeds in 1-200 that carrylastChildfail; on the restored runtime 199 of 200 pass. The one failure, seed 129, is not a blur defect: Chrome renders two blur-off captures of it differently about 70% of runs (a nestedwill-changecanvas next to a clip-path sibling). So when a check fails, the test recaptures blur off on up to 3 fresh pages, and if those disagree with the first, the seed is recorded as an unstable reference (listed in the run summary, no issue). With that, seed 129 passed 24 of 24 runs, and the 28 real failures stayed real.Failure recording and the issue job
onTestFailed(assertion, page crash, vitest timeout) inblur-nightly-failures.json, written before Chrome is stopped, so a close error cannot lose it. A Chrome launch failure records a seedless entry.::workflow commands are escaped for%, CR, LF and made inert if the message contains::, and markdown from the artifact is confined to code spans and fences. Both have tests with injection attempts.persist-credentials: false; permissions areissues: write, contents: readon that job only.What this does not do
KNOWN_BADis keyed by seed and ships empty.needs/ifreferences and thejqprograms locally, not withactionlint. First real run is the first scheduled one, or aworkflow_dispatchafter merge.Verification
Chrome 153 (
PRODUCER_HEADLESS_SHELL_PATH),--maxWorkers=2:generate.test.ts8 pass; default seeds 1,2,3 pass in about 6 s idle and 8 s under 14 CPU burners; report test 21 pass; inside-case differential file 24 pass after rebase. Reachability, classification, comment ratchet and citation checks clean.🤖 Generated with Claude Code