Repository navigation
feat(cli): read audio stems through your HeyGen account - #6993
Conversation
Measures where each stem is heard, leaves the drums' recurring hits to the music while it plays, names the stem of each remaining effect, writes a voice-free mix next to the stems, and describes each audible stem and the whole recording with Gemini.
Piano, guitar and other are mixed into melodic.wav next to the stems; levels, spans, descriptions, effects and pieces are read per group, and a group heard for under two seconds or 5% of the recording is left out of the reading.
…pt from them Stems are read one by one again; a stem heard under two seconds or 5% of the recording is still left out. Each stem's description gives the times of its changes and names the effects it hears. The overall and the BGM prompt are written from those descriptions and the measured spans, not by listening to the mix again; the BGM prompt orders the arrangement without seconds.
…usic Foley and effects in a rhythm are not music; the reading says so instead of a prompt that would make a remix generate a track the reference never had.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
VerificationCurrent head: The runtime checks below ran at
Description requests were mocked. Tests cover HeyGen API-key/access-token authentication, base64 audio, schema configuration, summary parsing with both a string and null music prompt, no credential/no request, failure warnings, cache reuse and no Google/Gemini key reads. Not covered: live hosted-description requests, desktop end-to-end testing, a full-repository suite, or a documentation-site build. The comment-ratchet check could not run with local Git 2.39 because This PR must remain unmerged until production audio support is serving and a live |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
RV2 inputs for the read_audio verification (from the handoff: run read_audio on the RV2 "That's Framer" reference through Desktop's real path, with the HeyGen credential only, and post the raw output). The files are on HeyGenverse (sign in with your HeyGen account to download):
The stems are 128 kbps MP3s uploaded with a How to use them:
|
|
RV2 cached-stem verification, using the supplied inputs. Ran commit Result: measured stems and generic Not covered yet: description through the live HeyGen route, pending the server audio-support rollout; fresh stem separation; Desktop's end-to-end path. Full raw combined stdout/stderr, unchanged: — Jerrai |
|
Live
|
Use portable stem path assertions and simplify response validation. Keep auth-derived URLs out of cache hashes and trim redundant comments. — Jerrai
![heygengenesis[bot]](https://avatars.githubusercontent.com/in/2913576?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verdict: APPROVE · 0 blocker · 0 important · 2 nits
No blockers. CI at this head is green (both CI runs, CodeQL, Windows render, regression); the remaining BLOCKED state is the review gate. The description cache key no longer includes the endpoint URL, and the two remaining CodeQL notes are explained in the body. Nits are inline.
Scope
Read end-to-end: describe.ts, read.ts, effects.ts, music.ts, format.ts, types.ts, the providers/shared.ts change, and both docs pages. Ran src/audio/read/ locally at this head: 7 files, 86 tests passed. Skimmed the tests. No prior human reviews on the PR.
— Claude (Genesis V2)
| entry.analysis.separation === "stems" && | ||
| !entry.analysis.description && | ||
| quality !== "fast" && | ||
| typeof (await describeAccess()) !== "string" |
There was a problem hiding this comment.
nit: for a signed-in user, this skips the cache whenever a stems reading has no description. Some readings can never get one: when no stem passes heardLongEnough (any clip under 2 s, or a near-silent one), describeStems returns early with description: null and no warning. Every later --read of that file then reruns the full analysis and returns cached: false. Gating on readings that have stems to describe fixes it:
| typeof (await describeAccess()) !== "string" | |
| entry.analysis.stems.length > 0 && | |
| typeof (await describeAccess()) !== "string" |
| before the effect (after it, when the music starts with it), `+12 dB over what | ||
| played before` against the recording's median level in the half second | ||
| before, or `out of silence`. A sound ends where the next one starts. Quick | ||
| runs of one sound are a single line with a count; hits on the music's beat |
There was a problem hiding this comment.
nit: since musicHits landed, the left-out drum hits also include sounds that recur in the drum stem while music plays, not only hits on the beat. The types.ts comment and the summary line (hits of the music's own drums left out) already say this; this sentence still describes only the old on-beat rule.
|
Lane sign-off (cutroom-dev:dsh) at head 631b21d.
|
What
Read a recording by its six audio stems: vocals, drums, bass, guitar, piano and other. Describe each audible stem, summarize the soundtrack, and add
voice-free mix:andbgm prompt:output lines. The structured music-prompt value is null when there is no music.Descriptions now use the signed-in HeyGen account or HeyGen API key, including credentials supplied by the desktop app. No separate Google or Gemini API key is used.
Every measured sound-effect event is labelled
effect. Similar sounds remain grouped asS1,S2and so on, with their time, source stem, band, attack, tail, peak and level above music preserved. The CLI reference and audio guide describe this output.Why
One HeyGen credential should be enough to read a soundtrack. Heuristic effect-kind labels were unreliable, so the output keeps the measurements instead of assigning a sound name. The soundtrack summary and music prompt use an API-enforced schema rather than parsing labels from free-form text.
Missing credentials or failed description requests produce one warning line explaining why descriptions are unavailable; the reader does not switch providers or guess a description.
Related work and merge prerequisites
audio --readrun through the HeyGen endpoint passed in a development environment using only a HeyGen CLI sign-in, without a Google or Gemini key: five stem descriptions and one overall summary, with all six requests returning HTTP 200.The production check is the first post-release run.
How
audio/mpegblocks through the shared HeyGen credential path.{ overall: string, bgm_prompt: string | null }through the API's JSON-schema output configuration.CodeQL notes
Test plan
GEMINI_API_KEYread made its test fail; restoring the implementation returned the suite to green.The mocked-request tests cover API-key and access-token headers, audio blocks, enforced output schema, both response fields including
bgm_prompt: null, missing credentials, failure warnings, cache reuse and the absence of provider-key reads.Initial runtime checks ran at
8dc92aeef7a73541ad98f38310f0330cd84a16cc. At631b21d967986c9815e80200a8eeef1065745614, both changed audio test files passed (41 tests, at most six workers), including malformed responses, mixed content blocks, cache identity and the portable stem-path assertion. The hosted Comments check passed at this head. No full-repository test suite was run locally.