Linked narration: editing a line updates its voice-over and captions together #6708

Activity

1. jrusso1020

   added

   enhancementNew feature or requestNew feature or request

   triage/needs-designEvidence is adequate; desired behavior or scope needs a maintainer decision.Evidence is adequate; desired behavior or scope needs a maintainer decision.

   on Oct 30, 2026
2. DhanushMurugesanG commented on Nov 3, 2026

   

   DhanushMurugesanG

   on Nov 3, 2026

   Author

   More actions

   Thanks for the labels! Here's a concrete design for this, so it's easier to say yes, no or "change X". I won't open any PR for it until the design is agreed, and I'll change it to match what you decide. Separately, while reading the code I found a small bug in the media-use audio engine: a partial --only tts run drops every voice it didn't re-voice. I've sent that as its own small PR: #6816.

   Proposed design

   1. narration.json at the project root, next to caption-overrides.json. Undo history tracks it there, but wouldn't under .hyperframes/. Each line has:

      • id, text, an optional voice/speed/lang
      • audio, durationSeconds
      • words (seconds, relative to the clip)
      • voiced: { textHash, voiceKey, audioHash }, recorded when the line was voiced

      Unknown keys are kept when the file is rewritten.

   2. Links:

      • <audio data-narration-line="l2"> keeps owning the clip's start and duration.
      • Caption word objects get an optional line: "l2", because caption spans are built by script and can't carry attributes. Untagged words fall back to the clip's time window.

   3. Lint (a project probe in packages/lint/src/project.ts, all warning or info, never blocking a render):

      • narration_manifest_invalid
      • narration_line_unlinked
      • narration_audio_stale (the text, voice or WAV changed)
      • narration_clip_timing_mismatch
      • narration_captions_out_of_sync
      • narration_transcript_mismatch (info)

   4. CLI:

      • hyperframes narration list | init | check
      • hyperframes narration set <id> ["text"]: re-voice one line into a temp folder, transcribe that clip only (outside the project, so patchCaptionHtml doesn't touch the project), splice that line's caption words, update the manifest and the clip's duration, and remap caption-overrides.json indices. The writes go through applyFileMutations with expectedVersion, as one history entry. --dry-run and --no-retime are supported.
      • A later PR would ripple later narration clips using the timeline plan code.

   5. Shared code: goes in @hyperframes/parsers (./narration, ./caption-array), because lint depends only on parsers.

   PRs, in order:

   1. lint
   2. narration list/init/check
   3. narration set
   4. ripple

   Studio "Re-voice this line" would come last, and only if you want it.

   Questions before I open PR 1

   1. Are you OK with a root narration.json, data-narration-line on <audio>, and line on caption words?
   2. Which caption arrays should set support: script/TRANSCRIPT with const/let/var? Is WORDS in, and GROUPS out?
   3. Default retiming: ripple only narration clips and their captions, warn about visuals, and never extend the root duration?
   4. When a line's word count changes, should per-word overrides on that line be dropped (and reported), or kept by index?
   5. History: one explicit window per set, or join an agent's open turn? Which label and --who?
   6. Superseded WAVs: delete them (undo restores them), or keep them? Are content-addressed names (assets/voice/l2-<hash8>.wav) OK?
   7. Kokoro only in v1, with other providers pointed to media-use --only tts?
   8. Is warning/info severity right, and is hashing the WAVs on each lint run acceptable?
   9. Do you want a narration check subcommand, or are lint/check enough?
   10. Is @hyperframes/parsers the right home, with findCaptionArray and Studio's extractTranscript moved onto it later?
   11. Should line ids link to the storyboard voiceover / SCRIPT.md from feat(studio): storyboard frame focus + voiceover iteration #1532?