Most video pipelines are a pile of scripts glued together with hope. This one is a pipeline. Every stage has one job, a typed contract in and out, and a reason to exist.
An end-to-end system that turns a long-form English video into a short, Chinese-dubbed, vertical highlight — automatically, and at a cost per video measured in cents, not dollars.
Not a wrapper around an API. A real pipeline: six stages, each one solving a problem the others can't.
S0 curate — score and shortlist source videos before you waste compute on the wrong ones
S1 translate — transcribe, translate, forced-align (WhisperX)
S2 dub — synthesize only the selected span, not the whole video (this alone cuts TTS cost ~90%+)
S3 highlight — score and select the moment worth watching; restructure it into a narrator/character dialectic
S4 broll — smart-crop for faces, fetch context assets, insert reaction cutaways at the right beat
S5 render — Remotion, composited, encoded, delivered with cover art and publish metadata
Deferred synthesis is the core efficiency idea: don't dub what you might throw away. Score first, dub last.
- Forced alignment, not guesswork — WhisperX Chinese wav2vec2 alignment closes the gap when a TTS provider gives you audio but no timestamps. Word-level captions, not "roughly when it should show up."
- Narrator/character dialectic — not a straight dub. An LLM restructures the highlight into an opening-conflict question, a mid-clip challenge, a close — the difference between "reading a transcript aloud" and "making someone want to keep watching."
- Face-aware reframing — 16:9 doesn't become 9:16 by cropping the middle and hoping. SCRFD detection keeps the speaker in frame.
- Reaction cutaways — emotion and object cues detected in the script pull matching GIFs/images/SFX and cut them in, timed to the beat.
- Everything retries, nothing silently eats a whole video — transient network failures retry with backoff; alignment failures degrade gracefully to proportional-timing captions instead of killing the run.
Python pipeline orchestration · WhisperX · DeepSeek for script restructuring · Fish Audio TTS · Remotion for composition · ffmpeg everywhere it matters.
Cut every stage that doesn't need to run. Retry every failure that might be transient. Never let an optional feature take down a required one. Ship the thing that works, then make it better — in that order, not the reverse.