Skip to content

Synthesise a soundtrack on the same clock as the cuts #331

Description

@LeadcodeDev

Part of #326. Batch 2 — depends on #336 for TimePoint and the beat grid.

AudioTrack requires a src. An LLM can write a score and cannot hand over a WAV, so every generated scenario is silent. Both reference reels synthesise their entire soundtrack from oscillators, noise and filters, with no audio file anywhere, and the SVG one renders it offline into a buffer before muxing. This is the single largest gap between a generated scenario and a generated reel, measured by how much of the perceived result it carries.

Owned files

  • crates/rustmotion-core/src/audio/ (new: voices.rs, score.rs, synth.rs, dsp.rs, mod.rs)
  • crates/rustmotion/src/encode/audio/ (the offline render and the mux path)

Do NOT write: schema/scenario.rs (#336 registers the audio.synth field), anything under engine/.

Deliverables

  1. A declarative score sharing the visuals' clock:

    "audio": { "bpm": 115,
      "voices": { "kick": { "type": "sine", "freq": [150, 42], "sweep": 0.14, "decay": 0.38 },
                  "hat":  { "type": "noise", "filter": {"type":"highpass","freq":7500}, "decay": 0.05 } },
      "score":  [ { "voice": "kick", "every": "1b", "from": "@2.2s", "to": "@12.6s" },
                  { "voice": "hat",  "every": "1b", "offset": "0.5b", "from": "@2.2s" } ],
      "master": { "compressor": { "threshold": -14, "ratio": 4 }, "limiter": true } }

    every, from, at and offset go through Place scenes on an absolute timeline with a beat grid #336's TimePoint parser. Sharing bpm and beat_offset with the cuts is the whole point: it is what makes an edit land on a downbeat rather than near one.

  2. A small, boring synth. Sine, square, saw and triangle oscillators; a noise buffer; biquad filters in lowpass, highpass, bandpass and notch; ADSR envelopes; a bus with gain, a compressor and a limiter. Rendered offline to an f32 buffer at 48 kHz.

  3. Coexistence with file-based tracks. audio accepts both the existing AudioTrack entries and a synth block in the same scenario, mixed into one bus.

  4. The limiter is a default, not an option. The SVG reel's author found his first mix clipping at 0 dB only by reading ffmpeg output after the fact.

  5. --frames a-b keeps its audio slice. A segment renders its own slice of the synthesised buffer, not a restart from zero, matching what the existing file-based path already guarantees.

Out of scope for this workstream: convolution reverb, sample loading, any effects rack. A synth that grows is a synth that stops being reviewable.

Acceptance

  • A scenario with no audio file and a 60-event score produces a muxed soundtrack
  • Two renders of that scenario produce byte-identical audio
  • Peak never exceeds -0.1 dBTP with the limiter on

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions