Repository navigation
Lessons in a voice chat: the music waits for the voice (0.11.1) - #39
Merged
Merged
Conversation
The instruction a notifyWhenDone render sends the model said "Until then, say briefly what to listen for", so in MulmoChat the model talked over the music. It now says to say nothing until the "finished playing" message arrives; the guide's sheet-music section and the notifyWhenDone description say to tell the user what to listen for before renderScore instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A voice host's model speaks and calls renderScore in one turn, so the card appeared and started playing while the speech was still going, and the music and the voice overlapped whatever the prompt said. The view now takes the protocol's isAudioPlaying: what plays by itself starts only after the voice has been quiet for 1.5 s, pauses when the voice speaks again and resumes where it was. Once the user clicks play or the bar, the voice no longer pauses it. In a host without a voice (isAudioPlaying never set) it plays at once, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Told to say nothing in the turn a voice host always gives it after a tool
result, the model described the player ("playing the music", "waiting to
play"). It now says only a few words that the music is about to play, like
"Here comes the sound.", then nothing until the "finished playing" message.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
If the voice started while play() was still pending (the audio buffering), `playing` was still false, so the view did not pause it and the music began under the voice. It now pauses whenever the element is not paused, which play() sets at once, and a start aborted that way waits for the next quiet instead of asking for a click. Checked in a browser with play() held pending for 1 s: paused at once and resumed after the voice; without the fix it played on under the voice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
If the user pressed play or the bar while autoplay was still waiting for the voice, `autoplayed` was never remembered, so a remount of the still-fresh card played it by itself again. Taking over now records it. Checked in a browser: a click during the wait writes autoplayed to viewState; before the fix nothing was written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cue chips called play() directly, so music the user started from a cue was still paused by the voice. The bar, the waveform and the cues now all go through playFrom, which takes over first. Checked in a browser: after a cue click the voice no longer pauses it; before the fix it did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while teaching with sheet music in MulmoChat's voice mode: the model's speech and the music overlapped, and the model filled the turn after renderScore with lines like "playing the music / waiting to play".
The music waits for the voice
A voice model speaks and calls renderScore in one turn, so the card appeared and autoplayed while that speech was still playing. The view now takes the protocol's
isAudioPlaying:QUIET_MS,quietWaitingui-plugin/src/vue/playback.ts);isAudioPlayingnever set) plays at once, as before.What the model says
A voice host always gives the model a turn after a tool result. The instruction for a
notifyWhenDonerender said "Until then, say briefly what to listen for", which talked over the music. It now asks for only a few words that the music is about to play ("Here comes the sound."), then nothing until the "finished playing" message. The guide and thenotifyWhenDonedescription say the same.Checks
quietWait).Both packages are 0.11.1; the plugin's peer stays
jinglescript ^0.11.0.🤖 Generated with Claude Code
work in jinglescript