Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 33 additions & 6 deletions docs/tools/audio/voice-activity-detector.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,14 +120,38 @@ state, or transmitted audio.

## Audio track integration

The examples module includes `VoiceActivityExample`, an `AudioTrackSink` adapter
using `StandaloneVoiceActivityDetector`. It buffers callback data into 10 ms
frames, converts mono or stereo input to 16 kHz mono with `AudioConverter`, and
reports each boolean decision. It also supports 44.1 kHz input.
The examples module includes `VoiceActivityExample`, an adapter using
`StandaloneVoiceActivityDetector`. It buffers incoming audio into 10 ms frames,
converts mono or stereo input to 16 kHz mono with `AudioConverter`, and reports
each boolean decision. It also supports 44.1 kHz input.

With the [Client API](/guide/client-api), it takes the `AudioChunk`s of a remote
track's `onAudio` subscription:

```java
import dev.onvoid.webrtc.client.Subscription;
import dev.onvoid.webrtc.examples.VoiceActivityExample;

peer.onTrack(track -> {
if (track.isAudio()) {
VoiceActivityExample vad = new VoiceActivityExample(speech -> {
// Enqueue the decision for your application/UI; return promptly.
});
Subscription audio = track.asAudio().onAudio(vad);

track.onEnded(() -> {
audio.close();
vad.close();
});
}
});
```

The client API has no audio subscription on a local track. To detect speech at
the microphone, attach the adapter as an `AudioTrackSink` to an `AudioTrack` of
the native API:

```java
VoiceActivityExample sink = new VoiceActivityExample(speech -> {
// Enqueue the decision for your application/UI; return promptly.
});
Expand All @@ -141,8 +165,11 @@ finally {
}
```

The adapter invokes the consumer on the audio callback thread. Keep work there
short. Apply application-level timing to produce stable speech-start/speech-end
The adapter invokes the consumer on the thread that delivers the audio: a thread
of the `onAudio` subscription, or the audio callback thread of a sink. Keep work
there short. A subscription that falls behind loses the oldest audio beyond its
buffer, so a detector that must stay live passes a short one, as in
`onAudio(Duration.ofMillis(200), vad)`. Apply application-level timing to produce stable speech-start/speech-end
events, rather than toggling a UI or recording for every frame. Audio format
changes clear the adapter's partial frame and reset its detector.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,18 +21,24 @@
import java.util.function.Consumer;
import java.util.Objects;

import dev.onvoid.webrtc.client.AudioChunk;
import dev.onvoid.webrtc.client.RemoteAudio;
import dev.onvoid.webrtc.client.Subscription;
import dev.onvoid.webrtc.media.audio.AudioConverter;
import dev.onvoid.webrtc.media.audio.AudioTrackSink;
import dev.onvoid.webrtc.media.audio.StandaloneVoiceActivityDetector;

/**
* Attach to an AudioTrack with addSink(), then removeSink() before close().
* Buffers arbitrary callback lengths into 10 ms frames and converts interleaved
* With the client API, subscribe it to a remote track with
* {@link RemoteAudio#onAudio(Consumer)}, then close the {@link Subscription}
* before close(). An AudioTrack of the native API, such as a microphone's,
* takes it with addSink(), then removeSink() before close().
* Buffers arbitrary chunk lengths into 10 ms frames and converts interleaved
* signed 16-bit PCM to 16 kHz mono, including input at 44.1 kHz. The consumer runs
* on the audio callback thread and should return promptly. It receives raw frame
* decisions; a UI should apply its own speaking-state timing off this thread.
* on the thread that delivers the audio and should return promptly. It receives raw
* frame decisions; a UI should apply its own speaking-state timing off this thread.
*/
public final class VoiceActivityExample implements AudioTrackSink, AutoCloseable {
public final class VoiceActivityExample implements Consumer<AudioChunk>, AudioTrackSink, AutoCloseable {

private final Consumer<Boolean> onDecision;
private final StandaloneVoiceActivityDetector detector;
Expand All @@ -51,6 +57,11 @@ public VoiceActivityExample(Consumer<Boolean> onDecision) {
detector = new StandaloneVoiceActivityDetector(StandaloneVoiceActivityDetector.Mode.AGGRESSIVE);
}

@Override
public void accept(AudioChunk chunk) {
onData(chunk.samples(), 16, chunk.sampleRate(), chunk.channels(), chunk.frames());
}

@Override
public synchronized void onData(byte[] data, int bitsPerSample, int sampleRate, int channels, int frames) {
if (closed) {
Expand Down
Loading