Repository navigation
feat: detect speech natively for local audio tracks - #490
Merged
Merged
Conversation
WebRTC's LocalAudioSource ignores AddSink, so VoiceActivityExample attached to a microphone track hears nothing. Say which tracks feed a sink, and point to AudioRecorder for the microphone.
LocalAudio.onAudio hands pushed audio as pushed, and a microphone's audio after audio processing, as the senders send it. Every AudioProcessing now carries a capture post-processor that hands its output to a sink, through AudioProcessing.setCapturedAudioSink and PeerConnectionFactory.setCapturedAudioSink.
The VAD guide, VoiceActivityExample and the audio processing guide show the processed microphone audio of the client API and of PeerConnectionFactory.setCapturedAudioSink.
# Conflicts: # docs/tools/audio/voice-activity-detector.md # webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/VoiceActivityExample.java
Replaces LocalAudio.onAudio with onVoiceActivity: the native VAD runs in the audio processing tap (microphone) or in a track sink (pushed audio), and only speech start and end reach Java. VoiceActivityConfig sets threshold and hangover.
devopvoid
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LocalAudio.onVoiceActivitytells when this side starts and stops speaking. The voice activity detector runs in native code and only the changes reach Java;VoiceActivityConfigsets the threshold and the hangover.WebRTC's microphone source hands no audio to sinks, so every
AudioProcessingnow carries a capture post-processor that detects speech in its output, after echo cancellation, noise suppression and gain control:PeerConnectionFactory.addVoiceActivityListener. Pushed audio, and remote tracks, take a native sink instead:AudioTrack.addVoiceActivityListener. The audio is mixed down to mono and resampled to 16 kHz first, so any rate works, 44.1 kHz included.