Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion docs/guide/advanced/encoded-transforms.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,20 @@ Set the transform of a video sender before negotiating when you can. The first t

The transform belongs to the native sender or receiver, not to the Java object: every `RTCRtpSender` instance for the same sender shares it, setting it through one replaces what was set through another, and disposing of an instance leaves it in place.

### With the Client API

The [Client API](/guide/client-api#encoded-transforms) takes the same transforms. Those of the peer options apply to every sender and remote track of the peer, and are set before the first frame passes either way:

```java
PeerOptions.builder()
.signaling(signaling)
.sendTransform(frame -> encrypt(frame))
.receiveTransform(frame -> decrypt(frame))
.build();
```

`Sender.transform(...)` and `RemoteTrack.transform(...)` replace them for one sender or track.

## Reading and Changing the Payload

`getData()` returns the payload in a `ByteBuffer`, from position zero to its size. The buffer holds a copy, which may be changed freely; `setData(...)` then writes it back:
Expand Down Expand Up @@ -91,7 +105,7 @@ Both do nothing for audio.

## End-to-End Encryption

A complete example with AES-GCM is in [`EncryptedRecordingExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java). Two points matter for any encrypting transform:
A complete example with AES-GCM, on the transforms of the client API's peer options, is in [`EncryptedRecordingExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java). Two points matter for any encrypting transform:

- **Keep the codec header in the clear.** The packetizer and depacketizer read the start of each frame: a VP8 receiver finds key frames and the frame size in the first bytes. Browsers leave 10 bytes of a VP8 key frame, 3 of a VP8 delta frame and 1 of an Opus frame unencrypted, and authenticate them as associated data instead. H.264 is split at its NAL unit start codes before it is sent, so encrypting it whole breaks the packetizer; prefer VP8, VP9 or AV1 for encrypted calls, or leave the NAL unit headers in the clear.
- **Make each frame self-contained.** Frames are lost, so each needs what it takes to be decrypted, such as its IV, and a receiver must drop a frame that fails to authenticate rather than pass it on.
Expand Down
26 changes: 26 additions & 0 deletions docs/guide/client-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,7 @@ Every method of the client API may be called from any thread, including from ins
| Your calls into the API | your thread | They don't block. The exceptions are `WebRtc.createPeer`, which waits briefly for the connection, and `WebRtc.close()`, which waits up to the close timeout. |
| Peer, track and channel handlers, future callbacks, `Signaling.send` | the peer's event thread | Yes, but that delays only the events of this peer. They must not wait for a future of this API, see below. |
| `onFrame` and `onAudio` handlers | a thread of each subscription | Better not. A slow video handler gets the newest frame next, and the frames in between are dropped; a slow audio handler loses the oldest audio beyond its buffer. |
| [Encoded transforms](#encoded-transforms) | a thread of each sender and receiver | Better not. A slow transform delays only its own frames, and one that falls several seconds behind has frames dropped. |
| `LocalTrack.onEnded` and `Devices.onChange` handlers, callbacks of the futures of local tracks and devices | the event thread of the runtime's local media | Yes, but that delays the other events of local media. They must not wait for a future of this API either. A track is released before its `onEnded` handlers run, so they never hold up a release. |
| WebRTC's own callbacks | WebRTC's threads | Not your code. The library only copies what they report and queues it. |

Expand Down Expand Up @@ -548,6 +549,31 @@ try (MediaRecorder recorder = new MediaRecorder(Paths.get("call.mkv"))) {

Closing the peer, or removing a sender, during a recording ends that track in the file, and `stop()` still finishes it. The [Media Recording](/guide/media/media-recording) guide covers containers and the recorder's events.

### Encoded Transforms

An [encoded transform](/guide/advanced/encoded-transforms) sees every encoded frame between the encoder and the network, or between the network and the decoder, which is where end-to-end encryption goes. The transforms of the peer options apply to every sender and every remote track of the peer:

```java
Peer peer = rtc.createPeer(PeerOptions.builder()
.signaling(signaling)
.sendTransform(frame -> encrypt(frame))
.receiveTransform(frame -> decrypt(frame))
.build());
```

The peer sets the send transform on each sender as its track is added, before negotiation, so no frame leaves without it and the encoder needs no restart. It sets the receive transform on each receiver before `onTrack` hands out the track, so the transform is in place before the first frame arrives.

A single sender or remote track takes a transform of its own, in place of the peer's, and `null` removes it:

```java
videoSender.transform(frame -> addMetadata(frame));
remoteTrack.transform(null); // frames reach the decoder unchanged
```

The first transform set on a video sender that already sends restarts its encoder, which costs a key frame. A transform set on a remote track misses the frames that arrive before its future completes, so a transform that every frame needs, such as decryption, belongs in the peer options. A recording of a sender holds its frames before the transform, a recording of a remote track after it.

Transforms run on a thread of each sender and receiver, not on the peer's event thread. A frame a transform throws on is dropped rather than passed on, so a failed encryption never sends a frame in the clear. The first such failure of each sender or receiver goes to the peer's `onError` handlers. The [`EncryptedRecordingExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java) encrypts a call with AES-GCM this way.

## Data Channels

```java
Expand Down
6 changes: 3 additions & 3 deletions docs/guide/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,13 +149,13 @@ mvn exec:java -D"exec.mainClass=dev.onvoid.webrtc.examples.JavaFxCameraExample"

## Encrypted Recording

The [`EncryptedRecordingExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java) encrypts a call end to end with encoded frame transforms, and records what the receiving side decrypted into a media file. See the [Encoded Transforms](/guide/advanced/encoded-transforms) and [Media Recording](/guide/media/media-recording) guides for the APIs it uses.
The [`EncryptedRecordingExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java) encrypts a call end to end with encoded frame transforms, with the [Client API](/guide/client-api#encoded-transforms), and records what the receiving side decrypted into a media file. See the [Encoded Transforms](/guide/advanced/encoded-transforms) and [Media Recording](/guide/media/media-recording) guides for the APIs it uses.

**Key features demonstrated:**
- Encrypting every encoded audio and video frame with AES-GCM in an `RTCEncodedFrameTransformer` on the senders, and decrypting it on the receivers
- Encrypting every encoded audio and video frame with AES-GCM in the send transform of the peer options, and decrypting it in the receive transform, from the first frame on
- Keeping the codec header of each frame in the clear for the packetizer, and authenticating it instead
- Dropping frames that fail to encrypt or to authenticate, rather than passing them on
- Recording the receivers with a `MediaRecorder`, without decoding or re-encoding
- Recording the remote tracks with a `MediaRecorder`, without decoding or re-encoding

::: info
Like the [Media File](#media-file) example, this one needs the `webrtc-java-media` module. The output file is optional; its extension picks the container.
Expand Down
Loading
Loading