feat: decode video with VA-API on Linux

.github/actions/build/action.yml

@@ -50,8 +50,8 @@ runs:
       uses: actions/cache/restore@v6
       with:
         path: ~/ffmpeg
-        key: ffmpeg-${{ env.FFMPEG_VERSION }}-${{ inputs.platform-name }}-${{ hashFiles('webrtc-media/src/main/cpp/dependencies/ffmpeg/CMakeLists.txt', 'webrtc-media/src/main/cpp/dependencies/ffmpeg/patches/*.patch') }}-${{ hashFiles('webrtc-media/pom.xml') }}
-        restore-keys: ffmpeg-${{ env.FFMPEG_VERSION }}-${{ inputs.platform-name }}-${{ hashFiles('webrtc-media/src/main/cpp/dependencies/ffmpeg/CMakeLists.txt', 'webrtc-media/src/main/cpp/dependencies/ffmpeg/patches/*.patch') }}-
+        key: ffmpeg-${{ env.FFMPEG_VERSION }}-${{ inputs.platform-name }}-${{ hashFiles('webrtc-media/src/main/cpp/dependencies/ffmpeg/CMakeLists.txt', 'webrtc-media/src/main/cpp/dependencies/ffmpeg/patches/*.patch', 'webrtc-media/src/main/cpp/dependencies/libva/*') }}-${{ hashFiles('webrtc-media/pom.xml') }}
+        restore-keys: ffmpeg-${{ env.FFMPEG_VERSION }}-${{ inputs.platform-name }}-${{ hashFiles('webrtc-media/src/main/cpp/dependencies/ffmpeg/CMakeLists.txt', 'webrtc-media/src/main/cpp/dependencies/ffmpeg/patches/*.patch', 'webrtc-media/src/main/cpp/dependencies/libva/*') }}-
 
     # One cache per OS: the dependencies do not differ between the platforms of
     # an OS. The project's own artifacts change with every build and stay out

.github/actions/prepare-linux/action.yml

@@ -31,9 +31,11 @@ runs:
     - name: Install required packages
       run: |
         sudo apt-get update -q
-        # nasm is what FFmpeg assembles its x86 code with, and meson builds
-        # dav1d, the software decoder of AV1 in the media module.
-        sudo apt-get install -y -q binutils cmake git locales lsb-release meson nasm ninja-build pipewire pipewire-pulse pkg-config python3 python3-setuptools rsync unzip wget xz-utils
+        # nasm is what FFmpeg assembles its x86 code with, meson builds
+        # dav1d, the software decoder of AV1 in the media module, and
+        # libva-dev has the VA-API headers FFmpeg is built against, for every
+        # target.
+        sudo apt-get install -y -q binutils cmake git libva-dev locales lsb-release meson nasm ninja-build pipewire pipewire-pulse pkg-config python3 python3-setuptools rsync unzip wget xz-utils
       shell: bash
 
     # Chromium Clang to be used with the clang toolchain file. The JNI code

docs/guide/build.md

@@ -29,9 +29,11 @@ git submodule update --init --depth 1 webrtc-media/third-party/ffmpeg webrtc-med
 Linux builds run on an x86-64 host and cross-compile for ARM. Install the build tools (Debian/Ubuntu package names):
 
 ```shell
-sudo apt-get install binutils cmake git make meson nasm ninja-build pkg-config python3 python3-setuptools rsync unzip wget xz-utils
+sudo apt-get install binutils cmake git libva-dev make meson nasm ninja-build pkg-config python3 python3-setuptools rsync unzip wget xz-utils
 ```
 
+`libva-dev` provides the VA-API headers FFmpeg is built against, for every target architecture, cross builds included. Only the headers are used: FFmpeg is linked against a small loader in `webrtc-media/src/main/cpp/dependencies/libva` instead of libva itself, so the libraries load on machines without libva. Set `-DLIBVA_INCLUDE_DIR=<dir>` (the directory holding `va/`) for headers somewhere other than `/usr/include` or `/usr/local/include`.
+
 The JNI library compiles against the libc++ headers of the WebRTC build. Those headers only support the Clang that the WebRTC branch pins, so the toolchain files expect that exact Clang in `/opt/clang`. Install it this way (the package name is in `.github/actions/prepare-linux/action.yml` and changes with the WebRTC branch):
 
 ```shell

docs/guide/media/media-files.md

@@ -224,7 +224,7 @@ The pictures are the same ones software decoding gives; in the tests, every fram
 
 **HEVC is decoded in hardware only.** A player decodes it on the media engine or GPU whether or not the flag is set, and never in software: `isHardwareDecoding()` is always `true` for it. Where the machine has no hardware decoder for HEVC, creating the player fails with an `IOException`; where the hardware does not take a stream (a profile it cannot decode) or fails while playing, playback stops with an error. The fallback below does not apply to it.
 
-- **Platforms:** macOS through VideoToolbox, Windows through Direct3D 12 (Windows 10 version 2004 or later, with a driver that decodes video through it) and otherwise Direct3D 11, and Linux (x86-64 and ARM64) through NVDEC on NVIDIA GPUs (the driver provides `libcuda.so.1` and `libnvcuvid.so.1`) and through Vulkan Video on the GPUs of Intel and AMD (the Vulkan loader `libvulkan.so.1` and a driver with the video extensions of the codec: Mesa's ANV or RADV, recent enough for each codec, AV1 and VP9 needing the newest). These libraries are loaded when a player asks for hardware, so a machine without them is not affected. The flag is accepted everywhere; where the platform has no hardware decoder, or the FFmpeg build has none for it (32-bit ARM), the video is decoded in software, except HEVC, and `isHardwareDecoding()` says so.
+- **Platforms:** macOS through VideoToolbox, Windows through Direct3D 12 (Windows 10 version 2004 or later, with a driver that decodes video through it) and otherwise Direct3D 11, and Linux (x86-64 and ARM64) through NVDEC on NVIDIA GPUs (the driver provides `libcuda.so.1` and `libnvcuvid.so.1`) and on the GPUs of Intel and AMD through VA-API (`libva.so.2` and `libva-drm.so.2` with a driver: Intel's `iHD` from intel-media-driver, or Mesa's `radeonsi`) and otherwise Vulkan Video (the Vulkan loader `libvulkan.so.1` and a driver with the video extensions of the codec: Mesa's ANV or RADV, recent enough for each codec, AV1 and VP9 needing the newest, which some distributions ship switched off). VA-API comes first because a Vulkan device can be created where its driver does not decode, and the player picks a device by whether it can be created. These libraries are loaded when a player asks for hardware, so a machine without them is not affected. The flag is accepted everywhere; where the platform has no hardware decoder, or the FFmpeg build has none for it (32-bit ARM), the video is decoded in software, except HEVC, and `isHardwareDecoding()` says so.
 - **Codecs:** H.264, H.265/HEVC, VP9 and AV1, where the GPU decodes them: a GPU that is a few years old may lack AV1, and on a Mac only an M3 or later has it. VP8, MPEG-4 and MJPEG are decoded in software, and so is AV1 where there is no GPU decoder for it (it is decoded by dav1d, which is built into the FFmpeg libraries).
 - **Fallback** (every codec but HEVC): a stream the hardware does not take, such as a profile it cannot decode, is decoded in software without the player noticing more than `isHardwareDecoding()` turning `false`. If the hardware fails before its first picture, the packets it was sent are decoded again in software and none is lost; if it fails after one, decoding goes on in software from the next key frame. A damaged packet is not taken for a hardware failure after the first picture: as with software decoding, playback stops with an error, and the hardware keeps the stream.
 - **What it saves** is processor time, not the copy: a decoded picture is read back from the media engine into system memory, and converted to I420 as WebRTC wants it. On an Apple M2, a 1080p H.264 stream took 11.6 ms of processor time per frame in software and 1.4 ms in hardware, and a 4K stream 35.7 ms against 4.2 ms, about seven to eight times less. Per frame, software on all cores is faster on the clock, which does not matter at playback speed. Windows gains less: on an NVIDIA GeForce RTX 3080 through Direct3D 12, a 1080p H.264 stream took about 2.1 ms of processor time per frame against 3.6 ms in software, and a 4K stream 5.4 ms against 6.6 ms. Direct3D 11 reads the picture back more slowly and took more than software at 4K (12.4 ms), which is why Direct3D 12 comes first. These are figures for one machine each and for synthetic media, not a promise; try it on your own streams before relying on it.

webrtc-media/src/main/cpp/dependencies/ffmpeg/CMakeLists.txt

@@ -215,6 +215,24 @@ elseif(LINUX AND CMAKE_SYSTEM_PROCESSOR MATCHES "^(x86_64|AMD64|amd64|aarch64|ar
         --enable-hwaccel=vp9_vulkan
         --enable-hwaccel=av1_vulkan
     )
+    # VA-API, the other way the GPUs of Intel and AMD decode, through the
+    # drivers distributions enable by default (Intel's iHD, Mesa's radeonsi),
+    # where Vulkan Video may still be switched off. FFmpeg links libva and
+    # libva-drm directly rather than loading them, so it is given a stand-in
+    # for them instead, dependencies/libva/va_loader.c, which loads libva with
+    # dlopen on its first call: the libraries load on a machine without libva
+    # as they do without CUDA or Vulkan. The build needs the libva headers of
+    # the host (libva-dev), found below. The display is opened on a DRM render
+    # node; neither X11 nor libdrm is needed.
+    set(FFMPEG_VAAPI TRUE)
+
+    list(APPEND FFMPEG_COMPONENTS
+        --enable-vaapi
+        --enable-hwaccel=h264_vaapi
+        --enable-hwaccel=hevc_vaapi
+        --enable-hwaccel=vp9_vaapi
+        --enable-hwaccel=av1_vaapi
+    )
 endif()
 
 # The parent adds the licenses of the NVDEC and Vulkan headers to the platform
@@ -250,6 +268,50 @@ foreach(PATCH ${FFMPEG_PATCHES})
     string(APPEND FFMPEG_COMPONENTS_TEXT "\npatch ${PATCH_NAME} ${PATCH_HASH}")
 endforeach()
 
+#
+# The libva headers, and the stand-in for libva that FFmpeg is linked against.
+# Both belong to the install: another version of the headers changes what
+# configure finds in them, and the stand-in is linked into the libraries.
+#
+if(FFMPEG_VAAPI)
+    # The host's headers, from libva-dev, and only those: a sysroot may have
+    # headers of its own, older ones, and which of them a search turned up
+    # would depend on the toolchain. They are the same for every architecture.
+    find_path(LIBVA_INCLUDE_DIR
+        NAMES va/va.h va/va_drm.h
+        PATHS /usr/include /usr/local/include
+        NO_DEFAULT_PATH
+        NO_CMAKE_FIND_ROOT_PATH)
+
+    if(NOT LIBVA_INCLUDE_DIR)
+        message(FATAL_ERROR
+            "No libva headers (va/va.h, va/va_drm.h), which FFmpeg's VA-API decoders are built "
+            "against. Install them, for example with: sudo apt-get install libva-dev, or set "
+            "LIBVA_INCLUDE_DIR to the directory that holds va/.")
+    endif()
+
+    file(STRINGS "${LIBVA_INCLUDE_DIR}/va/va_version.h" LIBVA_VERSION_LINES
+        REGEX "^#define VA_(MAJOR|MINOR|MICRO)_VERSION +[0-9]+")
+
+    foreach(PART MAJOR MINOR MICRO)
+        if(LIBVA_VERSION_LINES MATCHES "VA_${PART}_VERSION +([0-9]+)")
+            set(LIBVA_${PART} "${CMAKE_MATCH_1}")
+        else()
+            message(FATAL_ERROR "No VA_${PART}_VERSION in ${LIBVA_INCLUDE_DIR}/va/va_version.h")
+        endif()
+    endforeach()
+
+    # The version of the API, which is what libva.pc gives as its version and
+    # what configure compares against (libva >= 0.35.0).
+    set(LIBVA_API_VERSION "${LIBVA_MAJOR}.${LIBVA_MINOR}.${LIBVA_MICRO}")
+
+    get_filename_component(LIBVA_LOADER_DIR "${CMAKE_CURRENT_SOURCE_DIR}/../libva" ABSOLUTE)
+    file(SHA256 "${LIBVA_LOADER_DIR}/va_loader.c" LIBVA_LOADER_HASH)
+
+    string(APPEND FFMPEG_COMPONENTS_TEXT
+        "\nlibva headers ${LIBVA_API_VERSION} loader ${LIBVA_LOADER_HASH}")
+endif()
+
 #
 # dav1d is the software decoder of AV1, built from the third-party/dav1d
 # submodule with Meson into a prefix of its own, as a static library, which
@@ -410,6 +472,52 @@ function(ffmpeg_is_installed RESULT)
     endif()
 endfunction()
 
+#
+# Checks that the libraries just built take libva from the stand-in: none of
+# them may name libva as a dependency, which a pkg-config file of the host
+# found ahead of ours would make them do, and none may leave a function of it
+# undefined, which a function FFmpeg calls and the stand-in lacks would.
+# Either way they would not load without libva, or would fail on the call.
+#
+function(ffmpeg_check_libva_loaded_at_run_time)
+    find_program(READELF_EXECUTABLE NAMES ${CMAKE_READELF} readelf)
+
+    if(NOT READELF_EXECUTABLE)
+        message(WARNING "No readelf, so the FFmpeg libraries are not checked for a dependency on libva")
+        return()
+    endif()
+
+    file(GLOB INSTALLED_LIBS "${FFMPEG_INSTALL_DIR}/lib/*.so.*")
+
+    foreach(LIB ${INSTALLED_LIBS})
+        if(IS_SYMLINK "${LIB}")
+            continue()
+        endif()
+
+        execute_process(
+            COMMAND "${READELF_EXECUTABLE}" --dynamic --dyn-syms --wide "${LIB}"
+            OUTPUT_VARIABLE ELF_INFO
+            RESULT_VARIABLE ELF_RESULT
+            ERROR_QUIET)
+
+        if(NOT ELF_RESULT EQUAL 0)
+            message(FATAL_ERROR "readelf could not read ${LIB}")
+        endif()
+
+        if(ELF_INFO MATCHES "Shared library: \\[(libva[-.a-z0-9]*)\\]")
+            message(FATAL_ERROR
+                "${LIB} depends on ${CMAKE_MATCH_1}: FFmpeg was linked against the libva of the "
+                "host instead of the stand-in in ${LIBVA_LOADER_DIR}.")
+        endif()
+
+        if(ELF_INFO MATCHES "UND (va[A-Z][A-Za-z0-9_]*)")
+            message(FATAL_ERROR
+                "${LIB} leaves ${CMAKE_MATCH_1} undefined, a function of libva that FFmpeg calls "
+                "and ${LIBVA_LOADER_DIR}/va_loader.c does not define. Add it there.")
+        endif()
+    endforeach()
+endfunction()
+
 ffmpeg_is_installed(FFMPEG_INSTALLED)
 
 if(FFMPEG_INSTALLED)
@@ -894,16 +1002,16 @@ fi
         set(FFMPEG_MAKE "${MAKE_EXECUTABLE}")
     endif()
 
-    if(FFMPEG_NVDEC OR FFMPEG_VULKAN)
-        # configure looks for the NVDEC and Vulkan headers through pkg-config,
-        # so that needs to be there, and to know where the vendored headers
+    if(FFMPEG_NVDEC OR FFMPEG_VULKAN OR FFMPEG_VAAPI)
+        # configure looks for the NVDEC, Vulkan and VA-API headers through
+        # pkg-config, so that needs to be there, and to know where the headers
         # are.
         find_program(PKG_CONFIG_EXECUTABLE NAMES pkg-config pkgconf)
 
         if(NOT PKG_CONFIG_EXECUTABLE)
             message(FATAL_ERROR
-                "No pkg-config, which FFmpeg's configure uses to find the NVDEC and Vulkan headers. "
-                "Install it, for example with: sudo apt-get install pkg-config")
+                "No pkg-config, which FFmpeg's configure uses to find the NVDEC, Vulkan and VA-API "
+                "headers. Install it, for example with: sudo apt-get install pkg-config")
         endif()
 
         set(HEADERS_PC_DIR "${CMAKE_CURRENT_BINARY_DIR}/headers-pc")
@@ -918,6 +1026,21 @@ fi
 
             configure_file("${VULKAN_HEADERS_DIR}/vulkan.pc.in" "${HEADERS_PC_DIR}/vulkan.pc" @ONLY)
         endif()
+        if(FFMPEG_VAAPI)
+            # Only the va/ headers, copied, so that the include directory
+            # configure is given holds nothing else of the host's: on a cross
+            # build, -I/usr/include would put the host's C library in front
+            # of the target's.
+            set(LIBVA_STAGED_INCLUDE_DIR "${CMAKE_CURRENT_BINARY_DIR}/libva-include")
+            set(LIBVA_LOADER_LIB_DIR "${CMAKE_CURRENT_BINARY_DIR}/libva-loader")
+
+            file(REMOVE_RECURSE "${LIBVA_STAGED_INCLUDE_DIR}" "${LIBVA_LOADER_LIB_DIR}")
+            file(COPY "${LIBVA_INCLUDE_DIR}/va" DESTINATION "${LIBVA_STAGED_INCLUDE_DIR}")
+            file(MAKE_DIRECTORY "${LIBVA_LOADER_LIB_DIR}")
+
+            configure_file("${LIBVA_LOADER_DIR}/libva.pc.in" "${HEADERS_PC_DIR}/libva.pc" @ONLY)
+            configure_file("${LIBVA_LOADER_DIR}/libva-drm.pc.in" "${HEADERS_PC_DIR}/libva-drm.pc" @ONLY)
+        endif()
 
         # Only our directory: a sysroot or library path of the host would put
         # the includedir somewhere it is not, and the host's own vulkan.pc
@@ -928,12 +1051,37 @@ unset PKG_CONFIG_SYSROOT_DIR PKG_CONFIG_LIBDIR
 ")
     endif()
 
+    # The stand-in for libva is built for the target as FFmpeg is, with the
+    # same compiler and flags, into the static library libva.pc names.
+    # Position-independent, as it goes into shared libraries, and hidden, so
+    # that they do not export it.
+    set(LIBVA_LOADER_STEPS "")
+
+    if(FFMPEG_VAAPI)
+        set(LIBVA_LOADER_CFLAGS "${FFMPEG_EXTRA_CFLAGS}")
+
+        if(CMAKE_SYSROOT)
+            string(APPEND LIBVA_LOADER_CFLAGS " --sysroot=${CMAKE_SYSROOT}")
+        endif()
+
+        if(CMAKE_AR)
+            set(LIBVA_LOADER_AR "${CMAKE_AR}")
+        else()
+            set(LIBVA_LOADER_AR ar)
+        endif()
+
+        set(LIBVA_LOADER_STEPS
+"\"${CMAKE_C_COMPILER}\" ${LIBVA_LOADER_CFLAGS} -O2 -fPIC -fvisibility=hidden -I\"${LIBVA_STAGED_INCLUDE_DIR}\" -c \"${LIBVA_LOADER_DIR}/va_loader.c\" -o \"${LIBVA_LOADER_LIB_DIR}/va_loader.o\"
+\"${LIBVA_LOADER_AR}\" rcs \"${LIBVA_LOADER_LIB_DIR}/libva-loader.a\" \"${LIBVA_LOADER_LIB_DIR}/va_loader.o\"
+")
+    endif()
+
     set(FFMPEG_BUILD_SCRIPT "${CMAKE_CURRENT_BINARY_DIR}/build-ffmpeg.sh")
 
     file(WRITE "${FFMPEG_BUILD_SCRIPT}"
 "# Generated by CMake. Builds FFmpeg ${FFMPEG_VERSION} for webrtc-java-media.
 set -e
-${FFMPEG_SHELL_PREAMBLE}${DAV1D_BUILD_STEPS}cd \"${FFMPEG_BUILD_DIR_SH}\"
+${FFMPEG_SHELL_PREAMBLE}${DAV1D_BUILD_STEPS}${LIBVA_LOADER_STEPS}cd \"${FFMPEG_BUILD_DIR_SH}\"
 # FFmpeg names its version after the git tag it finds, and a shallow clone of
 # the submodule has none, so it would name itself after a commit hash. Handing
 # make the release the source says it is names it the same way however the
@@ -1016,6 +1164,10 @@ set MSYS2_PATH_TYPE=inherit
             "${FFMPEG_BUILD_SCRIPT}.")
     endif()
 
+    if(FFMPEG_VAAPI)
+        ffmpeg_check_libva_loaded_at_run_time()
+    endif()
+
     file(WRITE "${FFMPEG_COMPONENTS_FILE}" "${FFMPEG_COMPONENTS_TEXT}")
 
     ffmpeg_is_installed(FFMPEG_INSTALLED)

webrtc-media/src/main/cpp/dependencies/ffmpeg/FFmpeg-NOTICE.txt.in

@@ -26,6 +26,16 @@ META-INF/licenses/dav1d/COPYING of this jar. It is built from the release tag
   https://code.videolan.org/videolan/dav1d/-/tree/@DAV1D_VERSION@
 
 
+libva loader
+------------
+
+In the Linux platform jars for x86-64 and ARM64, libavutil and libavcodec
+contain a small loader for libva (VA-API), part of webrtc-java and under its
+license, linked in statically in place of libva itself. It loads the libva
+installed on the machine, if there is one, when a player first asks for VA-API.
+libva is not part of this jar, and the libraries do not need it to load.
+
+
 Source code
 -----------
 
@@ -57,7 +67,7 @@ HEVC/H.265
 Disclaimer regarding HEVC/H.265:
 This project decodes HEVC/H.265 video through hardware acceleration only, using
 the hardware decoders of the operating system or GPU (VideoToolbox, Direct3D
-11/12, NVDEC, Vulkan Video). It does not include, bundle, or distribute any
+11/12, NVDEC, VA-API, Vulkan Video). It does not include, bundle, or distribute any
 software-based HEVC encoder (such as x265), and it never decodes HEVC in
 software. The FFmpeg libraries in the webrtc-java-media platform jars that
 decode HEVC in hardware (all but 32-bit ARM Linux) contain FFmpeg's HEVC