Summary
Hardware transcoding fails with "Invalid argument" inside Plex's own internal codec probe, even after installing this driver and confirming it works when the Plex Transcoder binary is invoked manually with explicit VAAPI flags. The actual live transcode process (spawned automatically by Plex) never receives any -hwaccel/-init_hw_device flags at all — it falls straight to software (libx264) without attempting hardware.
Environment
- Host: Proxmox VE, Debian-based unprivileged LXC (unprivileged: 1) running a Docker host, with the linuxserver/plex container running inside that LXC (Docker-in-LXC)
- CPU/iGPU: Intel Core i5-10500T (Comet Lake), UHD Graphics 630
- Plex image: lscr.io/linuxserver/plex:latest
- plex-vaapi-driver: ghcr.io/thadamski/plex-vaapi-driver:latest, deployed via Docker Compose as an init-style service that copies driver files into a shared named volume, mounted into the Plex container at /vaapi
What's confirmed working
- Device passthrough is correct end-to-end: /dev/dri/renderD128 and /dev/dri/card1 are visible with correct permissions (group video, GID 44, consistent across the Proxmox host, the LXC, and the container)
- Standalone vainfo (using the system intel-media-va-driver package, unrelated to this project) succeeds cleanly both on the LXC host and inside the Plex container, returning the full expected H.264/HEVC/VP9 profile list
- LIBVA_DRIVER_NAME=iHD and LIBVA_DRIVERS_PATH=/vaapi are correctly set in the container environment and confirmed present (via /proc//environ) in both the main Plex Media Server process and the Plex Transcoder process (the latter required a wrapper script — see below, since Plex's launcher does not appear to reliably pass container env vars through to the transcoder subprocess in our testing)
- Manually invoking the transcoder binary directly succeeds:
LIBVA_DRIVER_NAME=iHD LIBVA_DRIVERS_PATH=/vaapi "/usr/lib/plexmediaserver/Plex Transcoder.real" \
-init_hw_device vaapi=va:/dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi \
-v verbose -i "<real movie file, HEVC 10-bit source>" -frames:v 5 -f null -
Output includes:
[AVHWDeviceContext] libva: va_openDriver() returns 0
[AVHWDeviceContext] Initialised VAAPI connection: version 1.22
[AVHWDeviceContext] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 26.1.6 (469cbbd).
This confirms the driver itself is fully functional against this exact hardware/kernel/container stack when invoked directly.
What's failing
When Plex spawns a transcode automatically (forcing a lower-quality stream via a client to trigger it), Plex Media Server's internal codec/hardware capability probe fails before any transcoder subprocess with hardware flags is ever spawned:
Codecs: testing hevc (decoder) with hwdevice vaapi
Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD128' (Intel CometLake-S GT2 [UHD Graphics 630])
Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: Invalid argument
Could not create hardware context for hevc
...
TPU: hardware transcoding: enabled, but no hardware decode accelerator found
TPU: hardware transcoding: final decoder: , final encoder:
This happens identically for H.264 and HEVC sources, and persisted:
- Across multiple docker compose up -d --force-recreate cycles
- After a plain docker restart (to rule out a stale in-memory capability cache from before the driver/env vars were in place)
- With and without a wrapper script forcing LIBVA_DRIVER_NAME/LIBVA_DRIVERS_PATH directly onto the Plex Transcoder binary (renamed original to .real, replaced with a script that exports the vars then execs the real binary — confirmed via /proc//environ that this wrapper's exported vars were present in the live process)
The actual live transcode process, once spawned, shows no hardware-related flags whatsoever in its command line — it goes straight to -codec:0 libx264 software encoding without ever attempting -hwaccel vaapi or -init_hw_device. This suggests Plex's main server process does its own separate, early internal capability check (distinct from the transcoder subprocess itself) that determines hardware won't be used, and that check is failing even though direct invocation of the same driver against the same device succeeds.
Question / request
Is Plex Media Server's own internal capability probe known to behave differently from a direct Plex Transcoder invocation — e.g., different init flags, a different code path, or something else that this project's compatibility fixes don't cover? Any pointers on how to trace what parameters that internal probe uses (vs. what we tested manually) would be hugely appreciated, since we've been unable to find a way to observe that specific call directly.
Happy to provide any additional logs, ps//proc output, or test additional driver/env configurations — we've been fairly deep in this already and have verbose logs available from most of the steps above.
Summary
Hardware transcoding fails with "Invalid argument" inside Plex's own internal codec probe, even after installing this driver and confirming it works when the Plex Transcoder binary is invoked manually with explicit VAAPI flags. The actual live transcode process (spawned automatically by Plex) never receives any -hwaccel/-init_hw_device flags at all — it falls straight to software (libx264) without attempting hardware.
Environment
What's confirmed working
Output includes:
This confirms the driver itself is fully functional against this exact hardware/kernel/container stack when invoked directly.
What's failing
When Plex spawns a transcode automatically (forcing a lower-quality stream via a client to trigger it), Plex Media Server's internal codec/hardware capability probe fails before any transcoder subprocess with hardware flags is ever spawned:
This happens identically for H.264 and HEVC sources, and persisted:
The actual live transcode process, once spawned, shows no hardware-related flags whatsoever in its command line — it goes straight to -codec:0 libx264 software encoding without ever attempting -hwaccel vaapi or -init_hw_device. This suggests Plex's main server process does its own separate, early internal capability check (distinct from the transcoder subprocess itself) that determines hardware won't be used, and that check is failing even though direct invocation of the same driver against the same device succeeds.
Question / request
Is Plex Media Server's own internal capability probe known to behave differently from a direct Plex Transcoder invocation — e.g., different init flags, a different code path, or something else that this project's compatibility fixes don't cover? Any pointers on how to trace what parameters that internal probe uses (vs. what we tested manually) would be hugely appreciated, since we've been unable to find a way to observe that specific call directly.
Happy to provide any additional logs, ps//proc output, or test additional driver/env configurations — we've been fairly deep in this already and have verbose logs available from most of the steps above.