feat(export): pick the video encoder from the host's hardware, with a libx264 fallback

[EtienneLescot]

The problem

crates/compositor/src/pipeline.rs hard-coded the AMD encoder in three places and refused to continue without it:

let enc_name = CString::new("h264_amf")?;
...
bail!("h264_amf introuvable");

AMF is AMD-only. On an NVIDIA or Intel host, or on any machine without a hardware H.264 encoder, that path could not open an encoder at all — native export failed outright rather than degrading. Development happens on an AMD GPU, which is exactly why this stayed invisible.

What shipped

ExportCodec::candidates() returns an ordered list of (encoder name, input pixel format). VideoEncoder::open keeps the first one avcodec_open2 actually accepts, logs which it kept and which it refused with their errors, and VideoEncoder::send hides the difference: D3D11 candidates take the compositor's texture unchanged, the rest get an av_hwframe_transfer_data download plus an NV12 → YUV420P de-interleave for the software encoders. All three call sites route through it, C0/C1 bench paths included — they carried the identical bail.

	H264	H265	input
AMD	h264_amf	hevc_amf	D3D11, zero-copy
NVIDIA	h264_nvenc	hevc_nvenc	D3D11, zero-copy
Intel	h264_qsv	hevc_qsv	NV12 system
any MFT	h264_mf	hevc_mf	NV12 system
software	libopenh264	libkvazaar	YUV420P system

Selection is automatic and derived from the host. There is no user-facing selector.

Three things the scoping issue got wrong

• libx264 is not in the shipped ffmpeg. The vendored build is LGPL (--disable-libx264 --disable-libx265). The software encoders that do exist are libopenh264 and libkvazaar, so those are the last resort. A test asserts it, so the fallback cannot quietly become a name that is not there.
• electron/media/ffmpegCapabilities.ts is gone, deleted in 9d6e7e1e with the web MP4 pipeline. There is no TS-side discovery left to consult; this list is the only one.
• h264_mf is listed NV12-only. It advertises d3d11 and opens in d3d11, then dies on the first send with Failed to set D3D manager: 80004001 once MediaFoundation resolves the software MFT. Since the choice is committed at open time, that form would fail the export instead of sliding to the next candidate.

Testing the paths this machine does not have

OPENSCREEN_EXPORT_ENCODER=<name> restricts the search to a single candidate — the only way to exercise the non-AMD paths from an AMD host without editing the source and rebuilding, and the only form that works against a packaged build or a user's machine. It deliberately falls back to nothing: a silent slide to AMF would make a check look like it passed. Follows OPENSCREEN_WGC_PREFER_SOFTWARE_ENCODER, which does the same for capture.

Acceptance

• Export succeeds with no AMF encoder. C0 on the frozen fixture at 1920x1080, forced to libopenh264: 360 frames, 54.7 fps, valid MP4. Unforced on the same machine: h264_amf, 235.5 fps, unregressed.
• The chosen encoder is observable. [pipeline] encodeur vidéo : libopenh264 (frames système), with the refused candidates and their errors appended when there were any.
• Colour is correct on the software path. PSNR between the forced and the AMF output: y 38.1 / u 41.8 / v 42.1. Chroma above luma is what a correct conversion looks like — a U/V swap lands near 15 dB.
• Unit coverage for the ordering, for the fallback names being ones the LGPL build actually has, and for the chroma de-interleave at both even and odd sizes. Both the ordering and the chroma tests were confirmed to fail on a deliberately broken version. 76/76 crate tests pass.

AV_CODEC_FLAG_GLOBAL_HEADER is set on every encoder context — all three sites mux MP4, and unlike AMF the software encoders emit no extradata without it. ffprobe reports extradata_size=30 on both outputs.

A failed run now deletes its own output file instead of leaving a moov-less MP4 under the name the user believes they exported.

Out of scope

• The preferSoftwareEncoder override for export — separate PR.
• WARP fallback for D3D device creation — separate PR.
• Probing with the first real composed frame rather than with avcodec_open2. Opening is a sufficient probe for every candidate that remains, but it is not a strong one; try_open carries the reasoning and the upgrade path.
• Rust has no CI job at all — cargo is never invoked by any workflow, so these tests only run locally.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ae3da82e-3168-4166-9e6f-30c755a412e8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/native-encoder-vendor-selection

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}