Skip to content

perf(video): accept GPU textures in HW encoders (Vulkan Video, NVENC, VAAPI) #351

@staging-devin-ai-integration

Description

Summary

Enable hardware video encoders to accept VideoData::Gpu directly from the compositor, completing the all-GPU pipeline path: decoder/compositor → encoder with zero CPU pixel touches.

Depends on: #350 (GPU VideoFrame variant)
Parent issue: #167 (web renderer context), but this is an independent pipeline improvement

Per-Encoder Work

Encoder File Interop Mechanism Effort
Vulkan Video vulkan_video.rs Same wgpu Vulkan device → native VkImage M
NVENC nv_av1.rs cuImportExternalMemory (CUDA↔Vulkan interop) L
VAAPI vaapi_h264.rs, vaapi_av1.rs Export wgpu texture → DMA-BUF → vaCreateSurfaceFromDmaBuf L

Recommended order

  1. Vulkan Video (most natural fit — same Vulkan device as wgpu, no cross-API interop)
  2. VAAPI (DMA-BUF is well-supported on Mesa)
  3. NVENC (CUDA↔Vulkan interop is more complex)

Architecture

Current path:
  Compositor (wgpu) → readback_rgba() → CPU Vec<u8> → encoder upload → encode

Proposed path:
  Compositor (wgpu) → VideoData::Gpu(wgpu::Texture)
    → encoder receives texture directly → encode (no CPU touch)

Acceptance Criteria

  • At least one encoder (Vulkan Video recommended) accepts VideoData::Gpu input
  • Encoder checks for GPU data and uses the texture directly when available
  • Fallback: if encoder doesn't support GPU input, auto-readback to CPU via to_cpu()
  • Benchmark: end-to-end compositor → encode latency with GPU path vs. CPU path
  • Document GPU requirements (Vulkan version, driver support)

Complexity

Effort: L (1–2 weeks per encoder)
Each encoder has different interop requirements. Vulkan Video is simplest since it shares the wgpu Vulkan device.

Notes

  • This is the final piece of the all-GPU pipeline: web renderer/decoder → compositor → encoder with zero CPU readback.
  • Can be implemented incrementally — start with Vulkan Video, add VAAPI and NVENC later.
  • The encoder should gracefully fall back to CPU readback if it receives VideoData::Cpu (backward compatible).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions