Skip to content

feat(core): add GPU texture variant to VideoFrame for zero-copy pipeline stages #350

@staging-devin-ai-integration

Description

Summary

Extend VideoFrame / VideoData to optionally carry a wgpu::Texture instead of CPU pixel data, enabling zero-copy handoff between GPU pipeline stages (compositor → encoder, decoder → compositor, web renderer → compositor, etc.).

This is a foundational change that benefits the entire pipeline, not just the web renderer. The Vulkan Video encoder already notes GPU texture input as a future goal.

Parent issue: #167 (web renderer context), but this is an independent core improvement
Enables: #349 (DMA-BUF zero-copy), future HW encoder GPU texture input

Architecture

// In crates/core/src/types.rs (or similar)
pub enum VideoData {
    /// CPU-side pixel buffer (current behavior).
    Cpu(PooledVideoData),
    /// GPU-resident texture — zero-copy between GPU stages.
    Gpu(GpuVideoData),
}

pub struct GpuVideoData {
    pub texture: wgpu::Texture,
    pub view: wgpu::TextureView,
    pub width: u32,
    pub height: u32,
    pub format: wgpu::TextureFormat,
    // Device reference for readback if needed
    pub device: Arc<wgpu::Device>,
    pub queue: Arc<wgpu::Queue>,
}

impl VideoData {
    /// Convert GPU data to CPU data (triggers readback).
    /// This is the backward-compatibility escape hatch.
    pub async fn to_cpu(&self) -> PooledVideoData { ... }

    /// Returns true if this is GPU-resident data.
    pub fn is_gpu(&self) -> bool { ... }
}

Implementation Steps

  1. Define VideoData enum in crates/core/ with Cpu and Gpu variants
  2. Update VideoFrame to use VideoData instead of raw PooledVideoData
  3. Add to_cpu() readback — nodes that only handle CPU data call this (backward compatible)
  4. Update compositor output — when the downstream node accepts GPU data, skip readback_rgba() and output VideoData::Gpu directly
  5. Update all existing nodes — most just call .to_cpu() if they receive GPU data (no behavior change)
  6. Benchmark — measure compositor → encoder pipeline with GPU passthrough vs. current CPU readback

Acceptance Criteria

  • VideoData enum with Cpu and Gpu variants in crates/core/
  • GpuVideoData carries: wgpu::Texture, TextureView, width, height, format, device/queue refs
  • VideoFrame uses VideoData instead of raw PooledVideoData
  • Compositor can output VideoData::Gpu (skip readback_rgba()) when downstream accepts it
  • .to_cpu() method for backward compatibility (triggers GPU readback on demand)
  • All existing tests pass (existing CPU paths unaffected)
  • New unit tests for VideoData enum, to_cpu() conversion
  • Benchmark: compositor output with GPU passthrough vs. current CPU readback

Complexity

Effort: L (1–2 weeks)
Touches core types used throughout the pipeline. Requires careful attention to wgpu device/queue lifetime management and backward compatibility.

Design Considerations

  • Device sharing: The compositor's wgpu::Device must be accessible to downstream nodes that want to use the GPU texture. This may require passing a device reference through the pipeline context or storing it in GpuVideoData.
  • Texture lifetime: wgpu textures are bound to their device. The texture must remain valid until the downstream node has consumed it. Consider using Arc<wgpu::Texture> or a pool.
  • Format negotiation: Nodes need a way to advertise whether they accept GPU data. This could be a new pin capability or an upstream hint.
  • Pool integration: PooledVideoData has an existing pool mechanism. GpuVideoData may need a similar texture pool to avoid per-frame allocation.

Notes

  • This issue is independent of the Servo plugin but was identified during the web renderer roadmap planning.
  • The Vulkan Video encoder (vulkan_video.rs) is a natural first consumer — it already operates on the same Vulkan device as wgpu.
  • Future HW encoder GPU texture input (NVENC via cuImportExternalMemory, VAAPI via DMA-BUF) would also consume VideoData::Gpu.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions