Skip to content

feat: Real-time AI background blur (portrait bokeh segmentation) #1

Description

@prostopasta

Problem Description

The previous uniform CSS blur filter applied a blur across the entire <video> element, blurring both the speaker and their surroundings. Additionally, WebRTC's native backgroundBlur constraint requires OS- or hardware-level camera driver support (e.g., macOS Apple Silicon / Windows 11 Studio Effects) which is unavailable on Linux and most standard USB webcams. Furthermore, DOM CSS filters do not apply to the raw camera stream captured by MediaRecorder.

Objective

Implement true virtual background blur (bokeh/portrait mode) where the speaker remains crisp and in focus while only the background/surroundings are blurred, with support for both the live preview and the recorded video export.


Architectural Options to Choose From

Option 1: Client-Side AI Segmentation via Google MediaPipe (Recommended for Web)

  • Technology: Google MediaPipe Selfie Segmentation (@mediapipe/selfie_segmentation or @mediapipe/tasks-vision).
  • Pipeline:
    1. Webcam feed frames are processed via WebAssembly / WebGL.
    2. The model outputs a human silhouette alpha mask at ~30 FPS.
    3. An off-screen <canvas> composites the feathered mask: blurring background pixels and layering the sharp person in front.
    4. canvas.captureStream() is fed into MediaRecorder so the exported video take includes the bokeh effect.
  • Pros: Runs 100% client-side in the browser, completely private (0 bytes sent to any server), high accuracy and frame rate.
  • Trade-offs:
    • Requires fetching the lightweight model weights (~3–4 MB).
    • Can be loaded via Progressive Enhancement (dynamically loading scripts only when AI Blur is toggled on) to keep the core teleprompter 0-dependency and offline by default.

Option 2: WebCodecs + ONNX Runtime Web / Transformers.js

  • Technology: Run an open-source segmentation model (e.g. MobileNet / DeepLabV3) through ONNX Runtime Web with WebGPU acceleration.
  • Pros: Flexible model selection, potential WebGPU acceleration.
  • Trade-offs: Larger download footprints and higher complexity compared to MediaPipe.

Option 3: External Virtual Camera Documentation (OBS Studio / v4l2loopback)

  • Technology: Documenting workflows for users to use OBS Studio (obs-backgroundremoval plugin) or dedicated webcam software (Camo, Elgato) and select the virtual camera device in the teleprompter.
  • Pros: 0 additional lines of JavaScript, 0 client-side bundle size increase, hardware accelerated with zero browser overhead.
  • Trade-offs: Requires user to run external software.

Key Technical Considerations

  1. Repository Philosophy: Maintain the zero-dependency, 100% offline nature of the base prompter by lazy-loading any ML segmentation scripts on-demand.
  2. Performance: Run segmentation at a lower resolution (e.g. 256x256 or 512x512) and upscale/feather onto the 1080p canvas to maintain smooth 30 FPS without overheating or dropping frames.
  3. Canvas Recording: Ensure the recorded MediaStream originates from the composite canvas rather than the raw getUserMedia stream when portrait blur is active.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions