Headless Window + pipelined (non-blocking) pixel readback - #398
Merged
sebcrozet merged 1 commit intoJul 16, 2026
Merged
Conversation
haixuanTao
added a commit
to haixuanTao/nexus
that referenced
this pull request
Jul 8, 2026
…ewer, pipelined readback Makes the nexus3d Python bindings usable for off-screen video capture: - NexusViewer(width, height): configurable resolution; set_draw_ui(False) keeps the egui panel out of captured frames. - NexusViewer.render() -> (H, W, 3) uint8 numpy frame export, like mujoco.Renderer.render(). - raytrace_frame() + set_raytracer_samples_per_frame/max_bounces/denoise + raytracer_backend(): kiss3d's GPU path tracer from Python, with progressive sample accumulation. sync takes the CPU-readback path while the tracer is active (the zero-readback kernel only updates the rasterizer's instance buffers, which the tracer's BVH never reads). - NexusViewer(..., headless=True): no OS window, no swapchain — frames render into an off-screen texture, never throttled by display vsync, works without a display server. - set_vsync(enabled)/vsync(): uncap a windowed viewer instead. - render_async()/render_flush(): pipelined capture — returns the previous frame while the current frame's GPU->CPU copy runs in the background. A windowed capture loop was vsync-locked at ~30 fps (blocking readback + Fifo swapchain = ~2 vblanks per iteration); headless + pipelined readback reaches 92 gen-fps at 640x480 (GPU physics) on an RTX 5080. Requires the kiss3d headless-capture branch (dimforge/kiss3d#398). Supersedes dimforge#7 and dimforge#8 (included here).
haixuanTao
force-pushed
the
feat/headless-capture-pipeline
branch
from
July 8, 2026 19:56
4440e6b to
7003b7d
Compare
haixuanTao
changed the title
Two additions for off-screen video capture: - Window::new_headless_with_setup(width, height, setup): a full-featured Window backed by no OS window and no swapchain (same headless canvas as OffscreenSurface, but exposing the whole Window API — custom renderers, ray tracer, snap* readbacks). Rendering is never throttled by the display's vsync and works without a display server. - snap_begin()/snap_finish() (begin_read_pixels/finish_read_pixels on the canvas): pipelined readback that enqueues the texture->buffer copy and buffer map without waiting, so the copy of frame N can be collected after frame N+1 renders instead of stalling the GPU pipeline every frame the way the blocking snap()/read_pixels() does. read_pixels() is refactored to begin+finish with unchanged behavior. Motivation: a windowed capture loop (render_frame + blocking snap) is vsync-locked — the blocking readback defeats swapchain frame pipelining, so every iteration eats ~2 vblanks (~30 fps at 60 Hz) regardless of GPU load. Headless + pipelined readback took a downstream nexus3d capture loop from 33 to 92 fps at 640x480. Builds on feat/persistent-readback-staging (dimforge#397).
sebcrozet
force-pushed
the
feat/headless-capture-pipeline
branch
from
July 16, 2026 11:27
7003b7d to
1e23ce7
Compare
sebcrozet
approved these changes
Jul 16, 2026
Comment on lines
+96
to
+98
| let img = ImageBuffer::from_vec(width, height, buf) | ||
| .expect("readback buffer was not big enough for image"); | ||
| Some(imageops::flip_vertical(&img)) |
Member
There was a problem hiding this comment.
I’m curious if we could avoid flipping the image and if we could cache img or take a &mut ImageBuffer argument to avoid CPU-side allocation of the image buffer.
Not blocking for this PR though.
Member
|
Thanks! |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #397 — merge #397 first; this PR then reduces to its single commit (
feat: public headless Window + pipelined (non-blocking) pixel readback). Until then the diff below includes #397's read_pixels changes.Two additions for off-screen video capture:
Window::new_headless_with_setup(width, height, setup): a full-featuredWindowbacked by no OS window and no swapchain — the same headless canvas asOffscreenSurface, but exposing the wholeWindowAPI (custom renderers, ray tracer,snap*readbacks). Never throttled by display vsync; works without a display server.snap_begin()/snap_finish(): pipelined readback that enqueues the texture→buffer copy and buffer map without waiting, so frame N's pixels are collected after frame N+1 renders instead of stalling the GPU pipeline every frame.read_pixels()is refactored onto the same begin/finish path with unchanged behavior.Motivation: a windowed capture loop (
render_frame+ blockingsnap) is vsync-locked — the swapchain defaults toAutoVsync(Fifo) and the blocking readback defeats frame pipelining, so every iteration eats ~2 vblanks (~30 fps at 60 Hz) regardless of GPU load. Headless + pipelined readback took a downstream nexus3d capture loop from 33 to 92 gen-fps at 640×480.🤖 Generated with Claude Code