Real-time spatial augmented reality on NVIDIA Jetson Orin using a ZED stereo camera and an HDMI projector. The system captures 3D scene geometry, builds a per-pixel camera-to-projector mapping, and warps live imagery through that mapping so projected light conforms to physical surfaces.
- What This Tool Does
- Use Cases
- Quick Start
- Hardware Requirements
- Software Dependencies
- Project Structure
- Detailed Setup
- Calibration Guide
- Running the System
- Configuration Reference
- Component Documentation
- Performance Tuning
- Troubleshooting
- Architecture Deep Dive
This projection mapping system solves a fundamental problem: how do you project imagery onto arbitrary 3D surfaces so that it appears undistorted?
When you point a projector at a non-flat surface (a corner, a sculpture, furniture), the projected image appears warped and stretched. This system corrects for that distortion in real-time by:
-
Learning the geometry — During a one-time calibration, the system determines exactly which projector pixel illuminates which camera pixel by projecting structured light patterns.
-
Building a lookup table — The calibration produces a LUT (Look-Up Table) that maps every projector pixel to a camera coordinate.
-
Warping in real-time — At runtime, the GPU shader samples the camera image through this LUT, effectively "unwarping" the projection so it conforms to the physical surface.
-
Depth-aware masking — The ZED camera provides depth data, allowing the system to only project onto surfaces within a specified distance range (e.g., project on the wall but not on a person walking in front).
Unlike systems that require a 3D model of the scene, this approach works on arbitrary unknown geometry. The structured light calibration captures the actual projector-camera correspondence without needing to reconstruct the 3D surface.
| Application | Description |
|---|---|
| Augmented Reality Installations | Project interactive content onto physical objects, walls, or architectural features |
| Stage Design | Map visuals onto set pieces without pre-modeling the geometry |
| Prototyping | Visualize how graphics will look on physical products before manufacturing |
| Interactive Exhibits | Create responsive projections that adapt to the physical environment |
| Research | Study projector-camera systems, structured light, and real-time GPU processing |
# 1. Clone and setup
cd /home/null_0/zed_proj
./setup.sh
# 2. Verify hardware (optional but recommended)
python3 -m tests.test_zed # Check ZED camera
python3 -m tests.test_opengl # Check OpenGL rendering
# 3. Calibrate (one-time, requires projector)
# - Darken the room
# - Aim camera and projector at target surface
# - Keep everything still during capture
python3 -m calibration.graycode_capture
python3 -m calibration.lut_generator
# 4. Run
python3 main.pyFor testing without a projector:
python3 -m calibration.graycode_capture --windowed
python3 -m calibration.lut_generator
python3 main.py --windowed| Component | Specification | Notes |
|---|---|---|
| Compute | NVIDIA Jetson Orin 64GB | JetPack 6.x (Ubuntu 22.04, CUDA 12.x) |
| Camera | ZED Stereo Camera v1 | USB 3.0, passive stereo (not disrupted by projector light) |
| Projector | Any HDMI projector | 1920x1080 default; configurable |
| Mounting | Rigid fixture | Camera and projector must not move relative to each other after calibration |
| Environment | Controllable lighting | Dark room required for calibration; runtime tolerates ambient light |
The ZED v1 uses passive stereo (two regular cameras) rather than active IR projection. This is critical because:
- Active IR sensors (Kinect, RealSense) are disrupted by projector light
- The projected content actually helps the ZED by adding texture to featureless surfaces
- No interference between the depth sensing and projection systems
┌─────────────┐
│ Projector │
└──────┬──────┘
│ HDMI
▼
┌───────────────────────┐
│ Jetson Orin │◄──── USB 3.0 ────┐
└───────────────────────┘ │
┌─────┴─────┐
│ ZED Camera│
└───────────┘
Both projector and camera aim at the same target surface.
Mount them rigidly — any movement invalidates calibration.
| Package | Version | Purpose |
|---|---|---|
pyzed |
5.1+ | ZED SDK Python bindings |
opencv-python |
4.8+ | Image processing (system package, NVIDIA-optimised) |
numpy |
1.26.4 | Array operations (pinned for OpenCV compatibility) |
PyOpenGL |
3.1.5+ | OpenGL bindings |
glfw |
Latest | Window management |
tensorrt |
10.x | ML inference (system package, optional) |
pycuda |
Latest | CUDA memory management (optional, for segmentation) |
- CUDA 12.x (included in JetPack)
- OpenGL 3.3+ (Mesa or NVIDIA drivers)
- Python 3.10+
zed_proj/
├── setup.sh # One-time environment setup
├── config.py # All tuneable parameters (edit this)
├── main.py # Runtime entry point
│
├── calibration/ # Offline calibration tools
│ ├── __init__.py
│ ├── graycode_generator.py # Generate 46 structured light patterns
│ ├── graycode_capture.py # Display patterns & capture with ZED
│ ├── graycode_decoder.py # Decode captures → correspondences
│ └── lut_generator.py # Build inverse LUT from correspondences
│
├── pipeline/ # Runtime components
│ ├── __init__.py
│ ├── zed_capture.py # ZED SDK wrapper (RGB + depth)
│ ├── depth_processor.py # Depth-range masking
│ ├── renderer.py # OpenGL LUT-warp renderer
│ └── segmentation.py # TensorRT semantic segmentation
│
├── shaders/ # GPU programs
│ ├── warp.vert # Vertex shader (fullscreen quad)
│ └── warp.frag # Fragment shader (LUT lookup + masking)
│
├── models/ # ML models
│ ├── __init__.py
│ └── download_models.py # Export DeepLabV3 to ONNX
│
├── tests/ # Verification tests
│ ├── __init__.py
│ ├── test_zed.py # Camera capture test
│ ├── test_opengl.py # OpenGL rendering test
│ └── test_calibration.py # Pattern generation test
│
└── utils/ # Shared utilities
└── __init__.py
cd /home/null_0/zed_proj
chmod +x setup.sh
./setup.shThis script:
- Pins NumPy to 1.26.4 (fixes OpenCV compatibility)
- Installs GLFW and other Python packages
- Verifies all dependencies are working
- Checks for connected hardware
# Test ZED camera (captures 10 frames, prints depth stats)
python3 -m tests.test_zed
# Expected output:
# ZED ready: 1280x720
# frame 0: rgb (720, 1280, 3) depth min=0.45m max=12.34m
# ...
# 10/10 frames captured in 0.35s (28.6 FPS)# Test OpenGL (opens window, renders colours for 3 seconds)
python3 -m tests.test_opengl
# Expected output:
# OpenGL: 3.3.0 NVIDIA 535.154.05
# Renderer: NVIDIA Tegra Orin (nvgpu)/integrated/SSE2
# 180 frames in 3.00s → 60.0 FPS
# OpenGL test PASSEDxrandr --queryLook for a second connected display (e.g., HDMI-0 connected 1920x1080). Note the display index for config.py.
Calibration establishes the projector-camera correspondence using Gray code structured light. This only needs to be done once, unless the camera or projector is moved.
-
Pattern Generation: 46 binary stripe patterns are created:
- 2 reference frames (full white, full black)
- 22 horizontal patterns (11 bits × 2 for positive/inverse)
- 22 vertical patterns (11 bits × 2 for positive/inverse)
-
Pattern Projection & Capture: Each pattern is displayed on the projector and captured by the ZED camera. The 150ms delay between display and capture ensures the projector has stabilized.
-
Decoding: For each camera pixel, the sequence of bright/dark observations forms a Gray code that uniquely identifies which projector pixel illuminated it.
-
LUT Building: The camera→projector correspondences are inverted to create a projector→camera LUT. Unmapped pixels (occluded or outside the projection area) are filled using neighbor averaging.
- Darken the room — Ambient light reduces pattern contrast and decoding accuracy
- Position hardware — Aim both camera and projector at the target surface
- Stabilize everything — Any movement during capture invalidates the calibration
- Check overlap — The camera must see the entire projected area
# With projector (fullscreen on secondary display)
python3 -m calibration.graycode_capture
# Without projector (windowed preview for testing)
python3 -m calibration.graycode_capture --windowedThe script will:
- Generate all 46 patterns
- Open a window/fullscreen on the projector
- Wait for you to press ENTER
- Cycle through all patterns, capturing each one
- Save captures to
calibration/captures/
Output:
Generated 46 patterns
ZED camera opened
Fullscreen on display 1
Ready. Make sure the scene is static and the room is dark.
Press ENTER in this terminal to start capture...
[1/46] white
[2/46] black
[3/46] h_00
...
[46/46] v_10_inv
Captured 46/46 frames → calibration/captures
python3 -m calibration.lut_generatorOutput:
Decoded 691200/921600 valid correspondences (75.0%)
Direct coverage: 1843200/2073600 (88.9%)
Filled 230400 holes via neighbor averaging
LUT saved → calibration/lut.npy shape=(1080, 1920, 2)
Coverage interpretation:
- >80% decoded: Excellent calibration
- 50-80% decoded: Acceptable; some areas may be noisy
- <50% decoded: Poor calibration; room may not be dark enough, or projector/camera don't overlap well
| Issue | Solution |
|---|---|
| Low decode percentage | Darken room further; increase CAPTURE_DELAY in config |
| Stripes visible in decoded image | Camera exposure too long; reduce ambient light |
| Edge artifacts | Ensure projector is in focus; avoid projecting on highly specular surfaces |
| Partial coverage | Adjust camera/projector positions to maximize overlap |
Delete old calibration and re-run:
rm -rf calibration/captures calibration/lut.npy
python3 -m calibration.graycode_capture
python3 -m calibration.lut_generatorpython3 main.pyThe system will:
- Load the calibration LUT
- Initialize the ZED camera
- Create an OpenGL window (fullscreen on projector or windowed preview)
- Enter the main loop:
- Capture RGB and depth from ZED
- Generate depth mask (optional)
- Warp RGB through LUT via GPU shader
- Display on projector
- Print FPS statistics every 30 frames
Output:
=== Dynamic Projection Mapping ===
LUT loaded: 1920x1080
ZED ready: 1280x720
Running — press Ctrl+C or close the window to stop
frame 30 | 29.8 FPS | 33.6 ms
frame 60 | 30.1 FPS | 33.2 ms
...
| Flag | Description |
|---|---|
--windowed |
Force windowed preview instead of fullscreen projector output |
--no-mask |
Disable depth masking (project everywhere regardless of depth) |
--lut PATH |
Use a specific LUT file instead of calibration/lut.npy |
Examples:
# Windowed preview (no projector needed)
python3 main.py --windowed
# Disable depth masking (useful for debugging)
python3 main.py --no-mask
# Use a different calibration
python3 main.py --lut /path/to/other_lut.npy
# Combine flags
python3 main.py --windowed --no-mask- Close the window — Click the X button (windowed mode)
- Ctrl+C — Interrupt from terminal
- Both methods trigger proper cleanup of camera and GPU resources
All parameters are in config.py. Edit this file to customize behavior.
class ZED:
RESOLUTION = sl.RESOLUTION.HD720 # 1280x720 @ 60fps max
DEPTH_MODE = sl.DEPTH_MODE.NEURAL # ML-enhanced depth (best quality)
FPS = 30 # Capture frame rate
DEPTH_MIN = 0.3 # Minimum depth in meters
DEPTH_MAX = 20.0 # Maximum depth in meters| Setting | Options | Notes |
|---|---|---|
RESOLUTION |
HD2K, HD1080, HD720, VGA |
Higher = sharper but slower |
DEPTH_MODE |
NEURAL, ULTRA, QUALITY, PERFORMANCE, NONE |
NEURAL is most accurate but slowest |
FPS |
15, 30, 60, 100 | Depends on resolution |
class Projector:
WIDTH = 1920 # Native projector resolution
HEIGHT = 1080
FPS = 60 # Informational only
DISPLAY_ID = 1 # Monitor index (0 = primary)Find your display ID with xrandr --query. The projector should be index 1 if it's the second display.
class Calibration:
BITS_X = 11 # ceil(log2(1920)) = 11
BITS_Y = 11 # ceil(log2(1080)) = 11
PATTERN_DIR = "calibration/patterns"
CAPTURE_DIR = "calibration/captures"
LUT_PATH = "calibration/lut.npy"
CAPTURE_DELAY = 0.15 # Seconds between display and captureIf using a different projector resolution, update BITS_X and BITS_Y:
- 4K (3840×2160): BITS_X=12, BITS_Y=12
- 720p (1280×720): BITS_X=11, BITS_Y=10
class DepthMask:
MIN = 0.5 # Minimum depth (meters)
MAX = 4.0 # Maximum depth (meters)
FILTER_KERNEL = 5 # Median filter size (odd number)Only surfaces between MIN and MAX meters from the camera will receive projection. Useful for:
- Excluding foreground objects (people walking in front)
- Excluding distant backgrounds
- Focusing on a specific depth range
class Segmentation:
ENABLED = False # Set True to enable
ONNX_PATH = "models/deeplabv3.onnx"
ENGINE_PATH = "models/deeplabv3.engine"
INPUT_SIZE = (512, 512)
TARGET_CLASSES = [0, 9, 15] # PASCAL VOC class IDsTarget classes (PASCAL VOC):
- 0: background
- 9: chair
- 15: person
When enabled, projection is masked to only these classes.
class Render:
TARGET_FPS = 30
VERTEX_SHADER = "shaders/warp.vert"
FRAGMENT_SHADER = "shaders/warp.frag"
WINDOWED_PREVIEW = True # False for fullscreen projector
PREVIEW_WIDTH = 960
PREVIEW_HEIGHT = 540Set WINDOWED_PREVIEW = False for actual projection use.
Wrapper around the ZED SDK for synchronized RGB and depth capture.
from pipeline.zed_capture import ZEDCapture
zed = ZEDCapture()
rgb, depth = zed.grab() # rgb: (H,W,3) uint8, depth: (H,W) float32 meters
zed.close()Creates binary masks from depth data with morphological cleanup.
from pipeline.depth_processor import DepthProcessor
proc = DepthProcessor(min_depth=0.5, max_depth=4.0)
mask = proc.create_mask(depth) # (H,W) uint8, 255=valid, 0=masked
# With segmentation
mask = proc.create_combined_mask(depth, seg_labels, target_classes=[0, 15])OpenGL renderer that warps camera imagery through the calibration LUT.
from pipeline.renderer import WarpRenderer
renderer = WarpRenderer("calibration/lut.npy", windowed=True)
while renderer.render(rgb, mask): # Returns False when window closed
rgb, depth = zed.grab()
mask = proc.create_mask(depth)
renderer.close()TensorRT-accelerated semantic segmentation.
from pipeline.segmentation import TRTSegmentation
seg = TRTSegmentation()
labels = seg.infer(rgb) # (H,W) int32 class labels
seg.close()from calibration.graycode_generator import generate_all_patterns
patterns = generate_all_patterns() # List of (name, ndarray) tuples
# patterns[0] = ("white", 255-filled array)
# patterns[1] = ("black", 0-filled array)
# patterns[2] = ("h_00", horizontal bit 0 pattern)
# ...from calibration.graycode_decoder import decode
proj_x, proj_y, valid = decode()
# proj_x: (cam_h, cam_w) int32 — projector column per camera pixel
# proj_y: (cam_h, cam_w) int32 — projector row per camera pixel
# valid: (cam_h, cam_w) bool — reliable correspondencesfrom calibration.lut_generator import build_lut, save_lut
proj_x, proj_y, valid = decode()
lut = build_lut(proj_x, proj_y, valid) # (proj_h, proj_w, 2) float32
save_lut(lut) # Saves to calibration/lut.npy| Change | Impact | Trade-off |
|---|---|---|
DEPTH_MODE = PERFORMANCE |
+10-15 FPS | Less accurate depth edges |
DEPTH_MODE = NONE |
+20 FPS | No depth masking available |
Segmentation.ENABLED = False |
+5-10 FPS | No semantic masking |
RESOLUTION = VGA |
+15 FPS | Lower image quality |
--no-mask flag |
+2-3 FPS | No depth processing |
-
Use MAXN power mode:
sudo nvpmodel -m 0 sudo jetson_clocks
-
Disable V-Sync (if tearing is acceptable):
# In renderer.py _init_window(): glfw.swap_interval(0) # Change from 1 to 0
-
Reduce capture delay (if projector is fast):
# In config.py: CAPTURE_DELAY = 0.10 # Reduce from 0.15
The system uses ~2GB GPU memory with segmentation enabled, ~500MB without.
To reduce memory:
- Disable segmentation
- Use lower camera resolution
- Use
DEPTH_MODE = PERFORMANCEorNONE
| Problem | Solution |
|---|---|
| "Cannot open ZED camera" | Check USB 3.0 connection; avoid hubs |
| Low FPS from camera | Use MAXN power mode; check USB bandwidth |
| Depth is all NaN | Ensure stereo baseline sees the scene; check DEPTH_MIN/MAX |
Verify ZED connection:
lsusb | grep -i stereo
# Should show: Bus 001 Device 002: ID 2b03:f582 StereoLabs ZED| Problem | Solution |
|---|---|
| "GLFW init failed" | Install: sudo apt install libglfw3 libglfw3-dev |
| "Window creation failed" | Check DISPLAY env var; ensure X11 is running |
| Black window | LUT may be invalid; re-run calibration |
| Shader compilation error | Check OpenGL version (`glxinfo |
| Problem | Solution |
|---|---|
| Low decode percentage (<50%) | Darken room; increase CAPTURE_DELAY |
| Visible stripes in output | Room not dark enough; camera exposure too long |
| "white.png not found" | Run graycode_capture before lut_generator |
| LUT has poor coverage | Adjust camera/projector to maximize overlap |
| Problem | Solution |
|---|---|
| "LUT not found" | Run calibration first |
| Projected image is mirrored | Check shader; ensure LUT matches projector orientation |
| Projection doesn't align | Recalibrate; camera or projector may have moved |
| Flickering | Check projector refresh rate; try swap_interval(1) |
| Problem | Solution |
|---|---|
| <20 FPS | Use MAXN mode; reduce resolution; disable segmentation |
| High latency | Disable V-Sync; reduce CAPTURE_DELAY |
| Memory errors | Reduce resolution; disable segmentation |
┌─────────────────────────────────────────────────────────────────────────┐
│ RUNTIME DATA FLOW │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ZED Camera (USB 3.0, writes to unified memory) │
│ │ │
│ ├─► RGB Image (uint8, 720×1280×3) │
│ │ │ │
│ │ └──► OpenGL Texture Unit 0 (glTexImage2D) │
│ │ │
│ └─► Depth Map (float32, 720×1280) │
│ │ │
│ └─► DepthProcessor │
│ │ │
│ └─► Binary Mask (uint8, 720×1280) │
│ │ │
│ └──► OpenGL Texture Unit 2 │
│ │
│ Pre-computed LUT (float32, 1080×1920×2) │
│ └──► OpenGL Texture Unit 1 (uploaded once at startup) │
│ │
│ Fragment Shader (runs per projector pixel): │
│ 1. Sample LUT at projector UV → get camera UV │
│ 2. Sample RGB texture at camera UV → get color │
│ 3. Sample mask texture at camera UV → get alpha │
│ 4. Output: color × alpha │
│ │ │
│ └─► Framebuffer ─► HDMI ─► Projector │
│ │
│ Optional: TensorRT Segmentation │
│ RGB ─► Resize ─► Normalize ─► TensorRT ─► Class Labels │
│ Combined with depth mask for selective projection │
│ │
└─────────────────────────────────────────────────────────────────────────┘
The Jetson Orin's CPU and GPU share physical RAM. This eliminates PCIe transfers:
Traditional discrete GPU:
Camera ─► CPU RAM ─► [PCIe copy] ─► GPU VRAM ─► Render
Jetson unified memory:
Camera ─► Unified RAM ◄─► GPU Render
│
└─► Direct access, no copy
Result: Lower latency, higher throughput, simpler code.
Gray code ensures adjacent values differ by only one bit, making the decoding robust to noise:
Binary: 0 1 2 3 4 5 6 7
000 001 010 011 100 101 110 111
↓ ↓↓ ↓ ↓ ↓↓ ↓ (multiple bit changes)
Gray: 0 1 3 2 6 7 5 4
000 001 011 010 110 111 101 100
↓ ↓ ↓ ↓ ↓ ↓ ↓ (single bit changes)
For a 1920-pixel width, we need 11 bits (2^11 = 2048 > 1920). Each bit is encoded as a stripe pattern:
- Bit 10 (MSB): Half black, half white
- Bit 9: Quarters alternating
- Bit 0 (LSB): Single-pixel stripes
By projecting each pattern and its inverse, we get robust decoding even with uneven illumination.
The vertex shader passes through a fullscreen quad:
gl_Position = vec4(aPos, 0.0, 1.0);
TexCoord = aTexCoord;The fragment shader does the heavy lifting:
// 1. Look up camera UV from LUT
vec2 camUV = texture(lutTexture, TexCoord).rg;
// 2. Flip Y for OpenGL coordinates
vec2 flippedUV = vec2(camUV.x, 1.0 - camUV.y);
// 3. Sample camera RGB
vec4 color = texture(zedTexture, flippedUV);
// 4. Apply mask
if (useMask == 1) {
color.rgb *= texture(maskTexture, flippedUV).r;
}
FragColor = color;Cost per frame: 2 texture lookups + 1 multiply per pixel = ~4M operations for 1080p, trivial for modern GPUs.
[Add your license here]
[Add contribution guidelines here]
- ZED SDK by Stereolabs
- DeepLabV3 by Google Research
- Gray code structured light technique from academic literature