Interactive Body Pose Visualization Installation with Sensor Integration
Created by Rita Olivença and André Rocha
For Periphera Festival 2025 | Trafaria, Almada, Portugal
FCT UNL (NOVA School of Science and Technology)
OSKAR uses computer vision and pose detection to create abstract visualizations of human movement. Body movements are captured through a webcam and rendered as stylized "Schlemer sticks" — geometric lines inspired by Oskar Schlemmer's Bauhaus theater work. The system integrates with micro:bit sensors for dynamic trail control.
Technologies: p5.js, ml5.js, MoveNet pose estimation, p5.mapper projection mapping, p5.webserial
Key Features:
- Dual-canvas architecture with independent projection mapping
- Real-time pose detection (17 body keypoints)
- Three distinct visualizer types with configurable behaviors
- Sensor-responsive trail system via micro:bit integration
- Physics-based springy line rendering with elasticity control
- Detection mask for selective area tracking
- Mirror mode toggle
- Fullscreen support
OSKAR offers three distinct visualization modes:
| Type | Description | Trail Behavior |
|---|---|---|
| White+Trail | Classic white Schlemmer sticks | Sensor-responsive trails |
| Clear | Pure white sticks, always clean | No trails (immune to sensors) |
| Springy+Trail | Physics-based white bezier curves | Sensor-responsive trails + elasticity |
| Key | Function |
|---|---|
| Projection Mapping | |
| C | Toggle calibration mode (drag corners to map) |
| S | Save current mapping |
| L | Load saved mapping |
| F | Toggle fullscreen |
| Canvas Visibility | |
| Shift+1 | Show/Hide Canvas 1 |
| Shift+2 | Show/Hide Canvas 2 |
| Visualizer Selection | |
| Alt+1 | Cycle Canvas 1 (White+Trail → Clear → Springy+Trail) |
| Alt+2 | Cycle Canvas 2 (White+Trail → Clear → Springy+Trail) |
| Pose Controls | |
| B | Toggle video background |
| T | Toggle trail effect |
| R | Recalibrate Schlemmer stick length (stand in T-pose) |
| K | Recalibrate Schlemmer stick length (alternative) |
| M | Toggle mirror mode |
| Detection Mask | |
| A | Toggle mask editing mode |
| E | Enable/disable mask filtering |
| X | Clear mask completely |
| Sensor Integration | |
| P | Connect sensor port / Enable simulation mode |
| + | Increase springy line elasticity |
| - | Decrease springy line elasticity |
| Mouse Actions | |
| Click | Add mask point (in mask edit mode) |
| Shift+Drag | Move mask point (in mask edit mode) |
| Double-click | Toggle fullscreen |
- Micro:bit receiver connected via USB (runs receiver code)
- 1-6 micro:bit senders with accelerometer data (sensor IDs 1-6)
- Serial communication at 115200 baud rate
- XY Motion (combined) controls trail frequency and persistence
- No motion = no trails
- High motion = dramatic trail effects (up to 300 trail frames)
- Z Motion automatically controls springy line elasticity (0.2-2.5 range)
- Most dynamic sensor wins (highest motion level controls effects)
- Press P to connect sensor port
- Select micro:bit port from browser dialog
- Sensors auto-register as S1-S6
- Move sensors to see immediate trail response
If no sensors available, Press P enables mouse simulation:
- Mouse position = XY sensor motion
- Automatic Z oscillation = springiness demo
-
Start a local server:
# Python 3 python3 -m http.server 8000 # Or Node.js npx http-server -p 8000
-
Open browser:
http://localhost:8000 -
Allow camera access when prompted
-
Stand in T-pose and press R to calibrate stick lengths
For projection installations:
- Set up your projector(s)
- Press Alt+1 / Alt+2 to select visualizers for each canvas
- Press C to enter calibration mode, drag corners to map projection surface
- Press C again to exit calibration
- Press A to define detection mask (optional - limits tracking area)
- Press P to connect sensors (optional - for dynamic trails)
- Press F for fullscreen
OSKAR supports two independent canvases for dual-projection setups:
- Canvas 1 & 2: Each can display any of the three visualizer types
- Independent mapping: Each canvas maps separately to projection surfaces
- Shared sensors: All visualizers respond to the same sensor data
- Clear visualizer exception: Always immune to sensor trail effects
- Press C → enter calibration mode (corners become draggable)
- Drag 4 corners to match your projection surface(s)
- Press C again → exit calibration mode
- Settings automatically saved to localStorage
Define a polygon to limit pose detection to specific areas (stage, interaction zone, etc.).
- Video clipping: Video shows only inside polygon
- Pose filtering: Only tracks people with ≥25% of keypoints inside mask
- Graphics freedom: Schlemmer sticks extend freely beyond mask boundary
- Persistent: Saved to localStorage
- Press A → enter mask edit mode (fullscreen video with overlay)
- Click to add points (minimum 3 points)
- Shift+Drag to move existing points
- Press A again → exit edit mode
- Press E to toggle mask on/off without clearing
- Press X to clear mask completely
9 Fixed Connections inspired by Oskar Schlemmer's Bauhaus theater work:
- Right ankle → Right knee
- Right knee → Right hip
- Right hip → Right shoulder
- Right shoulder → Right wrist
- Left ankle → Left knee
- Left knee → Left hip
- Left hip → Left shoulder
- Left shoulder → Left wrist
- Left shoulder ↔ Right shoulder (centered line)
Calibration: Press R while standing in T-pose to set base stick length
The Springy+Trail visualizer uses physics simulation for dynamic line behavior:
- Spring Strength: How quickly lines return to rest position
- Damping: How much bounce vs. stiffness
- Elasticity: Overall multiplier (0.1 = very stiff, 3.0 = very bouncy)
- +/- keys: Manual elasticity adjustment
- Z sensor motion: Automatic elasticity control (overrides manual)
- Real-time response: Immediate visual feedback
| Mode | Description | Use Case |
|---|---|---|
| ON (default) | Mirrored display | Camera facing user (like a mirror) |
| OFF | Non-mirrored | Camera facing away / rear projection |
Toggle with M key - affects both video and pose detection
Uses p5.mapper for quad-corner mapping to any surface.
- Canvas resolution: 1280×720 (16:9)
- Uses WebGL transforms for hardware acceleration
- Mapping data saved to localStorage automatically
- Independent mapping for each canvas
17 detected keypoints via MoveNet:
| Index | Body Part | Index | Body Part |
|---|---|---|---|
| 0 | Nose | 9 | Left Elbow |
| 1 | Left Eye | 10 | Right Elbow |
| 2 | Right Eye | 11 | Left Hip |
| 3 | Left Ear | 12 | Right Hip |
| 4 | Right Ear | 13 | Left Knee |
| 5 | Left Shoulder | 14 | Right Knee |
| 6 | Right Shoulder | 15 | Left Ankle |
| 7 | Left Wrist | 16 | Right Ankle |
| 8 | Right Wrist |
| Issue | Solution |
|---|---|
| Camera not working | Access via localhost or https (not file://) |
| Check browser camera permissions | |
| Pose detection issues | Ensure adequate lighting and full body visibility |
| Check if detection mask is active (press A to check) | |
| Sticks wrong length | Press R while in T-pose to recalibrate |
| Canvas not showing | Press Shift+1 or Shift+2 to toggle visibility |
| Sensors not connecting | Install micro:bit receiver code, check USB connection |
| No trail effects | Press P to connect sensors or enable simulation |
| Browser serial issues | Use Chrome/Edge, ensure HTTPS or localhost |
| Reset all settings | Console: localStorage.clear() then location.reload() |
OSKAR/
├── index.html # Main HTML with library imports
├── sketch.js # Main controller with sensor integration
├── SchlemerVisualizer.js # Base visualizer class
├── WhiteSchlemerVisualizer.js # White sticks with trails
├── ClearRedSchlemerVisualizer.js # Clear white sticks (no trails)
├── SpringyBlueSchlemerVisualizer.js # Physics-based springy curves
├── PoseVisualizer.js # Legacy pose visualizer
├── sketchPose.js # Backup/alternative version
├── style.css # Styling
├── libs/ # Local p5.js + ml5.js libraries
├── maps/ # Projection mapping data
└── README.md # This file
Architecture:
sketch.js: Main controller with dual canvas system, pose detection, and sensor integrationSchlemerVisualizer.js: Base class for all visualizer types- Three specialized visualizers with different trail behaviors
- Sensor integration via OskarBit micro:bit system
Concept & Development:
- Rita Olivença
- André Rocha
Institutional Support:
- FCT UNL (NOVA School of Science and Technology)
- Periphera Festival
Technical Framework:
- p5.js - Creative coding
- ml5.js - Machine learning
- MoveNet - Pose detection
- p5.mapper - Projection mapping
- p5.webserial - Serial communication
Artistic Inspiration:
- Oskar Schlemmer - Bauhaus Theater & Triadisches Ballett
Multidisciplinary arts festival exploring technology, interactivity, and human connection through digital media, performance, and installation art.
Location: Trafaria, Almada, Portugal
Year: 2025
Institution: FCT UNL
Created for Periphera Festival 2025 exhibition.
For inquiries about reuse or adaptation, please contact the creators.
Created with ❤️ for Periphera Festival 2025