GazEstim

Estimate and visualise where a bird is looking, from DeepLabCut pose tracking.

GazEstim takes a raw DeepLabCut (DLC) pose table and produces:

1. a physically-constrained, cleaned skeleton rendered over the real arena image;
2. an azimuthal gaze visualisation — two monocular fields plus the binocular overlap; and
3. a table of kinematic features (head direction, angular velocity, position, speed), each with a per-frame confidence estimate, saved as a MATLAB .mat plus a summary figure.

It runs from a single MATLAB function or directly from Python.

Contents

• Features
• Repository layout
• Requirements
• Installation
• Quick start
• Outputs
• Body parts and roles
• How it works
• Troubleshooting
• Performance
• Acknowledgements
• License

Features

• Label-agnostic DLC reader. Header rows, an optional index column and column order are detected automatically; body-part labels are read from the file, not hardcoded.
• Physically-constrained cleaning. Low-likelihood detections, single-frame jumps, impossible bone lengths and anatomically impossible poses are rejected and re-estimated from surrounding frames, keeping the animal's size and joint ordering consistent.
• Exact spatial overlay. DLC coordinates live in the video's pixel space; the background image is stretched to that space so the skeleton overlays the arena 1:1.
• Azimuthal gaze model. Two monocular cones and one binocular cone are projected from the head, driven by the head–beak axis and configurable field-of-view angles.
• Kinematic features. Head direction, angular velocity, centre of gravity and speed, each with a likelihood and an approximate uncertainty, exported to a .mat file with a four-panel summary figure.
• Two front-ends. A MATLAB wrapper (process_gaze) that prepares inputs and calls the pipeline, or the Python master script driven by a JSON config, with no MATLAB required.
• Fast rendering. The static background is rasterised once and only the moving artists are redrawn per frame (blitting), with frames piped directly to ffmpeg.

Repository layout

functions/video/process_gaze.m   MATLAB entry point (prepares a config, runs the pipeline)
configfiles/
  master_gaze.py                  orchestrator: reads a JSON config, runs the tools
  HexArena.png                    default background image (subject removed)
toolboxes/
  pose_clean.py                   DLC reader + physically-constrained cleaning
  pose_render.py                  skeleton + gaze cones rendered over the background
  features.py                     kinematic features and the summary figure
  make_plate.py                   utility: remove the subject from a photo (inpainting)
docs/USAGE.md                     full parameter and output reference
Example_data.csv                  sample DLC table
Example_data_rendered.mp4         sample rendered clip

The three components live in separate folders to fit an existing MATLAB project structure. process_gaze.m resolves the sibling folders from its own location; params.root, params.toolboxes and params.masterScript override this if the components are moved.

Requirements

• Python 3.8+ with numpy, matplotlib, pillow and scipy (opencv-python is needed only to regenerate a background with make_plate.py).
• ffmpeg on the system PATH, for video output.
• MATLAB is optional; it is only needed for the process_gaze front-end.

Installation

git clone https://github.com/JesusJBallesteros/GazEstim.git
cd GazEstim
python -m pip install -r requirements.txt

Install ffmpeg from ffmpeg.org, or on Windows with winget install Gyan.FFmpeg. To use the MATLAB front-end, add the repository to the MATLAB path (addpath(genpath('GazEstim'))).

Quick start

From MATLAB

% Defaults: clean, render the video, using the bundled background
process_gaze('Example_data.csv');

% With options
p = struct();
p.startTime = 10;  p.endTime = 40;   % analyse 10-40 s of the recording
p.pCut      = 0.6;                   % stricter likelihood gate
p.features  = true;                  % also export kinematic features + figure
r = process_gaze('Example_data.csv', p);   % r.output, r.features, r.figure, ...

help process_gaze lists every parameter and its default.

From Python

python configfiles/master_gaze.py config.json

where config.json mirrors the parameters in snake_case. A minimal example:

{
  "csv": "Example_data.csv",
  "output": "out.mp4",
  "background": "configfiles/HexArena.png",
  "video_w": 1250, "video_h": 1160,
  "features": true
}

See docs/USAGE.md for the complete parameter and output reference.

Outputs

output	how to request	contents
Rendered video	default	Skeleton and gaze cones over the arena, at the downsampled frame rate.
Features .mat	features = true	Struct features with per-frame angle_deg, angvel_deg_s, centroid_x/y, speed_px_s, each with *_likelihood, *_estimated, *_sd/*_ci95, plus timestamp_ms from recording start.
Summary figure	with the features	Head-direction rose, angular velocity vs direction, occupancy heatmap, speed over time.
Preview still	previewFrame = <n>	A single PNG of one frame; no ffmpeg required.

Body parts and roles

Labels are read from the CSV. List them with:

process_gaze('Example_data.csv', struct('listParts', true));

Any subset can be used, and non-standard names are mapped onto the pipeline's anatomical roles (beak, head, back, left_wing, right_wing, tail):

p.parts      = {'bill','nape','back','tail'};
p.roles.beak = 'bill';
p.roles.head = 'nape';

Each role enables specific checks, and missing roles degrade gracefully rather than failing:

role(s)	used for
beak + head	gaze direction (required for the cones) and eye placement
head + back	body-length scale, which sets the jump threshold
back + wings	wing bone lengths, eye lateral offset, head-behind-wings check
back + tail	tail bone length and the cone length

A label with no matching role is still cleaned and drawn, with a generic colour and no bone links.

How it works

Cleaning. For each frame a joint is treated as implausible, and re-estimated from surrounding frames, when its likelihood is below a threshold, when it jumps further than a fraction of the body length between frames, when a rigid bone length departs from its median, or when the head falls behind the wing line. Re-estimation interpolates the joint's direction from neighbouring frames while snapping its length to the canonical bone length, so the animal keeps a constant size.

Coordinates. DLC coordinates are expressed in the source video's pixel space. The background image is stretched to videoWidth x videoHeight so that the two align exactly; this must match the video DLC analysed, which is not necessarily the size of the background image file.

Gaze. The head–beak vector defines the forward axis. Each eye is placed a fraction of the head–beak distance ahead of the head and offset laterally. A monocular field of configurable width is drawn from each eye, and their frontal overlap forms the binocular field.

Kinematics. Head direction is measured as the angle of the head–beak vector relative to the video vertical. Because it is circular it is unwrapped before differentiation, and all rates use the true (non-uniform) timestamps. The centre of gravity is the mean of the selected labels, and speed is its time derivative.

Troubleshooting

• No module named 'numpy' — Python ran, but the interpreter found lacks the packages (commonly a Microsoft Store stub, or a system Python while packages live in Anaconda). Set params.pythonExe to the right interpreter, or params.autoInstall = true to install into the one found. The interpreter is verified and cached on first use.
• ffmpeg errors mid-render — the renderer pipes frames to ffmpeg, so if it exits the write fails. The error quotes ffmpeg's own message; common causes are the output file being open in a player, a non-writable folder, or a path without a video extension.
• Odd video dimensions — H.264 requires even width and height; the render size is adjusted by at most one pixel to satisfy this.

More detail is in docs/USAGE.md.

Performance

On the bundled example (about 16,000 frames after downsampling, 912x846 output), cleaning takes a few seconds and rendering runs at roughly 30-40 frames per second on a single CPU core, using well under 100 MB of memory. Feature extraction without video is fast enough to process long recordings in one pass; use video = false to skip rendering.

Acknowledgements

GazEstim was developed by Jesús J. Ballesteros. The pipeline was implemented in close collaboration with Anthropic's Claude, used as a pair-programming assistant for the design and implementation of the cleaning, rendering and feature-extraction code.

License

Released under the GNU General Public License v3.0. See LICENSE.