VideoAgent-RL

I wanted to know how much of a tool-using video agent's behavior actually has to be learned. Recent systems wrap a large language model in a set of tools and train the whole controller end to end with reinforcement learning, usually a method called GRPO, which is expensive and learns several decisions at once. So I built the smallest version of the problem I could. I froze the parts that look at the video and pick an answer, and let the agent learn only the controller, the part that decides what to do next. Then I dropped four controllers into the same setup, from a purely random one up to PPO, and watched what changed.

The short version of what I found is that less is going on than I expected. Even a purely random controller nearly ties the learned ones on accuracy, the three learners land within a fraction of a point of each other, and the simplest of them gets there about three times sooner. Almost all of the value worth learning is in when to stop, not which tool to call, because the frozen answer function barely cares which frames it gets. So for this kind of agent a simple learned stopping rule recovers most of what heavy reinforcement learning gets credited for, and the real bottleneck is the part that reads the frames. The full write-up is in paper/, where main.tex is the source and main.pdf the rendered paper.

How it works, in one paragraph

Each video question is a small Markov decision process. I sample 32 frames per video and encode them once with a frozen CLIP image encoder, and I encode the question and options with the matching text encoder, so frames and text live in one space. A controller reads the state, issues tool actions that gather frames into an evidence pool, and at some point answers. Every controller hands off to the same frozen, deliberately weak reader that picks the option closest to the gathered evidence. Because perception and answering are fixed and shared by every method, any difference between controllers is the controller's doing and nothing else. Everything runs on cached vectors, so a training step is just a few dot products on a CPU, and the whole study cost under fifteen dollars.

How to run it

The cloud pipeline lives in app.py and runs on Modal: extract the features once on a GPU, then train and evaluate everything on CPU. To reproduce locally from cached features, point VIDEOAGENT_DATA at a copy of the data volume and use reproduce.py:

VIDEOAGENT_DATA=./_data python3 reproduce.py train     --seeds 0 1 2
VIDEOAGENT_DATA=./_data python3 reproduce.py eval      --seeds 0 1 2
VIDEOAGENT_DATA=./_data python3 reproduce.py aggregate --seeds 0 1 2
VIDEOAGENT_DATA=./_data python3 make_tables.py
VIDEOAGENT_DATA=./_data python3 make_curves.py

How to navigate the repo

The environment and the controllers:

• configs.py: paths, hyperparameters, the observation layout, and the frozen-backbone registry, all in one place.
• env.py: the environment itself, a Gymnasium decision process over the eight-action tool interface with the frozen CLIP reader.
• policies.py: the four controllers behind one interface, Random, UCB1, REINFORCE, and PPO, plus a fixed heuristic baseline.

The data and the features:

• data.py: loads NExT-QA and EgoSchema, builds the official splits, and reads the manifests.
• features.py: the frozen CLIP and BLIP-2 encoders, the per-video feature extraction, and the probes that measure the reader's ceiling.
• features_egoschema.py: the same extraction adapted to EgoSchema.

Training, evaluation, and analysis:

• train.py: the training loop for Random, UCB1, and REINFORCE, with a maskable PPO branch, all logging the same learning-curve format.
• eval.py: greedy evaluation by category, the no-evidence floor and leakage checks, and the zero-shot EgoSchema transfer.
• ablation.py: the decision-isolation ablation that separates choosing a tool from choosing when to stop.
• compare.py: aggregates the run outputs into the main table, the ablation table, and the learning-curve figure.

Reproduction and reporting:

• reproduce.py: the local multi-seed driver that trains, evaluates, and aggregates to a mean and standard deviation.
• make_tables.py: emits the LaTeX tables from the aggregated results.
• make_curves.py: regenerates the learning-curve figure with per-seed bands.
• app.py: the Modal entrypoints that run the whole pipeline in the cloud.

The write-up:

• paper/: the paper, main.tex plus the rendered main.pdf.
• RESULTS.md: a short results summary.
• poster/: the poster version of the same study.