Project page · Preprint, 2026
🌐 Live site: kingdroper.github.io/AVP
Vision-Language-Action (VLA) models commonly map language instructions and visual observations to actions in a single forward pass. While conceptually simple, this entangles instruction comprehension, spatial scene understanding, and motor control within one learning objective, so the action expert must implicitly relearn cognitive and perceptual capabilities already present in the pretrained VLM.
We propose AVP (Action with Visual Primitives), an end-to-end VLA architecture with a visual-primitive-centric interface between the VLM and the action expert: the VLM infers the next-stage target and emits compact, spatially grounded primitive tokens (points, boxes, sub-goal markers, memory-and-order anchors); the action expert consumes these tokens and focuses solely on kinematic mapping. Primitive supervision is derived directly from end-effector kinematics, eliminating the need for manual spatial annotation.
On real-robot pick-and-place tasks, AVP improves success rate by 27.61% over π₀.₅ and outperforms other recent methods, with consistent gains in data efficiency, spatial-compositional generalization, and object-level transfer.
- Explicit VLM ↔ action-expert interface. A Policy Steering channel carries visual primitives from a reasoning-capable VLM to a flow-matching action expert, demarcating their learning responsibilities and avoiding duplicated perception inside the policy.
- Form-agnostic primitive protocol. Points, bounding boxes, sub-goal markers, memory primitives, and order-and-memory primitives all flow through the same interface — without changing the underlying policy architecture.
- Kinematics-derived supervision. Ground-truth primitives are projected from the recorded end-effector trajectory plus a one-time camera calibration, so the supervision pipeline scales to new platforms at essentially zero additional annotation cost.
- Pose Primitives — single-step end-effector anchors (grasp + placement).
- Goal Primitives — (source, destination) region pairs per sub-task.
- Memory Primitives — markers that persist across frames so the model can refer to objects that have left the field of view.
- Order + Memory Primitives — explicit indices (1 → 2 → 3, turning red once executed) for sequential targeting.
The project page hosts real-robot rollouts on:
| Task | Highlights |
|---|---|
| Chinese chess manipulation | Dense-board sequential moves |
| Multi-instruction composition (red / black) | Chained instructions in one rollout |
| Cross-domain generalization | Zero-shot transfer to unseen objects and backgrounds |
| Snake-game sequential targeting | Long-horizon ordered targeting via memory-and-order primitives |
See them in action on the live project page.
*Equal contribution · †Project Leader · ‡Corresponding author
| Affiliation | |
|---|---|
| Weilong Guo*†‡ | Anyverse Dynamics |
| Yuchen Wang* | Tsinghua University |
| Renping Zhou | Tsinghua University |
| Wenda Xu‡ | Anyverse Dynamics |
| Gao Huang‡ | Tsinghua University |
@article{guo2026avp,
title = {Action with Visual Primitives},
author = {Guo, Weilong and Wang, Yuchen and Zhou, Renping and Zhang, Yunfeng
and Fang, Rui and Meng, Yue and Xu, Wenda and He, Yuan and Huang, Gao},
journal = {arXiv preprint},
year = {2026}
}This repo hosts the static project page for AVP. No build step — plain HTML / CSS / JS deployed via GitHub Pages.
git clone https://github.com/Kingdroper/AVP.git
cd AVP
python3 -m http.server 8000
# open http://localhost:8000AVP/
├── index.html main page
├── README.md this file
├── .nojekyll tells GitHub Pages to skip Jekyll
└── static/
├── css/style.css dark-theme styling + animations
├── js/main.js scroll reveal + bibtex copy
├── images/ framework & primitives figures
└── videos/ demo clips (web-optimized) + posters
See static/videos/README.md for the ffmpeg one-liner used to compress clips
for web playback, and for the HTML snippet that swaps a placeholder tile for a
real <video> element.