MiniVLA-NAV

Mini Vision-Language-Action Navigation with CLIP and PyBullet

Given prompt: "an object useful to drink coffee"

Given prompt: "go to the red mug"

---

Overview

MiniVLA-NAV is a Mini Vision-Language-Action (VLA) project I created to start experimenting with embodied navigation guided by a Vision-Language Model (VLM).
The system uses OpenAI CLIP (ViT-B/32) to connect text and perception: the robot receives a natural-language prompt, analyzes its camera feed, and navigates toward the region that best matches the description.

The entire environment (robot, world, textures, and objects) is built procedurally in PyBullet.
The goal is to explore how a pretrained vision-language model can produce primitive navigation behavior without explicit object detection or reinforcement learning.

---

How It Works

At each simulation step:

1. The robot captures an RGB frame from its onboard camera.
2. The image is split into K vertical sectors.
3. Each sector is encoded using CLIP’s image encoder.
4. The cosine similarity with the text embedding determines relevance.
5. The sector with the highest score defines the heading.
6. Forward velocity is scaled by similarity confidence.
7. The robot moves smoothly toward the most semantically aligned region.

This setup forms a minimal VLA loop: language → vision → action.

Demo Gallery (GIFs)

The project demonstrates strong performance on both simple prompts such as "a red ball on the ground", where the robot consistently moves toward the target, and on more abstract or relational prompts like "the opposite of an apple" (successfully identifying a pear) or "an object useful to drink coffee" (finding a mug). It even generalizes to compositional descriptions such as "a ball that is the same color as the sun", correctly selecting a yellow sphere.

The algorithm can also exhibit exploratory behavior, as illustrated in the "orange mug on the grass" example, where the robot actively scans its surroundings to locate the object when it is not immediately visible.

Prompt: "big red ball on the ground"	Prompt: "an object useful to drink coffee"
Prompt: "an orange mug on the grass"	Prompt: "a small cyan ball on the ground"
Prompt: "find the biggest fruit"	Prompt: "the opposite of an apple"
Prompt: "a toy that can fly"	Prompt: "a ball that is used in a sport"
Prompt: "an object that contains sauce"	Prompt: "a ball that is the same color as the sun"

---

Installation

git clone https://github.com/rantaluca/MiniVLA-NAV.git
cd MiniVLA-NAV
pip install pybullet torch torchvision matplotlib pillow
pip install git+https://github.com/openai/CLIP.git

Python ≥ 3.9 is recommended.

---

Running the System

To test the agent:

python main.py

Then type a prompt when requested, the prompt should be the target object description: "a red ball on the ground"

To visualize the environment:

python env.py –gui –n_objects 20

---

Parameters

Parameter	Description	Default
K	Number of image sectors	5
fov_deg	Camera field of view	120°
v_max	Maximum forward velocity	15.0
w_gain	Angular velocity gain	7.5
conf_th	Confidence threshold	0.08
SMOOTH_v	Velocity smoothing factor	0.15
SMOOTH_w	Angular smoothing factor	0.03
GOAL_SIM_THRESHOLD	Goal found threshold	0.27
GOAL_FRAMES_REQUIRED	Nb of frames before stopping	20