ActiveCrop

Uncertainty-Aware Active Perception for Safe VLM-Based Robotic Crop Intervention

Research codebase investigating whether a VLM-based robot can reduce unsafe physical interventions by actively seeking additional visual information when perception is uncertain.

Repository	github.com/AvoCahDoe/ActiveCrop
Package	activecrop (src/ layout)
Python	≥ 3.11
License	MIT
Live demo	activecrop.vercel.app
Demo source	React + Plotly SPA in demo/
Demo images	Project-AgML/tomato_ripeness_detection (Laboro Tomato)
Default VLM	Mock provider (OpenAI-compatible / local stubs ready)
Task	Tomato state classification → safe pick / observe / abstain
Keywords	active perception, VLM, uncertainty, robotics, agriculture, calibration, intervention safety

---

Research question

A conventional VLM robotic pipeline assumes one observation is enough:

Camera → VLM → Decision → Action

In cluttered and dynamic greenhouses that assumption often fails. ActiveCrop asks:

Camera → VLM → Uncertainty → sufficient?
  ├── YES → risk evaluation → PICK / ABSTAIN
  └── NO  → select viewpoint → observe again → re-evaluate

Hypothesis: uncertainty-aware active perception reduces unsafe interventions while keeping extra observations and cost low.

Objectives

ID	Goal
O1	Evaluate single-view VLM crop-state perception
O2	Test whether VLM confidence predicts perception errors
O3	Select informative viewpoints when evidence is insufficient
O4	Decide intervene vs observe using uncertainty and risk
O5	Stress-test under occlusion, clutter, motion, and stale views

Concrete task

Classify each tomato as UNRIPE | RIPE | DAMAGED | UNCERTAIN, then choose:

• PICK — intervene when evidence and risk allow
• OBSERVE_AGAIN — acquire another viewpoint
• ABSTAIN — refuse unsafe action when risk remains high

The robot must never be forced to intervene when visual evidence is insufficient.

---

Features

• Synthetic greenhouse scene generator (RGB, depth, masks, GT)
• Eight fixed multi-view cameras on one scene
• Per-tomato ground truth recomputed per viewpoint (VLM-independent)
• VLM provider abstraction: mock (default), OpenAI-compatible, local
• Single-view baseline metrics: accuracy, F1, ECE, Brier, selective accuracy/coverage
• Safe intervention policy with confidence-threshold sweep
• Active perception baselines: random, entropy (IG/cost), VLM-guided
• ActiveCrop risk model: weighted uncertainty + occlusion + cost → PICK / OBSERVE / ABSTAIN
• Dynamic greenhouse (leaf/crop motion, stale observations)
• Benchmark suite M1–M5 × Easy / Medium / Hard / Dynamic
• Interactive React + Plotly showcase (demo/)

---

Repository layout

ActiveCrop/
├── README.md
├── LICENSE
├── CITATION.cff
├── pyproject.toml
├── .env.example
├── configs/
│   ├── models/          # VLM provider defaults
│   ├── environments/    # easy | medium | hard | dynamic
│   └── experiments/     # baseline → activecrop → benchmark
├── src/activecrop/
│   ├── simulation/      # scenes, cameras, viewpoints, dynamics
│   ├── vlm/             # providers, schemas, analyze CLI
│   ├── uncertainty/     # entropy / confidence utilities
│   ├── active_perception/
│   ├── intervention/    # risk model & policies
│   ├── evaluation/      # metrics & experiment runners
│   └── visualization/
├── scripts/             # demo asset export
├── demo/                # Vite React showcase
├── experiments/         # local runs/results (gitignored artifacts)
└── tests/

---

Status (phases)

Phase	Capability
1	Synthetic greenhouse scene generator (RGB, depth, GT)
2	Multi-view cameras (eight fixed viewpoints)
3	Ground truth per tomato, recomputed per view
4	VLM provider abstraction (Mock default)
5	Single-view baseline evaluation
6	Safe intervention policy + threshold sweep
7	Random active perception vs single-view
8	Entropy-based active perception (IG / cost)
9	VLM-guided viewpoint selection
10	ActiveCrop risk model vs all prior methods
11	Dynamic greenhouse (motion, stale observations)
12	Main benchmark M1–M5 × Easy/Medium/Hard/Dynamic

---

Install

python -m venv .venv
# Windows Git Bash / Linux / macOS
source .venv/bin/activate
pip install -e ".[dev]"

Copy .env.example to .env only if you use a real OpenAI-compatible or local VLM endpoint. Mock VLM needs no credentials.

React showcase (local)

# Optional: refresh static benchmark + episode frames
PYTHONPATH=src python scripts/export_demo_assets.py

cd demo
npm install
npm run dev

Routes: / landing · /docs concepts & math · /results charts · /try interactive examples (/play redirects to /try).

Live: https://activecrop.vercel.app
Linked to github.com/AvoCahDoe/ActiveCrop — pushes to main redeploy automatically (Vercel root directory: demo/).

# Manual production deploy from demo/ (optional)
npx vercel --prod

Re-run export_demo_assets.py and push before expecting chart/episode updates on Vercel.

---

Quick start

Generate a scene (single view)

python -m activecrop.simulation.generate_scene
python -m activecrop.simulation.generate_scene --viewpoint LEFT --seed 42

Generate all viewpoints

python -m activecrop.simulation.generate_scene --all-viewpoints --seed 42 --name scene_multiview

Viewpoints: FRONT, LEFT, RIGHT, FRONT_LEFT, FRONT_RIGHT, ABOVE, CLOSE_LEFT, CLOSE_RIGHT.

Run Mock VLM analysis

python -m activecrop.vlm.analyze --seed 42 --save-scene --name scene_vlm_demo

Default provider is mock (configs/models/default.yaml). Set provider: openai or local for real endpoints.

Evaluations

python -m activecrop.evaluation.evaluate_baseline --n-trials 20 --name baseline_demo
python -m activecrop.evaluation.evaluate_intervention --n-trials 20 --name intervention_demo
python -m activecrop.evaluation.evaluate_random_view --n-trials 10 --name random_view_demo
python -m activecrop.evaluation.evaluate_entropy_view --n-trials 5 --name entropy_view_demo
python -m activecrop.evaluation.evaluate_vlm_view --n-trials 3 --name vlm_view_demo
python -m activecrop.evaluation.evaluate_activecrop --n-trials 3 --name activecrop_demo

Artifacts land under experiments/results/<id>/ (config.yaml, metrics JSON, prediction/trajectory JSONL as applicable).

CLI entry points

Console script	Module
activecrop-generate-scene	activecrop.simulation.generate_scene
activecrop-analyze	activecrop.vlm.analyze
activecrop-evaluate-baseline	activecrop.evaluation.evaluate_baseline
activecrop-evaluate-intervention	activecrop.evaluation.evaluate_intervention
activecrop-evaluate-random-view	activecrop.evaluation.evaluate_random_view
activecrop-evaluate-entropy-view	activecrop.evaluation.evaluate_entropy_view
activecrop-evaluate-vlm-view	activecrop.evaluation.evaluate_vlm_view
activecrop-evaluate-activecrop	activecrop.evaluation.evaluate_activecrop

---

Output contract (React-ready)

Single view (default / --viewpoint):

experiments/runs/scene_<timestamp>/
  rgb.png
  depth.png
  depth.npy
  camera.json
  ground_truth.json
  scene_meta.json
  masks/
    tomato_<id>.png

All viewpoints (--all-viewpoints):

experiments/runs/scene_<timestamp>/
  scene_meta.json
  views/
    FRONT/{rgb.png,depth.png,depth.npy,camera.json,ground_truth.json,masks/}
    LEFT/...

VLM (analyze --save-scene): adds predictions.json next to scene artifacts.

---

Tests

pytest

---

Citation

If you use ActiveCrop in academic work, please cite:

@software{ActiveCrop2026,
  author    = {El Boubkraoui, Farid},
  title     = {{ActiveCrop}: Uncertainty-Aware Active Perception for Safe VLM-Based Robotic Crop Intervention},
  year      = {2026},
  version   = {0.1.0},
  url       = {https://github.com/AvoCahDoe/ActiveCrop},
  license   = {MIT}
}

Machine-readable citation metadata: CITATION.cff.

---

Author

Farid El Boubkraoui
Email: farid.elboubkraoui@w-ays.de
GitHub: @AvoCahDoe

---

License

This project is licensed under the MIT License.