AgenticLab Human

AgenticLab Human is a humanoid upper-body manipulation repo for turning a voice or text instruction into a robot execution flow:

voice/text task -> VLM planner -> ActionSequence
  -> RGB-D capture -> YOLO detection -> GraspNet grasp candidates
  -> X5 HTTP robot execution

The current hardware path focuses on the X5 humanoid upper body with an Orbbec RGB-D camera, a fine-tuned closed-set YOLO detector, and a separate GraspNet HTTP service.

Start Here

The main setup and operation guide is:

• docs/main_tutorial.md

That tutorial is the primary entry point for installation, module order, and the basic end-to-end workflow. This README stays intentionally short and points to the module documents when more detail is useful.

Main Modules

• Voice input and speech-to-text: docs/voice/voice_module.md
• Planning adapter and VLM/Fast Downward task parsing: src/agenticlab_human/planning/
• X5 HTTP client/server and robot backend: docs/X5/client_server.md
• GraspNet HTTP service: docs/grasp/grasp_client_server.md
• No-planning X5 execution pipeline: docs/pipeline/execution_without_planning.md
• YOLO training and deployment notes: docs/perception/yolo_train.md

Quick Commands

Install the base package from the repo root:

python -m pip install -e .

Run the packaged speech-to-text demo:

python -m agenticlab_human.voice.demo_speech_to_text

Run the planner adapter with explicit text and a static image:

python -m agenticlab_human.planning.voice_to_planner \
  --task-text "put the apple into the blue box" \
  --image-path data/data_for_test/task_parser/01_sort1_color.png

Start the X5 server on the robot/server PC:

python -m agenticlab_human.execution.robot.x5.server \
  --config configs/robot/x5_config.yaml

Capture one RGB-D frame from the X5 server:

python -m agenticlab_human.execution.robot.x5.client \
  --server-url http://192.168.1.15:8000 \
  --save-dir output/x5_http_captures/local

Start the GraspNet service from the GraspNet-capable environment:

OMP_NUM_THREADS=12 PYTHONPATH=src python -m \
  agenticlab_human.perception.grasping.server \
  --config configs/perception/graspnet_config.yaml \
  --camera-config configs/perception/camera_config.yaml \
  --camera-name Gemini335 \
  --checkpoint data/checkpoints/minkuresunet_realsense.tar \
  --device cuda:0 \
  --host 127.0.0.1 \
  --port 8010

Run a real X5 pick-and-place pipeline:

python -m agenticlab_human.execution.pipeline pipeline \
  --object number_block_3 \
  --target yellow_bin \
  --config configs/execution/x5_pipeline.yaml \
  --execute

Run a saved action sequence loop:

python -m agenticlab_human.execution.pipeline_loop \
  --plan output/execution/<run_id>/action_sequence.json \
  --max-loops 3 \
  --execute

Current Status

The repo can run X5 pick-place and pick-place-pour style demos through the HTTP robot backend. Known remaining limits are mostly physical execution issues: grasp pose reachability, IK feasibility, calibration quality, and scene-specific detector/grasp robustness.