Hand-tracking Dobot examples

This folder contains two example scripts that use MediaPipe hand-tracking to control a Dobot Magician robot with simple hand gestures via your webcam.

Files

• hand_tracking_rotational.py — mode-based control where different finger-counts switch which robot joint/axis is controlled (rotational control). Also toggles the suction cup.
• hand_tracking_translational.py — direct translational control: open/closed hand toggles which axes control (X/Y/Z) and finger counts toggle suction.

Overview

Both scripts use MediaPipe to detect a single hand from the webcam, convert landmark positions to simple boolean "finger up" flags, and map gestures to Dobot motions using the magician object provided by the Dobot SDK (imported via from DobotEDU import *). The camera's palm center is used to compute offsets that are converted into small position deltas for the robot.

Requirements

• Python 3.8+ (3.10+ recommended)
• A working webcam
• Dobot Magician hardware (or simulator) and Dobot Link available on the Dobot Lab platform.
• Python packages:
  • opencv-python
  • mediapipe
  • numpy

Install (example, Windows PowerShell)

Open Dobot Lab on the browser and navigate to Python Lab:

python -m pip install --upgrade pip
pip install opencv-python mediapipe numpy

Note: The Dobot Python package (or the module named DobotEDU) may already have opencv-python and mediapipe already installed.

How to run

Copy the code into python lab and run the code.

Controls / Gesture mapping

These mappings are implemented in each script. The README below summarizes the behavior in each file.

hand_tracking_rotational.py

• 1 finger up: switch to mode 1 (controls joint 1 / rotation)
• 2 fingers up: switch to mode 2 (controls joint 2)
• 3 fingers up: switch to mode 3 (controls joint 3)
• 4 fingers up: toggle suction cup on/off (requires you to hold 4 fingers upright for ~2 seconds; the script debounces by time)
• Close your fist and move your hand left/right to rotate/move the selected joint/axis according to the current mode
• The code clamps coordinates to safe ranges before commanding motion

hand_tracking_translational.py

• Hand open (not fist): camera X (left/right) → Dobot Y; camera Y (up/down) → Dobot Z (inverted to match robot coordinate system)
• Hand closed (fist): camera X → Dobot X
• 2 fingers up: suction ON
• 3 fingers up: suction OFF
• Coordinate clamping and a small movement threshold are used to reduce jitter

Known / likely issues and runtime gotchas

• Missing Python packages: if mediapipe, opencv-python, or numpy are not installed the script will crash at import time.
• Camera permission / busy camera: If another app is using the camera, cv2.VideoCapture(0) may fail or return no frames.
• Safety / workspace limits: the scripts move the physical robot. Before running, verify the clamped ranges in each script match a safe workspace for your robot and end effector. Unexpected moves can cause collisions.
• Jitter & repeated commands: The scripts reduce jitter with thresholds, but you may still see repeated small moves. Consider reducing sensitivity or increasing the movement threshold.

Safety notes (important)

• Always run the robot with an empty workspace or with emergency stop within reach while testing. Start with very low sensitivity values and small safe coordinate ranges.
• Test movement logic without the end effector attached first. Use dry runs where the robot is moving in the air to confirm axes map the way you expect.

Troubleshooting

• "ImportError: No module named 'mediapipe'" → run pip install mediapipe.
• "NameError: name 'magician' is not defined" → ensure the DobotEDU module is present and that it creates a magician object on import; alternatively create/instantiate the Dobot controller before running the main loop in each script.
• Camera returns black frames or ret is False → try a different camera index (change VideoCapture(0) to VideoCapture(1) or higher) or close other apps using the camera.

Development / testing tips

• Run the script while watching the console to see printed mode and debug output. The scripts already print debug messages when modes change or when the hand is lost.
• Use a small sensitivity and small movement ranges for initial testing.

Acknowledgements

• MediaPipe for hand landmark detection
• OpenCV for camera capture and UI

Author

Ashvin Lobo
Project: Hand Gesture Controlled Dobot Magician
Language: Python 3
Version: *1.0