This repository is dedicated to the development of a table tennis robot. Our goal is to create a fully automated robot capable of playing ping pong with humans, leveraging the characteristics of deep learning LSTM architecture. We aim to achieve this on lower-tier, cost-effective devices. The challenge involves addressing the following tasks:
-
Camera Localization: Implementing camera-based localization for tracking the ping pong ball.
-
Ping Pong Ball Detection: Developing algorithms for detecting the ping pong ball in real-time.
-
Trajectory Prediction Model and Training: Building a trajectory prediction model based on LSTM.
-
Mechanical Design and Control: Designing the robot's mechanical components and implementing control mechanisms (using raspberrypi).
This project is mainly run on Ubuntu. Things will not perform as expected if you are not running from a Debian based OS
First, clone the project using git or just download the zip file and extract it. And you will have to install the nessesary python package in requirements.txt with pip3 or whatever you want. For example:
pip3 install -r requirements.txtDetect the table tennis ball by its color, contour, and movement
1. Original frame
2. Binarize with HSV color threshold
3. Bounding box and merge
4. Considered area
5. Last 20 frames of considered data
6. Newly appeared object (orange object)
7. Chose the closest one to the previous result
-
-cc, --create_config : Use this flag to create config, otherwise use config to run detection
-
-c, --config : config name (nessesary)
-
-s, --source : camera source (nessesary)
-
-f, frame_size : frame size (only needed when create config, default is 640x480)
-
-r, --frame_rate : frame rate (only needed when create config, default is 30)
-
-cr, --color_range : color range file name (only needed when create config, default is cr3)
-
-i, --inmtx : camera intrinsic matrix file name (only needed when create config in 3d setup)
-
--non_3d_setup : use this flag to setup camera without 3d setup
Predict the ball by deep learning model.
Homography matrix frome pixel plane to table
Homogeneous transformation
-
Get the world coordinates of the camera position
-
Get the two-dimensional coordinates of table tennis ball in the frame
-
Performing a homography transformation on the coordinates of the table tennis ball
-
Consider the transformation coordinates as the projection point coordinates of the ball
-
Calculate the equation of the line using camera coordinates and porjection point coordinates
This script is designed for simulating ball data with various configurations. And save as training data files to train the ball-prediction model. It provides command-line arguments to customize simulation parameters and operations.
- --GUI : Enable GUI mode (default: False).
- -l : Length of the simulated data (default: 1000).
- -n : Name for the output files (default: "nb60test"), which will be stored at /ball_simulate_v2/dataset.
- --mode : Simulation mode ("default", "fit", "ne", "predict", "normal", "normalB", "normalB60", "normalBR") (default: "normalB60").
- --merge : Merge two datasets into one (default: False).
- --merge_a : File name of the first dataset to merge (default: "train.bin").
- --merge_b : File name of the second dataset to merge (default: "train.bin").
Simulate data with recommend length and name:
python3 ball_simulate_v2/simulate.py -l 1000000 -n medium --mode normalBRIn the study, we use the length of 1000000 for the medium dataset and 10000000 for the huge dataset. Meanwhile, we chose the mode normalBR to process the simulation as the most realistic mode.
- fit : Set the simulation mode to fitting.
- ne : Set the simulation mode to no error.
- predict : Set the simulation mode to prediction.
- normal : Set the simulation mode to normal.
- normalB : Set the simulation mode to normal with additional configuration 'B'.
- normalB60 : Set the simulation mode to normal with configuration 'B60'.
- normalBR : Set the simulation mode to normal with configuration 'BR'.
This script is designed for training and testing a ball simulation model using various configurations. The script provides command-line arguments to customize training parameters, such as learning rate, batch size, epochs, model type, and more.
Make sure you run the code in the root of the project
python3 ball_simulate_v2/train.py [args]- -lr, --learning_rate : Learning rate for the training process (default: 0.001).
- -b, --batch_size : Batch size for training (default: 64).
- -e, --epochs : Number of epochs for training (default: 30).
- -m, --model_type : Type of the model (default:
medium; availible values:small,medium,medium_var,big,large). - -mom, --momentum : Momentum for the
SGDMoptimizer (default: 0.01). (SGDM ONLY) - -d, --dataset : Dataset name.
- -s, --scheduler_step_size : Step size for the learning rate scheduler (default: 0).
- -w, --weight : Path to the pre-trained model weight file (default: None).
- -n, --name : Save name for the training.
- -o, --optimizer : Optimizer type (default:
adam; avalible values:adam,sgdm). - --num_workers : Number of workers for data loading (default: 0). This may speed up the training. Setting to 2 is a good choice.
- --test : Test the model on the validation set (default: False).
- --mode : Training mode (
default,fit,ne,predict,normal,normalB,normalB60,normalBR) (default:normalBR). More details can be found in Modes or in /core/Constants.py
- fit : Set the model mode to fitting.
- ne : Set the model mode to no error.
- predict : Set the model mode to prediction.
- normal : Set the model mode to normal.
- normalB : Set the model mode to normal with additional configuration 'B'.
- normalB60 : Set the model mode to normal with configuration 'B60'.
- normalBR : Set the model mode to normal with configuration 'BR'.