This project is to use a data-driven learning-based approach to improve the performance of the scooping technique. I design a neural network to predict a set of parameters given the RGB-D image of the bin scenario. There are in total 5 parameters of scooping: the finger contact position p, the angle between the finger direction and the vertical direction (i.e., the pitch angle) alpha, the horizontal orientation of the gripper rotated around the vertical direction (i.e., the yaw angle) beta, the gripper aperture, and the finger length differecne. So far, I considered two parameters: the finger contact position p and the yaw angle beta. Other parameters are set as a fixed value.
This dataset includes a) RGB heightmap images showing the real scene of objects, b) 2-D arrays for ground truth labels. The groud truth label is of the same size as the RGB heightmap, and obtained following our model-based scooping method. Each point in the label is associated with a class that represents for a background or successful/failed scooping. Here shows an example (grey: good, black: bad, white: background):
Other data can be found via:
We also collect the data by the robot performs trial and errer in real experiments. The ground truth label is obtained by a successful/failed scooping in the real experiment. Some data can be found in
We use a Realsense SR300 camera to get the RGB image and the depth image. Then, we combine the RGB image and the depth image to make the RGB-D heightmap. A heightmap is an RGB-D image obtained from a 3D point cloud, describing the 3D information of the bin scenario. Each pixel in the heightmap is in linear relation to its horizontal position in the world frame and corresponds to a value indicating the height-from-bottom information.
python utils/heightmap.py
-Input: RGB heightmap and depth heightmap -Output: The dataset for the pre-train model
python collecting_data_annotating_20210503.py
python data_collection_20210419.py
-Input: Dataset that includes heightmap images and ground truth label -Output: Model weights
training_network.ipynb
See the video video Number of training data: 11016 Success rate: 67.5%