DeepGrasp: Modeling digital humans with task-driven learning

Bartlomiej Borzyszkowski
[bartlomiej.borzyszkowski@epfl.ch]

School of Computer and Communication Sciences
Swiss Federal Institute of Technology Lausanne (EPFL)

Task-driven learning with deep neural networks is a powerful, emerging approach to elucidate and model computations in the brain. This work focuses on human-object interaction and aims to better understand, predict, and model human behaviour using digital avatars. We train a Graph Convolutional Network (GCN) with spatial and temporal attention on multi-modal data, including body pose and touch signals. Our results show that the model learns a robust neural representation capable of solving multiple classification and regression tasks in 3D space, such as action recognition and object classification. The results demonstrate that task-driven multi-modal learning can capture meaningful structure in human grasping behavior, offering a step toward more accurate models of the sensorimotor system and advancing research in computational neuroscience.

Refer to the corresponding publication for more details: [link]

Installation

The software has been tested on Ubuntu 22.04 with CUDA 12.4 and Python3.10.

First, install the Python Development Headers:

sudo apt-get install python3.10-dev

We recommend installing further dependencies in the virtual environment:

python3 -m venv venv_DeepGrasp
source venv_DeepGrasp/bin/activate

The project uses PyTorch Geometric Temporal, extension library of PyTorch for Temporal Graph Neural Networks (T-GCNs) with a GPU acceleration. As a first step, install PyTorch, PyTorch Geometric and PyTorch Geometric Temporal for your CUDA Version following official instructions. Recommended versions are:

• PyTorch: torch-2.4.1+cu124 torchaudio-2.4.1+cu124 torchvision-0.19.1+cu124
• PyTorch Geometric: torch-geometric-2.6.1 pyg_lib-0.4.0+pt24cu124
• PyTorch Geometric Temporal: torch-geometric-temporal-0.54.0

Finally, install the remaining project requirements:

pip install -r requirements.txt

Download dataset

This work is based on the GRAB dataset. Follow steps in the source repository to download the dataset.

• Download GRAB (ZIP files) from the source website. Please do NOT unzip the files yet.

• Make sure to get access to the object_meshes.zip before continuing to the next steps.

• Put all the downloaded ZIP files for GRAB in your destination directory.

• Run the following command to extract the ZIP files:

  python3 -m tools.unzip_grab  --grab-path $PATH_TO_FOLDER_WITH_ZIP_FILES \
                               --extract-path $PATH_TO_EXTRACT_GRAB_DATASET_TO

• The extracted data should be in the following structure:

      GRAB
      ├── grab
      │   │
      │   ├── s1
      │   └── ...
      │   └── s10
      │
      └── tools
          ├── object_meshes
          └── object_settings
          └── subject_meshes
          └── subject_settings
          └── smplx_correspondence
  

Download body models

This work is based on the 3D body models SMPL-X and MANO that capture detailed body, face, and hand movements in a unified framework. SMPL-X incorporates facial expressions and articulated fingers, allowing for expressive and naturalistic human representations. The models are widely used in computer vision, graphics, and deep learning, enabling realistic human pose estimation and motion synthesis from images, videos, or motion capture data.

• Download the SMPL-X model from the original source: https://smpl-x.is.tue.mpg.de

• Download the MANO model from the original source: https://mano.is.tue.mpg.de

• Extract the ZIP files with models in your desired directory.

• The extracted data should be in the following structure:

      models
      ├── smplx
      │   │
      │   ├── SMPLX_FEMALE.pkl
      │   └── SMPLX_MALE.pkl
      │   └── ...
      │
      └── mano
          ├── MANO_RIGHT.pkl
          └── MANO_LEFT.pkl
          └── ...

Start the project

• Data preprocessing:
  

  Modify desired configuration parameters at configs/preprocessing_cfg.yml.

  python3 run_preprocessing.py  --grab-path $PATH_TO_EXTRACTED_GRAB_DATASET \
                                --model-path $SMPLX_MODEL_FOLDER \
                                --out-path $PATH_TO_SAVE_PREPROCESSED_DATASET

• Training:
  

  Modify desired configuration parameters at configs/training_cfg.yml.

  To track your ML experiments via neptune.ai, export env variables NEPTUNE_API_TOKEN & NEPTUNE_PROJECT for your Neptune project. If not specified, tracking is done offline.

  python3 run_training.py  --work-dir $TRAINING_DIRECTORY \
                           --data-path $PATH_TO_PREPROCESSED_DATASET \
                           --expr-ID $ID_OF_YOUR_EXPERIMENT

  Detailed results will be available in your work_dir and can be inspected as TensorBoard and Neptune artifacts.

Contact and technical support

• Bartlomiej Borzyszkowski
  Mail: bartlomiej.borzyszkowski@epfl.ch
  Web: borzyszkowski.github.io