An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in breast cancer treatment

We developed a Multi-modal Response Prediction (MRP) system for instant prediction in breast cancer treatment with Neoadjuvant Therapy (NAT). Using longitudinal multidisciplinary data from 3,352 breast cancer patients, MRP mimics physician assessments, aided by a cross-modal learning module for facilitating the MRP practicality in multi-center hospitals. Validated across centers and reader studies, MRP exhibited comparable robustness and generalizability to breast radiologists, significantly outperforming humans in pCR prediction in the Pre-NAT phase. Clinical utility assessment shows MRP's potential: reducing treatment toxicity by 35.8% in Pre-NAT and preventing surgeries by 16.7% in Post-NAT, without mispredictions. Our work enhances AI applications in personalized treatment decisions.

Neoadjuvant Therapy Pathway

Model system

Environment Setup

Start by installing PyTorch 1.8.1 with the right CUDA version, then clone this repository and install the dependencies.

$ conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=11.1 -c pytorch
$ pip install git@github.com:yawwG/MRP.git
$ conda env create -f environment.yml

Code Description

This codebase has been developed with python version 3.7, PyTorch version 1.8.1, CUDA 11.1 and pytorch-lightning 1.5.9. Example configurations for MG-based and MRI-based classification can be found in the ./configs. All training and testing are done using the run.py script. For more documentation, please run:

python run.py --help

The preprocessing steps for each dataset (MRI and Mammogram) can be found in datasets/image_dataset.py The dataset using is specified in config.yaml by key("dataset").

Train and Test

Training the MRI and Mammogram based model for pCR prediction with the following command:

python run.py -c imrrhpc.yaml --train --test
python run.py -c imgrhpc.yaml --train --test

Multimodal contribution analysis

To ensure that the model's interpretability and predictable performance, we explicitly demonstrate the contribution of multi-modalities in the model's training.

python ./utils/contribution.py

Decision curve analysis analysis

We explored in two specific clinical scenarios: personalizing Pre-/Mid-NAT management of non-pCR patients to avoid toxic therapy and optimizing Post-NAT management of pCR patients to reduce unnecessary surgeries.

python ./utils/dca_analysis.py

Contact details

If you have any questions please contact us.

Email: ritse.mann@radboudumc.nl (Ritse Mann); taotanjs@gmail.com (Tao Tan); y.gao@nki.nl (Yuan Gao)

Links: Netherlands Cancer Institute, Radboud University Medical Center, Maastricht University, St Joseph’s Healthcare Hamilton and The University of Hong Kong