Cell Nucleus Segmentation with Deep Learning

Cell Nucleus Presentation - Ronald Domi

Overview

This project implements deep learning models for binary cell nucleus segmentation in histopathology images. The goal is to automatically detect and segment cell nuclei across multiple tissue types using the PanNuke dataset.

What is Image Segmentation?

Image segmentation is a computer vision task that assigns a label to every pixel in an image. In this project, we perform binary segmentation to classify each pixel as either:

• Nucleus (foreground)
• Background

Dataset: PanNuke

The PanNuke dataset is a large-scale pan-cancer histology dataset containing:

• 19 tissue types: Breast, Colon, Bile-duct, Esophagus, Uterus, Lung, Cervix, Head & Neck, Skin, Adrenal Gland, Kidney, Stomach, Prostate, Testis, Liver, Thyroid, Pancreas, Ovary, Bladder
• 256x256 image patches with corresponding segmentation masks
• 6 nuclei categories: Neoplastic, Inflammatory, Connective/Soft tissue, Dead cells, Epithelial, Background
• 7,000+ training patches across 3 folds

For this project, masks are converted to binary format (nucleus vs. background).

Models

Simple U-Net

A lightweight encoder-decoder architecture with skip connections:

• 3 encoder blocks with max pooling
• 3 decoder blocks with transposed convolutions
• Skip connections to preserve spatial information

Attention U-Net

Enhanced U-Net with attention gates that help the model focus on relevant features:

• 4 encoder blocks + bottleneck
• Attention gates on skip connections
• Better performance on complex segmentation tasks

Loss Function

Combined loss for handling class imbalance:

• Binary Cross-Entropy (BCE): Standard pixel-wise classification loss
• Dice Loss: Overlap-based loss for better segmentation boundaries
• Combined: 50% BCE + 50% Dice

Training Configuration

• Epochs: 20
• Batch size: 64
• Learning rate: 1e-3
• Optimizer: Adam
• Device: CUDA (GPU) when available

Project Structure

Bio394/
├── simple_model.py          # Simple U-Net implementation
├── attention_unet_model.py  # Attention U-Net implementation
├── train_models.ipynb       # Training notebook
├── analysis.ipynb           # Analysis and visualization
├── Fold 1/                  # PanNuke dataset (not in repo)
│   ├── images/
│   └── masks/
└── checkpoints/             # Saved model weights (not in repo)

Results

Training on 500 images for 20 epochs shows convergence of both training and validation loss, demonstrating the model's ability to learn nucleus segmentation patterns.

References

1. Gamper, J., et al. "PanNuke Dataset Extension, Insights and Baselines." arXiv:2003.10778 (2020)
2. Gamper, J., et al. "PanNuke: An open pan-cancer histology dataset for nuclei instance segmentation and classification." European Congress on Digital Pathology (2019)
3. Ronneberger, O., et al. "U-Net: Convolutional Networks for Biomedical Image Segmentation." MICCAI (2015)
4. Oktay, O., et al. "Attention U-Net: Learning Where to Look for the Pancreas." MIDL (2018)