Skip to content

fix: make classification transformers module runnable on modern dependencies and fix evaluation pipeline#207

Open
DhruvrajSinhZala24 wants to merge 1 commit intoML4SCI:mainfrom
DhruvrajSinhZala24:fix/191-classification-transformers-runtime
Open

fix: make classification transformers module runnable on modern dependencies and fix evaluation pipeline#207
DhruvrajSinhZala24 wants to merge 1 commit intoML4SCI:mainfrom
DhruvrajSinhZala24:fix/191-classification-transformers-runtime

Conversation

@DhruvrajSinhZala24
Copy link
Copy Markdown

Problem

The DeepLense_Classification_Transformers_Archil_Srivastava/ module is not runnable on a fresh setup.

Running either train.py or eval.py results in multiple failures due to missing dependencies, outdated APIs, and incorrect evaluation logic. This blocks the full training/evaluation pipeline and makes the module unusable for new contributors.

This affects both:


Root Cause

The failures originate from a combination of dependency gaps, API changes, and logic errors:

  • No module-level requirements.txt, causing import errors (wandb, timm, einops, torchmetrics)
  • torchmetrics API changes (missing required task argument)
  • Deprecated PyTorch usage:
    • torch.has_mps no longer valid
    • CosineAnnealingWarmRestarts(verbose=...) no longer supported
  • Incorrect evaluation logic:
    • micro_auroc computed as mean([]) → NaN
    • Missing softmax(dim=-1) in evaluation
    • Incorrect predictions used for confusion matrix
  • W&B entity hardcoded, preventing external usage

Solution

1. Dependency Fix (Reproducibility)

  • Added module-level requirements.txt
  • Ensures the module can be installed and run independently

2. torchmetrics Compatibility

  • Updated metric calls to match current torchmetrics API
  • Added required task parameter

3. Evaluation Fixes

  • Fixed AUROC computation to avoid NaN (proper micro AUROC handling)
  • Added missing softmax(dim=-1)
  • Corrected prediction handling for confusion matrix

4. PyTorch Compatibility

  • Replaced torch.has_mps with:
    • torch.backends.mps.is_available()
    • torch.backends.mps.is_built()
  • Removed unsupported verbose=True from scheduler

5. W&B Configuration

  • Added --entity argument (defaults to $WANDB_ENTITY)
  • Added --run_id support
  • Removed hardcoded entity dependency

6. Documentation

  • Updated README.md with setup instructions:
    • pip install -r requirements.txt
    • Usage of --entity

Changes Made

  • requirements.txt → added missing dependencies
  • eval.py
    • Updated torchmetrics usage
    • Fixed AUROC computation
    • Added softmax(dim=-1)
    • Fixed confusion matrix predictions
    • Added configurable W&B arguments
  • train.py
    • Removed deprecated scheduler argument
    • Added --entity
  • utils.py
    • Updated MPS device detection
  • README.md
    • Added setup + usage instructions

Validation

  • Verified syntax using:
    python -m compileall -q .
  • Ran synthetic evaluate() test:
    • No runtime errors
    • AUROC and accuracy computed correctly
  • Confirmed training loop initializes without scheduler errors
  • Verified MPS fallback behavior

Impact

  • Restores full training + evaluation pipeline for this module
  • Makes the module runnable on modern environments
  • Fixes incorrect evaluation outputs (NaN AUROC)
  • Improves usability for new contributors

Closes #191
Closes #192

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant