[POC] Phase 1.5: Add Classification Head Support for ModernBERT-base-32k

[henschwartz]

Overview

Add classification head support for ModernBERT-base-32k. Phase 1 integrated the base model successfully, but classification requires either fine-tuning or a fine-tuned classifier model.

Related: Issue vllm-project#995, POC Document: docs/issue-995-modernbert-32k-poc.md
Depends on: Issue #3 (Phase 1 - Base Model Integration)

Current Status

• ✅ Base model loads successfully (Phase 1 complete)
• ✅ Model processes sequences up to 32K tokens (base model forward pass)
• ❌ Classification head missing (model is base MLM, not classifier)

Problem

The llm-semantic-router/modernbert-base-32k model is a base MLM model (ModernBertForMaskedLM) without classifier weights. To use it for classification tasks (PII detection, intent classification, jailbreak detection), we need:

1. A fine-tuned classifier model, OR
2. Fine-tune the base model with a classification head, OR
3. Create a new classification head and train it

Tasks

• Research options for classification head:
  • Search for fine-tuned 32K classifier models on HuggingFace
  • Evaluate fine-tuning approach (dataset, training pipeline)
  • Evaluate creating new classification head approach
• Choose approach based on research
• If fine-tuning:
  • Prepare training dataset
  • Set up training pipeline
  • Fine-tune base model with classification head
  • Evaluate fine-tuned model
• If finding model:
  • Download fine-tuned classifier model
  • Test model loading and inference
  • Verify classification accuracy
• If creating new head:
  • Implement classification head creation
  • Train classification head
  • Evaluate trained model
• Test classification inference on sample sequences
• Document classification approach and results

Deliverables

• Classification head support implemented
• Classification inference works on sample sequences
• Test results documented (accuracy, latency, memory usage)
• Approach documented (fine-tune vs. find model vs. create head)

Acceptance Criteria

• Classification head support added (via fine-tuning, fine-tuned model, or new head)
• Classification inference works on sample sequences
• Classification accuracy verified (if applicable)
• Performance metrics documented
• Approach documented