[POC] Phase 1: Integrate ModernBERT-base-32k model loading and basic inference

[henschwartz]

Overview

Integrate ModernBERT-base-32k model from HuggingFace into the Rust codebase. This is Phase 1 of the POC for extending context window from 512 to 32,768 tokens.

Related: Issue vllm-project#995, POC Document: docs/issue-995-modernbert-32k-poc.md

Model Information

• HuggingFace Model: llm-semantic-router/modernbert-base-32k
• Context Window: 32,768 tokens (64x increase from 512)
• Architecture: ModernBERT with RoPE
• Model Card: https://huggingface.co/llm-semantic-router/modernbert-base-32k

Tasks

• Download llm-semantic-router/modernbert-base-32k from HuggingFace
• Test loading with TraditionalModernBertClassifier::load_from_directory()
• Verify 32K context support (check config)
• Test basic inference on sample sequences (512, 1K, 8K, 16K, 32K tokens)
• Document any issues or needed adjustments

Deliverables

• Model loads successfully from HuggingFace
• Basic inference works on sample sequences
• Test results documented (latency, memory usage, accuracy)
• Any code adjustments documented

Acceptance Criteria

• ModernBERT-base-32k model loads successfully
• Model processes sequences up to 32K tokens
• Model maintains performance on short sequences (512 tokens)
• No breaking changes to existing API

Files to Modify

• candle-binding/src/model_architectures/traditional/modernbert.rs
• Model loading code
• Configuration files

[henschwartz]

/assign