Systems for SemEval-2026 Task 13: Detecting Machine-Generated Code with Multiple Programming Languages, Generators, and Application Scenarios. We develop and compare high-capacity neural encoders against lighter, interpretable pipelines across all three subtasks, with a shared focus on robustness to distribution shift and class imbalance.
A single corpus is projected onto three classification tasks of increasing granularity. The primary metric for all subtasks is Macro-F1.
| Subtask | Problem | Classes |
|---|---|---|
| A | Binary detection | human vs. machine |
| B | Generator-family attribution | human + 10 LLM families (11-class) |
| C | Mixed-source & adversarial detection | human, machine, hybrid, adversarial (4-class) |
A central finding shapes the evaluation: strong in-domain validation does not transfer to the official test set. Subtasks A and B are evaluated under heavy language/generator shift (near open-set), while Subtask C is closed-set (no shift), so its validation is a reliable proxy for test.
- CodeBERT (base) — fine-tuned end-to-end for binary classification (max 256 tokens).
- Frozen UniXcoder + LR — UniXcoder as a frozen feature extractor with a logistic-regression head.
- Structural LightGBM — 30+ hand-engineered stylistic/structural features (indentation autocorrelation, naming-convention ratios, keyword density, empty-line ratio, character entropy), per-language z-score normalized. Best out-of-distribution system.
- Skeleton Sparse — multi-view TF-IDF + SGD over three abstracted views (decommented / delexicalized / punctuation-only), fused with fixed weights.
- Direct multiclass baseline — CodeBERT (base) → 11-class distribution, with downsampling of the human majority.
- Hierarchical pipeline — UniXcoder binary filter (human vs. machine) → AI-only multiclass stage (GraphCodeBERT / UniXcoder). Trained with Focal Loss + auxiliary Supervised Contrastive Loss; sliding-window training, middle-truncation validation.
- CodeBERT baseline — at 256 and 512 tokens (context-window ablation).
- Sliding-window baseline — 512-token windows, 256 stride, mean-pooled (negative result; see below).
- Balanced pipeline — two-level capping by generator–language pair and by label (adapted from arXiv:2604.21365).
- ModernBERT (base, 1024 tokens) — native long-context encoder. Final system for Subtask C.
The Val-ID ranking is essentially reversed on the test set. CodeBERT reaches the highest in-domain score (0.973) but collapses under shift, predicting machine for ~90% of test samples. The Structural LightGBM is the most robust, with the best OOD-proxy Macro-F1 (0.847). No single model wins across all eight languages. (Full per-model / per-language table in the paper.)
| Stage | Model | Macro-F1 |
|---|---|---|
| Binary filter (human vs. machine) | UniXcoder | 0.9050 |
| AI-only multiclass (10 families) | GraphCodeBERT / UniXcoder | 0.60–0.61 |
Human code is cleanly separable, but family attribution stalls: once human code is removed, the family embeddings collapse into a single overlapping space (homogenization + minority-class underrepresentation).
| Model | Max len | Test Macro-F1 |
|---|---|---|
| CodeBERT baseline | 256 | 0.5175 |
| CodeBERT baseline | 512 | 0.5398 |
| CodeBERT sliding-window | 512 (256 stride) | 0.5086 |
| Balanced pipeline (CodeBERT) | 512 | below baseline |
| ModernBERT (final) | 1024 | 0.5668 |
| ModernBERT + label smoothing 0.1 | 1024 | 0.5649 |
The controlled 256→512→1024 window sweep localizes the gain on the hybrid and adversarial classes; tiling the snippet into independently labelled windows corrupts exactly those composite classes. Label smoothing yields no measurable benefit.
E. Bernacchi · N. Gutierrez · C. Matarazzo · G. Taddia · L. Tufo