Efficient LLM Preference Classification Through Position Bias Mitigation and Architectural Symmetry

Official reproducibility repository for the manuscript:

"Efficient LLM Preference Classification Through Position Bias Mitigation and Architectural Symmetry"
Applied Intelligence (Springer Nature), Manuscript ID: APIN-D-26-05359.

---

Architecture Overview

This repository provides the end-to-end experimental pipeline, the data-partitioning protocol, the training and analysis code, and the aggregated per-configuration metrics behind the tables and statistical tests reported in the paper.

Status of the released results. All 42 experimental runs across Blocks A, B, C, and D are complete and verified across three seeds each. results/paper_results.json covers the full factorial experiment, data volume control, all four backbones, multi-turn ablation, and the complete four-point data-efficiency curve. Run python3 reproduce_tables.py --strict to verify the complete suite (exits 0). The release includes the per-run prediction arrays (predictions/*.npz) for all 42 runs.

---

Key Highlights & Findings

• Leakage-Free Partitioning: 57,477 multi-turn LMSYS Chatbot Arena interactions partitioned strictly by id prior to augmentation into disjoint subsets: Train pool (44,477), Dev (2,000), Calibration (3,000), and Evaluation (8,000). Split disjointness is verified at runtime.
• Position Bias Invariance: Training-time swap augmentation elevates position-flip consistency from 0.825 to 0.920. A duplicate-augmentation volume control achieves only 0.815, proving the gain stems from positional symmetry rather than data volume.
• Super-Additive Synergy: Neither swap augmentation nor Siamese difference fusion alone improves log loss; only their combination achieves significant log-loss reduction, yielding an interaction of $-0.0044$ (95% CI $[-0.0064, -0.0023]$) across three random seeds ($p < 0.001$).
• Strict Calibration Separation: Post-hoc temperature calibration ($T = 1.647$) is fitted exclusively on the 3,000-sample calibration split and evaluated on the 8,000-sample test set, reducing ECE from 0.0719 to 0.0094.
• Cross-Dataset Transfer: Zero-shot evaluation on MT-Bench Human Judgments achieves 51.40% accuracy against a 38.54% majority floor without dataset-specific fine-tuning.

---

Repository Structure

.
├── README.md                      # Documentation & reproduction guide
├── LICENSE                        # MIT License
├── requirements.txt               # Python dependencies, pinned to the recorded environment
├── reproduce_tables.py            # Zero-GPU table reproduction; --strict gates on completeness
├── test_local.py                  # CPU test suite: splits, maths, algorithms, cost model
├── lmsys_revision_v2.ipynb        # Primary reproduction notebook (Kaggle / Jupyter ready)
├── notebook_src.py                # Plain-Python pipeline source (jupytext percent format)
├── build_nb.py                    # Regenerates the .ipynb from notebook_src.py
├── check_presets.py               # Pre-flight: resolves/caches all four backbones before training
├── make_figure1.py                # Regenerates Figure 1 from named constants (matplotlib only)
├── job1_ad.slurm                  # HPC job 1: Block A (factorial) + Block D (data efficiency)
├── job2_cb.slurm                  # HPC job 2: Block C (backbones) + Block B (ablation)
├── job_all.slurm                  # HPC single allocation: all 42 runs
├── assets/
│   └── pipeline_diagram.png       # Figure 1, generated by make_figure1.py
└── results/
    ├── paper_results.json         # Aggregated per-configuration metrics, CIs, and environment record
    └── final_tables.tex           # LaTeX tables rendered from those metrics

Directories created at run time and excluded by .gitignore: data/, predictions/, tokcache/, failed/.

---

Quickstart & Reproducibility

1. Installation

git clone https://github.com/Vatsal057/llm-preference-classification.git
cd llm-preference-classification
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Five-Second Reproduction (Zero GPU / Zero Data Download)

Aggregated metrics and bootstrap confidence intervals across seeds 42, 1337 and 2024 are packaged in results/paper_results.json:

python3 reproduce_tables.py            # print the tables
python3 reproduce_tables.py --strict   # additionally assert the run set is complete

This reproduces:

• Matched-condition comparison across the $2 \times 2$ factorial, with the duplicate-augmentation volume control.
• Interaction term $\beta_{\text{int}} = -0.0044$ (95% CI $[-0.0064, -0.0023]$, $p < 0.001$).
• Position-bias analysis: flip consistency $0.825 \to 0.920$, volume control $0.815$, and accuracy conditioned on the true winner's position.
• Post-hoc temperature calibration, global vs per-class, $T = 1.647$, ECE $0.0719 \to 0.0094$.
• Leave-one-out component ablations with paired-bootstrap CIs.
• Zero-shot MT-Bench transfer, 51.40% against a 38.54% majority floor.
• Complete four-backbone comparison (BERT-base, RoBERTa-base, DeBERTa-v3-small, and DeBERTa-v3-extra-small) and the full data-efficiency curve.

Reproduction scope

Running python3 reproduce_tables.py --strict programmatically asserts that all four backbones are present with at least three seeds each, that the data-efficiency curve is complete, and verifies every reported claim, exiting 0.

Per-run prediction arrays (predictions/*.npz, 42 runs) are included in this repository, containing evaluation and MT-Bench class probabilities alongside ground-truth labels.

3. Verify Algorithmic Logic & Invariants (CPU)

Run the full local unit test suite to verify the mathematical and algorithmic invariants (Algorithm 1 greedy multi-turn truncation, leakage-free stratified splitting, swap augmentation balance, temperature fitting, paired bootstrap hypothesis testing, and learning rate schedule):

python3 test_local.py

Note: If train.csv is not present locally, the test suite automatically validates against a deterministic 57,477-row synthetic benchmark, ensuring zero barrier to testing.

---

Dataset Acquisition (For Retraining From Scratch)

If you wish to retrain the models from scratch rather than reproducing from saved prediction runs:

1. Via Kaggle CLI:

   mkdir -p data
   kaggle competitions download -c llm-classification-finetuning -p data/
   unzip data/llm-classification-finetuning.zip -d data/

2. Via Web Browser: Accept competition rules at Kaggle: LLM Classification Finetuning and download train.csv into ./data/.

---

End-to-End Training

The pipeline is resumable: each finished run writes results/<run_id>.json and predictions/<run_id>.npz, and the runner skips any run whose artefacts already exist. A killed session therefore costs only its in-flight run.

The experiment manifest is organised into blocks, selectable by environment variable. All default to on except the optional full-pool scale-up:

Block	Env var	Runs	Purpose	Est. (1 GPU)
A	RUN_BLOCK_A	15	$2 \times 2$ factorial + duplicate volume control	12.7 h
B	RUN_BLOCK_B	6	multi-turn ablation + all-off reference	4.8 h
C	RUN_BLOCK_C	12	four backbones × three seeds	16.0 h
D	RUN_BLOCK_D	9	data-efficiency curve	4.0 h
E	RUN_BLOCK_E	3	optional full-pool scale-up (off by default)	—

Block A underpins every table except the backbone comparison, and Block D's largest point is anchored to Block A's full-model run, so run A before D or C.

On an HPC cluster (SLURM)

# 1. Pre-flight on the login node. Confirms all four backbones resolve BEFORE you spend GPU time.
python3 test_local.py        # CPU only, a few seconds
python3 check_presets.py

# 2. Submit. Two resumable jobs, or one allocation.
sbatch job1_ad.slurm         # Blocks A + D
sbatch job2_cb.slurm         # then Blocks C + B; reuses job 1's runs
# or
sbatch job_all.slurm         # all 42 runs

# 3. Gate on completeness before using any number.
python3 reproduce_tables.py --strict

preset_location() falls back to the bare preset name, so from_preset downloads from Kaggle Hub. The node needs network access plus KAGGLE_USERNAME / KAGGLE_KEY, or pre-staged local preset directories. check_presets.py verifies this up front.

Each job script sets SESSION_BUDGET_H below its --time wall limit so the runner stops launching new runs and still reaches the analysis stage, which is what writes the tables. If the scheduler kills the job first, completed runs survive but no tables are produced.

USE_MULTI_GPU=1 enables data-parallel training and roughly halves wall-clock on two devices. It splits BATCH_SIZE across devices, so measure the speedup with a smoke run before relying on the estimates above.

On Kaggle

1. Import lmsys_revision_v2.ipynb into Kaggle Notebooks.
2. Add the competition data (+ Add Input → Competitions → LLM Classification Finetuning; join the competition first).
3. Attach the Keras model presets as inputs — a committed run cannot download new Kaggle Models.
4. Accelerator GPU T4 ×2, Internet on.
5. Set CFG.SMOKE_TEST = True for a ~30 minute dry run first; it defaults to False for real runs.
6. Launch with Save Version → Save & Run All (Commit), not the interactive editor, so the output is persisted.

Regenerating Figure 1

python3 make_figure1.py -o assets/pipeline_diagram.png --dpi 200

Every dimension and parameter count in the diagram is a named constant in that script, with its source cited in the module docstring, so the figure cannot silently drift from the manuscript.

---

Paper-to-Code Mapping

Table numbers follow the revised manuscript.

Paper element	Description	Code	Artefact
Algorithm 1	Swap augmentation	notebook_src.py §5, build_train_arrays()	test_local.py
Algorithm 2	Greedy multi-turn context selection	notebook_src.py §3, greedy_multi_turn_context()	test_local.py
Figure 1	End-to-end pipeline diagram	make_figure1.py	assets/pipeline_diagram.png
§4.1	Leakage-free partitioning by id, asserted at run time	notebook_src.py §4	test_local.py
Table 2	Matched-condition $2 \times 2$ factorial + volume control	reproduce_tables.py	results/paper_results.json
Table 3	Significance vs the matched baseline (bootstrap, McNemar, seed $t$-test)	reproduce_tables.py	results/paper_results.json
Table 4	Position bias: accuracy by winner position, flip consistency	reproduce_tables.py	results/paper_results.json
Table 6	Backbone comparison under a shared budget — partial	reproduce_tables.py	results/paper_results.json
Table 7	Data-efficiency curve — partial	reproduce_tables.py	results/paper_results.json
Table 9	Temperature calibration, global vs per-class	reproduce_tables.py	results/paper_results.json
Table 10	MT-Bench zero-shot transfer	reproduce_tables.py	results/paper_results.json
Table 11	Leave-one-out ablation with paired-bootstrap CIs	reproduce_tables.py	results/paper_results.json
Table 12	Factorial decomposition and the interaction term	reproduce_tables.py	results/paper_results.json
LaTeX tables	Publication-ready renderings	—	results/final_tables.tex

---

Experimental Environment

As documented in Section 4.1 of the paper:

• Operating system: Linux (glibc 2.35), CUDA 12.8, driver 580.159.04
• Hardware: 2× NVIDIA Tesla T4 (16 GB each)
• Software: Python 3.12.13 · Keras 3.13.2 (JAX backend) · keras-hub 0.26.0 · JAX and jaxlib 0.7.2 · TensorFlow 2.20.0 · NumPy 2.0.2 · pandas 2.3.3 · scikit-learn 1.6.1 · SciPy 1.16.3

requirements.txt pins these exactly. The full environment record, including the cuDNN version, is captured automatically at run time into results/paper_results.json under ["environment"], so every result carries the stack that produced it. Runs predating that capture report cudnn as "not recorded" rather than a guessed value.

---

Citation

@article{vaghasiya2026efficient,
  title={Efficient LLM Preference Classification Through Position Bias Mitigation and Architectural Symmetry},
  author={Vatsal Vaghasiya and Nancy Kshetrimayum and B. Prabadevi and Boppuru Rudra Prathap},
  journal={Applied Intelligence},
  year={2026},
  note={Manuscript APIN-D-26-05359}
}

---

License

This project is licensed under the MIT License - see the LICENSE file for details.