_{Anish Laddha, Nitesh Pradhan, Gaurav Srivastava · Jun 2026 · Submitted to ACL ARR, target NAACL 2026}

Using a large proprietary model as an evaluator is the default in NLP, and it is expensive. We benchmarked 16 small judges (0.6B to 14B parameters, from the Qwen, Llama, and Phi-4 families) against 3 large models (70B to 120B) on 10 datasets, at 64,824+ judgments per configuration, to find out how much you actually give up by going small.

• A 14B Phi-4 judge reaches 89.55% binary-judgment accuracy. A 3-model majority-vote ensemble nudges that to 89.61%
• Overthinking turns out to be domain-dependent. Quick 10-token verdicts match or beat long chain-of-thought reasoning on math judging, while reasoning wins on general-knowledge tasks by up to 23%
• Multi-agent debate (Reflect-Critique-Refine) hurt accuracy in every configuration we tested, which was not what we expected going in
• Getting to 9,000+ experiment configurations on A100 and V100 GPUs meant multi-GPU vLLM pipelines, GPTQ quantization, and tensor parallelism

_{Anish Laddha, Nitesh Pradhan, Gaurav Srivastava · Jun 2026 · Submitted to ACL ARR, target NAACL 2026}

The companion survey to SLMJury. It maps 101 works on small models as judges across five dimensions, covering supervised fine-tuning, preference optimization, reinforcement learning with verifiable rewards, and representation-based probing.

• Fine-tuned 3B to 8B judges match proprietary-model accuracy on standard benchmarks, and mixed panels get there at over 7x lower cost, which puts private on-device evaluation within reach
• Evaluation-specific training beats raw scale, and reasoning tokens only pay off when they surface genuinely new signal
• Under position-consistent protocols, much of what reads as judge skill comes down to heuristics, and every judge degrades on hard pairs, frontier models included

SLMJury The benchmark harness behind the paper, packaged so anyone can rerun it. It ships on PyPI with a CLI and a Python API. Single-model, majority-vote, debate, and persona judging strategies Multi-GPU vLLM serving with GPTQ quantization and tensor parallelism Dataset-aware answer parsers on regex and SymPy, released through GitHub Actions

Multimodal Content Moderation A moderation service that checks text and images together, so an innocuous caption on a flagged image does not sail through the filter. 98% text and 96% image accuracy, behind a HuggingFace, then OpenAI, then keyword fallback chain Sub-20ms cached responses against 200 to 500ms live, using Redis Docker Compose behind nginx: Express API, React 19 dashboard, MongoDB, JWT auth, RBAC

ArxivAssistant A RAG navigator that pulls papers from arXiv at query time rather than searching a corpus someone indexed months ago. Every answer cites its sources. Two-pass search (title match, then keyword fallback), plus direct arXiv ID lookup Full PDF extraction with PyMuPDF, embedded locally using all-MiniLM-L6-v2 FAISS retrieval into a LangChain chain over Qwen2.5-7B-Instruct, with no paid APIs

Awesome-SLM-as-a-Judge The 101 works behind the survey, organized under its five dimensions instead of one flat list. Covers fine-tuning, preference optimization, verifiable rewards, and probing Hosts the full survey text and its interactive figures as a static site

• Took a compliance review that ran for months down to 30 minutes, at 98.3% accuracy (59 of 60 checkpoints) on a 500+ page Detailed Project Report. I owned the RAG pipeline end to end: table-aware chunking, multi-query generation, hybrid search, reranking, and 4-phase structured chain-of-thought prompting
• Designed MixSet, which swaps binary AI detection for a continuous 0 to 1 contribution score. Any label scheme can be derived from that score without relabeling the corpus. Built across a 15-task, 585-prompt, 10-model matrix over 75K documents

• Shipped a GPT-4 Streamlit engine (OpenAI API, SerpAPI, BeautifulSoup) that automated competitor news comparisons across 4 broadcast sources, cutting editorial review effort by 80% over 100+ articles
• Built a multi-output regression model that predicts Digit.in expert phone review scores across 6 categories at R² = 0.97 to 0.98, trained on 300+ internal device records and validated by the editorial team

Languages

ML and AI

Data

Web and Backend

Infrastructure