Compact Language Models in Retrieval-Augmented Generation

This repository accompanies the paper:

Compact Language Models in Retrieval-Augmented Generation:
Accuracy–Latency Trade-offs under a Unified Pipeline
Alexandros Papafragkakis, Yannis Tzitzikas
University of Crete & FORTH-ICS

Accepted as a full paper at the 14th EETN Conference on Artificial Intelligence (SETN 2026), Chania, Crete, September 9–11, 2026.
To appear in the conference proceedings, published by ACM in the International Conference Proceedings Series (ACM Digital Library).

Paper (preprint): https://www.overleaf.com/read/rvztwdfhfrxh#9d2996

Overview

Retrieval-Augmented Generation (RAG) systems are increasingly deployed in environments where latency, cost, and hardware limits matter as much as answer quality.

This work provides a systematic evaluation of compact open-source language models (1B–8B parameters) within a controlled RAG pipeline, isolating the effect of the generator model while keeping retrieval fixed.

We evaluate six models across two knowledge-intensive QA benchmarks and analyze:

• Accuracy: Hits@1, F1, Exact Match (EM)
• Latency: retrieval, generation, end-to-end
• Bottlenecks: retrieval-bound vs generation-bound behavior

For full details, see the paper.

---

Key Findings

• Model size alone is a poor predictor of RAG performance in the 1B–8B range.
• Mid-size models (≈3B) can approach 8B accuracy when retrieval quality is strong.
• Smaller models benefit disproportionately from retrieval augmentation.
• Latency bottlenecks differ by model size:
  • 1B–3B: mostly retrieval-bound
  • 7B–8B: increasingly generation-bound
• Mistral 7B lies on the accuracy–latency efficiency frontier for factual QA in our setup.

---

Evaluated Models

• Llama 3.2 (1B, 3B)
• Gemma 2 (2B)
• Phi-3 Mini (3.8B)
• Mistral 7B
• Llama 3.1 8B

All models are evaluated under the same retrieval configuration.

---

Benchmarks

• MetaQA (single-hop) — movie-domain QA
• WC2014QA — FIFA World Cup factual QA

---

Experimental Setup (Summary)

• Fixed retrieval pipeline (embeddings, hybrid search, reranker, prompt template)
• Deterministic decoding (temperature = 0, max_new_tokens=128)
• 4-bit quantization (bitsandbytes NF4)
• Latency measured end-to-end via HTTP API round-trip
• GPU (server): AMD Radeon RX 9070XT (16GB VRAM)

See Section 3 of the paper for full methodological details.

---

Reproducing Results

This repository assumes access to the SemanticRAG HTTP API used in our experiments.

High-level workflow

1. Configure the API endpoint and model identifiers
2. Run the evaluation scripts (per dataset, per model)
3. Aggregate metrics and generate plots/tables

Detailed configurations (retrieval settings, prompt template, normalization, and metrics) are described in the paper.

---

Notes on Fair Comparison

To isolate generator effects, we keep retrieval and prompting fixed across models:

• Same embedder, hybrid retrieval parameters, reranker, and target top-k
• Same decoding settings (temperature = 0) and maximum generation length
• Same evaluation normalization applied to all model predictions

---