When Large Language Models Know the Table

A Framework for Assessing Data Contamination in Tabular Datasets

Official implementation of the COLM 2026 paper "When Large Language Models Know the Table: A Framework for Assessing Data Contamination in Tabular Datasets" (Silvestri, Veglianti, Giorgi, Silvestri, Tolomei — Sapienza University of Rome).

Large language models (LLMs) may perform well on public tabular benchmarks not because they generalize, but because they were already exposed to that data during pretraining. This repository provides a principled framework to detect this kind of data contamination in tabular datasets, going beyond coarse verbatim-memorization tests.

Overview

Given a public tabular dataset, the framework:

1. Derives controlled dataset variants that selectively destroy or preserve specific properties of the data (row-level dependence, semantics, category identity).
2. Constructs multiple-choice contamination probes over two complementary tasks — completion (recover masked cells) and existence (spot the authentic row among fabricated ones).
3. Evaluates LLMs against non-neural baselines that capture what is achievable from chance and from column-level marginal statistics alone.
4. Applies a statistical test (McNemar's test against each baseline) to decide, with a formal criterion, whether observed performance is evidence of contamination rather than ordinary task competence or generic tabular reasoning.

This design lets the framework separate authentic dataset exposure from world knowledge or statistical regularities, and — unlike prior memorization-based approaches — assign a graded, statistically grounded verdict per dataset–model pair rather than a binary flag.

Dataset variants

Variant	Transformation	Tests for
real	Unmodified cleaned dataset.	Reference condition — where contamination should be most visible.
like	Independently resamples each column from its own marginal distribution, destroying row-wise dependence.	Column-level statistical knowledge vs. authentic row knowledge.
swapped	Within-column derangement (categorical values permuted, numerical values mirrored) so no value maps to itself.	Robustness to relabeling / reliance on relational structure.
obfuscated	Feature names, target, and categorical values replaced with anonymous codes (f1, c0, ...).	Dependence on semantic cues (feature/label names) vs. world knowledge.

Probes and evaluation

• Completion probes mask a subset of columns in a real row and ask the model to pick the correct values among five options.
• Existence probes show five full candidate rows and ask the model to identify the one that genuinely occurs in the dataset.
• Both are scored against a random baseline (uniform choice) and two marginal log-likelihood baselines — deterministic (greedy) and stochastic (softmax-sampled) — built from column-wise empirical distributions.
• A dataset–model pair is flagged as contaminated when the model's accuracy is statistically significantly above the random baseline on the real variant (McNemar's test, α = 0.01); comparisons against the marginal-informed baselines further grade the strength of the evidence.

Results

The framework was evaluated on eight public tabular datasets (adult, blood, credit, diabetes, gamma, iris, mushroom, titanic) plus a synthetic negative control, across six open-weight LLMs spanning multiple families and scales (Mistral-7B, Qwen-7B/14B/32B, Llama-8B/70B).

Two headline findings from the paper:

• Memorization-based tests are too coarse. Prior verbatim-memorization approaches (Bordt et al., 2024) report near-zero reconstruction accuracy on adult and titanic for every model tested, even though our framework detects clear contamination signals on both.
• Contamination grows with model scale. Larger models are consistently more likely to express the effects of prior dataset exposure, which means scale-related performance gains on public tabular benchmarks may be partly explained by contamination rather than pure generalization.

Contamination registry

The table below reproduces the paper's contamination registry (Table 2): a ✓ indicates the framework detected statistically significant contamination for that dataset–model pair on the real variant (α = 0.01); a ✗ indicates no detected contamination.

Dataset	Mistral-7B	Qwen-7B	Llama-8B	Qwen-14B	Qwen-32B	Llama-70B
adult	✓	✓	✓	✓	✓	✓
blood	✗	✗	✗	✗	✗	✗
credit	✗	✓	✓	✓	✓	✓
diabetes	✗	✗	✗	✗	✗	✗
gamma	✗	✗	✗	✗	✗	✗
iris	✗	✓	✗	✓	✓	✓
mushroom	✗	✗	✗	✗	✗	✗
synthetic	✗	✗	✗	✗	✗	✗
titanic	✓	✓	✓	✓	✓	✓

Contamination is consistently detected on adult, credit, iris, and titanic, while blood, diabetes, gamma, and mushroom show no detectable signal under this protocol — including the synthetic negative control, as expected since it has no possible prior exposure. Full per-variant accuracies, statistical tests, and ablations (temperature, coverage) are reported in the paper's Appendix D.

Installation

Requires Python 3.11+. GPU-backed local inference (via vllm) requires a CUDA-capable GPU; cloud-only backends (Groq, Gemini) do not.

Option A — uv (recommended)

git clone https://github.com/hercolelab/tabular_contamination.git
cd tabular_contamination
uv sync

uv sync resolves and installs all dependencies declared in pyproject.toml, including the bundled LLM-Tabular-Memorization-Checker package in editable mode. Run any script with uv run python <script>.py, or scripts under script/ directly with bash.

Option B — pip

git clone https://github.com/hercolelab/tabular_contamination.git
cd tabular_contamination
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

API keys

Cloud-hosted models (Groq, Gemini) require API keys. Create a .env file at the repository root:

GROQ_API_KEY=your_groq_key_here
GEMINI_API_KEY=your_gemini_key_here

Local models (Mistral, Qwen, Llama) are served through vllm and require no API key, but do require a compatible GPU and enough VRAM for the selected model size.

Repository structure

tabular_contamination/
├── src/
│   ├── contamination.py     # Row-level contamination benchmark runner (main pipeline)
│   ├── memorization.py      # Memorization benchmark runner (Bordt et al.-style tests)
│   ├── baseline.py          # Random / deterministic / stochastic baselines
│   ├── plots.py             # Result aggregation and figure generation
│   └── utils/
│       ├── contamination_constants.py       # Coverage levels, dataset tags
│       ├── dataset_cleaning_and_split.py    # Dataset cleaning pipeline
│       ├── dataset_contamination.py         # Dataset-variant + probe generation
│       ├── load_uci_dataset.py              # UCI dataset loaders
│       ├── synthetic_dataset.py             # Synthetic negative-control dataset
│       ├── metrics.py                       # Accuracy, McNemar's test
│       ├── manual_summary_metrics.py        # Summary/registry aggregation
│       ├── models_list.py                   # Model name -> backend mapping
│       ├── memorization_utils.py            # Gemini / Groq / vLLM LLM interfaces
│       ├── console_logger.py                # Structured CLI logging
│       └── LLM-Tabular-Memorization-Checker/  # Bundled `tabmemcheck` package
├── data/
│   ├── <dataset>/<dataset>.csv              # Raw UCI/Kaggle datasets
│   ├── <dataset>/<dataset>_cleaned.csv       # Cleaned datasets used by the framework
│   └── contamination_probes/<dataset>/       # Generated probe files (.jsonl) per variant/coverage
├── script/
│   ├── contamination.sh     # Batch launcher for src/contamination.py (model x dataset x coverage grid)
│   └── memorization.sh      # Batch launcher for src/memorization.py
├── contamination_query.py   # Single-probe playground for the contamination pipeline
├── memorization_query.py    # Single-test playground for the memorization pipeline
├── pyproject.toml           # uv/pip project definition
└── requirements.txt         # pip dependency list

Results from batch runs are written under results/contamination/ and results/memorization/, mirroring the dataset / coverage / model hierarchy used by the scripts.

Usage

1. Batch experiments (main entry point)

The reproducible way to run the full evaluation grid is through the script/ launchers, which wrap src/contamination.py and src/memorization.py:

# Contamination: all models x all datasets x all four variants (real, like, obfuscated, swapped)
bash script/contamination.sh

# Restrict to specific dataset variants (positional args)
bash script/contamination.sh real like

# Memorization: Bordt et al.-style header / row-completion / feature-completion / first-token tests
bash script/memorization.sh

Edit the MODELS, DATASETS, COVERAGES, and TEMPERATURES arrays at the top of each script to select which combinations to run — comment out entries you don't need. Both scripts print structured progress logs and exit non-zero if any run fails.

Under the hood, each script invokes the corresponding pipeline directly, e.g.:

uv run python src/contamination.py \
  --model_name qwen_14B \
  --dataset titanic \
  --coverage 20 \
  --temperature 0 \
  --dataset_tags real like obfuscated swapped

uv run python src/memorization.py \
  --dataset_name titanic \
  --model_name qwen_14B \
  --temperature 0.0 \
  --tests header first_token row_completion feature_completion

Baselines (random, deterministic, stochastic) are computed separately and are required for the statistical tests in contamination.py:

uv run python src/baseline.py --dataset titanic --coverage 20

2. Single-probe playgrounds (debugging)

Two lightweight scripts let you inspect prompt formatting and model output without running a full batch:

contamination_query.py — sends one contamination probe to a model and prints the prompt, raw response, and parsed answer:

uv run python contamination_query.py

Model and probe file are currently set in-file (model_name, probes_path); edit probes_path to target a different dataset, variant (real, like, obfuscated, swapped), or coverage level.

memorization_query.py — runs one tabmemcheck memorization test against a single dataset:

uv run python memorization_query.py \
  --dataset adult \
  --test row_completion \
  --model-name gemini-2.5-flash-lite \
  --num-queries 25 \
  --temperature 0.0

Available --test values: header, row_completion, feature_completion, first_token, feature_names, feature_values, sample, dataset_name, all.

Citation

If you use this framework in your research, please cite:

@inproceedings{silvestri2026colm,
  title     = {When Large Language Models Know the Table: A Framework for Assessing Data Contamination in Tabular Datasets},
  author    = {Silvestri, Matteo and Veglianti, Fabiano and Giorgi, Flavio and Silvestri, Fabrizio and Tolomei, Gabriele},
  booktitle = {Conference on Language Modeling (COLM)},
  year      = {2026}
}