GRIPS eval

Minimal evaluation harness for GRIPS - German Reasoning, Idioms, Puzzles & Wordplay (Sprachspiel) - a German wordplay/logic quiz benchmark.

Three steps: collect model responses → judge them with an LLM judge → score (overall + per-mechanic + hard-subset). It works on any GRIPS dataset by swapping --dataset:

• ellamind/grips - the public benchmark dataset (925 items, text-only)
• private companions (*-private) use the same schema, so the harness is drop-in
  • ellamind/grips-private - held-out text benchmark dataset
  • ellamind/grips-vision-private - held-out vision benchmark dataset

Install

This is a uv project. uv creates and manages the virtualenv from pyproject.toml, so just prefix every command with uv run (no manual venv/activate). The first uv run auto-syncs the env; or sync explicitly once:

uv sync

_{(Plain pip works too: pip install -r requirements.txt, then drop the uv run prefix.)}

API keys - copy .env.example to .env and fill in the keys you use (auto-loaded by collect.py/judge.py), or export them as env vars:

cp .env.example .env
# then edit .env - set only the providers you'll call:
#   OPENAI_API_KEY, OPENROUTER_API_KEY, ANTHROPIC_API_KEY,
#   GEMINI_API_KEY, MISTRAL_API_KEY, AZURE_OPENAI_API_KEY
#   HF_TOKEN   (only for gated/private datasets)

1. Collect responses

Query the model under test. Works against any OpenAI-compatible endpoint, plus the native Anthropic API. --dataset defaults to ellamind/grips; pick the right --base-url / --api-key-env for your provider:

# OpenAI (default base URL)
uv run collect.py --model gpt-5.4-mini --api-key-env OPENAI_API_KEY

# Azure OpenAI  (--model is your DEPLOYMENT name, not the model id; needs --token-param max_completion_tokens)
# replace YOUR_RESOURCE + YOUR_DEPLOYMENT with your own (quote the URL):
uv run collect.py --model YOUR_DEPLOYMENT \
  --base-url "https://YOUR_RESOURCE.cognitiveservices.azure.com/openai/v1/" \
  --api-key-env AZURE_OPENAI_API_KEY --token-param max_completion_tokens

# Anthropic (native API)
uv run collect.py --model claude-sonnet-4-6 \
  --base-url https://api.anthropic.com --api-key-env ANTHROPIC_API_KEY

# Google Gemini (OpenAI-compatible endpoint)
uv run collect.py --model gemini-3.5-flash \
  --base-url https://generativelanguage.googleapis.com/v1beta/openai/ \
  --api-key-env GEMINI_API_KEY

# Mistral
uv run collect.py --model mistral-large-2512 \
  --base-url https://api.mistral.ai/v1 --api-key-env MISTRAL_API_KEY

# OpenRouter (any model)
uv run collect.py --model deepseek/deepseek-v4-pro \
  --base-url https://openrouter.ai/api/v1 --api-key-env OPENROUTER_API_KEY

# OpenRouter pinned to a specific upstream provider (reproducible - fail loud, no fallback)
uv run collect.py --model deepseek/deepseek-v4-pro \
  --base-url https://openrouter.ai/api/v1 --api-key-env OPENROUTER_API_KEY \
  --extra-body-json '{"provider":{"order":["deepseek"],"allow_fallbacks":false}}'

Reasoning / extended thinking

Enable reasoning per provider - and always verify it actually engaged, because the reporting differs:

# OpenAI / Gemini / OpenRouter / Mistral - reasoning effort
uv run collect.py --model gpt-5.5 --reasoning-effort high --max-tokens 64000

# Anthropic - extended thinking (NOT --reasoning-effort, which is silently ignored here).
# Pass --test-max-tokens ABOVE the thinking budget (the preflight check uses it).
uv run collect.py --model claude-sonnet-4-6 \
  --base-url https://api.anthropic.com --api-key-env ANTHROPIC_API_KEY \
  --anthropic-thinking enabled --reasoning-max-tokens 32000 --max-tokens 64000 --test-max-tokens 33000
#   ...or let the model choose depth:  --anthropic-thinking adaptive

Where the reasoning signal shows up (check before trusting an "on" run):

• OpenAI-style (OpenAI/Azure/Gemini-compat/OpenRouter/Mistral): usage.completion_tokens_details.reasoning_tokens.
• Gemini: reasoning_tokens stays 0 - it's the total_tokens − prompt − completion gap.
• Anthropic: reported as output_tokens / thinking blocks (no reasoning_tokens field); --reasoning-effort is a no-op - use --anthropic-thinking.

Writes runs/responses/<run-id>/responses.jsonl. Re-running resumes the same run: --retry-incomplete redoes empty/truncated responses, --trials N does variance runs, --limit N caps items, --vision sends images.

2. Judge

LLM-as-judge against the reference answer + explanation (semantic match, not string match). We recommend gpt-5.4-mini - a reliable, cheap judge for this dataset.

uv run judge.py --dataset ellamind/grips \
  --judge-model openai/gpt-5.4-mini \
  --base-url https://openrouter.ai/api/v1 --api-key-env OPENROUTER_API_KEY

Scans runs/responses/ and writes runs/ratings/<judge>/<response-run>/judgments.jsonl (skips already-judged items; --retry-errors to redo failures). The exact judge prompt + strict-JSON {"rating":"YES"|"NO","rationale":...} schema live in judge.py.

3. Score

# overall accuracy only (judgments alone):
uv run score.py runs/ratings/<judge>/<response-run>/

# + hard-subset and per-mechanic (joins the dataset by id):
uv run score.py runs/ratings/<judge>/<response-run>/ --dataset ellamind/grips --by-mechanic

Prints overall accuracy; with --dataset it also reports the hard-subset accuracy (items where hard=true) and the per-mechanic breakdown - these are looked up from the dataset by id, since the judge keeps only the item id (it's a generic grader, not coupled to the dataset schema). YES counts as correct; it warns if any items are unjudged.

Vision

The vision companion set carries images. Pass --vision to send them to a vision-capable model:

uv run collect.py --dataset <vision-dataset> --vision --model <vision-model> --base-url ... --api-key-env ...

Notes

• Methodology is fixed: the collection prompt, judge prompt, and YES/NO schema are identical to what produced the published GRIPS results - don't change them if you want comparable numbers.
• Judge: in our experiments, gpt-5.4-mini is a very good judge for this dataset (reliable and cheap) and is what produced the reference numbers.
• Single run by default. Use --trials N (collect) for variance estimates.

License

Code: Apache-2.0 (see LICENSE). Dataset (ellamind/grips): CC-BY-4.0.

Acknowledgements

• This project is supported by the OpenEuroLLM project, co-funded by the Digital Europe Programme under GA no. 101195233.
• This project is supported by the LLMs4EU project, co-funded by the Digital Europe Programme under GA no. 101198470. For more information see LLMs4EU website.
• This project is supported by the German Federal Ministry for Economic Affairs and Energy (BMWE) through EU-SAI/SOOFI: Sovereign Open Source Foundation Models for European Intelligence (grant number 13IPC040J).
• This project is supported by the Ministry of Economic Affairs, Industry, Climate Action and Energy of the State of North Rhine-Westphalia and the European Union through LLM4KMU: Optimierter Einsatz von Open Source Large Language Models in KMU (grant number EFRE-20801077).

---

Disclaimer: This is a research project and not an official ellamind product.

---