DACH KI-Quellenbenchmark 2026 — Welle 1 (Baseline)

Open, reproducible benchmark of organic brand visibility in AI answers across Germany, Austria and Switzerland. Wave 1 (baseline) was collected on 2026-07-27: 180 brand-open commercial prompts, three search-grounded LLM engines, three runs each — 1,620 API observations. All aggregated data, the frozen panel and the collection pipeline are in this repository.

Results and figures: https://eullrich.com/research/quellenbenchmark-2026/

What is measured

Three independent signals, per defined measurement surface:

• Mention rate — the brand name appears in the answer text.
• Citation rate — the brand's domain appears as a clickable HTTPS source.
• Grounding success rate — the model actually ran a web search for the run.

A run counts as a valid observation only if it made at least one web-search call and returned at least one machine-readable HTTPS citation.

Panel (frozen before collection)

Categories	E-Commerce (hiking boots), SaaS (CRM), Industrial (coding/marking), Local (self-storage), YMYL (mortgage advice), Travel (family hotels)
Markets	DE · AT · CH (localised prompts)
Providers	90 (5 per category × market), verified before the run
Prompts	180 (6 categories × 3 markets × 10 scenarios), brand-open
Engines	openai/gpt-5-mini, google/gemini-3.5-flash, google/gemini-3.5-flash-lite
Runs	3 per prompt × engine → n = 1,620

Repository layout

scripts/
  collect-wave-1.mjs          # runs the 1,620 API queries (concurrency-limited, retries)
  repair-wave-1.mjs           # re-runs failed/blocked queries
  aggregate-wave-1.mjs        # builds wave_1_results.json from the receipts
  validate-monitor-panel.mjs  # panel schema/quota validation
  freeze-panel.mjs            # freezes the candidate panel into panel_v1.json
  engine-canary-smoke-test.mjs
data/monitor/
  panel_v1.json               # frozen panel: 180 prompts, 90 providers
  engines_v1.json             # engine roster v1.2 (shared request contract)
  wave_1_results.json         # aggregated mention/citation rates (global + per model)
  wave_1_summary.json         # run log for all 1,620 runs (ok/fail, latency, citations)
  wave_1_results.csv          # aggregated results, spreadsheet-friendly
  *.md                        # provenance: eligibility, roster decisions, conflict declaration, reviews

Not included here: raw API response bodies (wave-1-raw/, contain auth/billing headers) and the 1,620 per-run receipts. The aggregated results and the run log fully back every number on the results page.

Reproduce

Requires Node.js ≥ 20.3 (built-in fetch). No dependencies.

# 1. validate the frozen panel
node scripts/validate-monitor-panel.mjs

# 2. run the collection (costs real API money via OpenRouter)
OPENROUTER_API_KEY=… node scripts/collect-wave-1.mjs --execute --i-approve-api-costs

# 3. aggregate receipts into wave_1_results.json
node scripts/aggregate-wave-1.mjs

The request contract (from engines_v1.json): OpenRouter Chat Completions, openrouter:web_search with engine: native, max_tool_calls: 5, max_tokens: 4000. Sampling parameters (temperature, top-p) are not pinned — provider defaults are used, so runs are not bit-deterministic; every published figure is a mean over 90 runs.

Independence

Own Node.js batch collector over the OpenRouter API; no third-party visibility tool fed into this open panel. Per data/monitor/conflict-declaration-2026-07-27.md: at the collection date there was no current or prior client relationship with any tested provider in any of the six categories, and no brand commissioned, funded or influenced the study.

Cite

Ullrich, E. (2026): DACH KI-Quellenbenchmark 2026, Welle 1 (Baseline). eullrich.com/research/quellenbenchmark-2026, retrieved 2026-07-27.

License

Data and documentation: CC BY 4.0 (see LICENSE). Scripts may additionally be reused under the MIT license. Attribution: Eugen Ullrich, eullrich.com.