Stupid Test

A deliberately tiny benchmark of pattern recognition and conversational fluidity in language models.

Live presentation: https://makefunstuff.github.io/stupid-test/

The entire test

Start a fresh conversation and send:

click clack

The expected answer is:

bang

If the model misses, reply with the answer it should have produced. Then continue alternating the two cues:

USER         EXPECTED ASSISTANT
click clack  bang
bang         click clack
click clack  bang
bang         click clack

No system prompt explains the game. The model that adopts the convention in the fewest turns wins.

Why test something this small?

Frontier models are usually demonstrated with large, highly structured tasks: a repository, a goal, files, conventions, tests, errors, and repeated feedback. Those tasks are genuinely difficult, but their size also gives the model abundant clues about what mode to enter. Much of the interaction resembles patterns seen during training.

click clack → bang is the opposite:

• computationally trivial;
• socially under-specified;
• almost free of redundant clues;
• taught only through the human's behavior.

The model must treat earlier turns as demonstrations, infer that the human is creating a private convention, preserve the mapping, and suppress its default urge to explain. This is a small probe of jagged intelligence: enormous execution ability can coexist with failure on a tiny interaction that requires a fluid shift of conversational mode.

Pattern recognition is not fluidity

The benchmark separates two capabilities that are easy to conflate.

Pattern recognition

The model must:

1. treat previous messages as demonstrations rather than unrelated prompts;
2. identify the two relevant phrases;
3. infer the two-way mapping;
4. retain it over later turns.

Conversational fluidity

The model must then:

1. recognize that the human is establishing a local interaction mode;
2. abandon generic explain-and-assist behavior;
3. act on the inferred convention;
4. answer with only the expected phrase.

A model can explicitly describe the alternating pattern and still fail the test. That is not a pattern-recognition failure. It is a failure to convert recognition into participation.

Why this approach?

This project is an observation before it is a benchmark program. The point is not to pretend that four syllables settle the question of intelligence. The point is to start with the smallest probe that can expose the behavior.

Simple probes have useful properties:

• Legible failure: the transcript shows exactly what the model did instead.
• Low confounding: no tools, retrieval, hidden task instructions, or elaborate scoring model are needed.
• Behavior over narration: explaining the rule does not count as following it.
• Cheap repetition: many models can be compared for less than one dollar.
• Easy criticism: weaknesses in the protocol are visible and can be tested directly.

A complex demo may look more impressive while making failure harder to interpret. Before building one, try the stupid thing first.

Canonical protocol

Each model receives three independent runs. Every run:

1. starts with an empty conversation and no explicit system prompt;
2. sends eight fixed user cues, alternating click clack and bang;
3. replays the complete visible transcript on each request;
4. never changes the next cue in response to model output;
5. uses temperature 0 when supported;
6. disables optional reasoning and uses the lowest effort when reasoning is mandatory;
7. limits the completion to 1024 tokens, including reasoning tokens;
8. disables tools and web access.

Using a fixed eight-turn sequence matters. Adaptive stopping would give different models different evidence. Here every model sees the same user messages in the same order; stopping is applied only during scoring.

Provider controls are recorded honestly. If a model does not support temperature or cannot disable reasoning, the result stores that difference rather than pretending the requests were identical.

Scoring

Primary: phrase match

The primary normalizer lowercases the response, extracts its ASCII word sequence, and joins those words with spaces.

Output	Result	Why
bang	pass	expected phrase
Bang!	pass	punctuation ignored
**bang** 💥	pass	formatting and emoji ignored
The answer is bang	fail	added words
click clack, bang	fail	added phrase

A model acquires the rule at the first of two consecutive phrase matches, one in each direction. Scores range from 1 to 7. No complete pair within eight turns is DNF. The leaderboard uses the median of three runs; DNF has a comparison value of 9.

Secondary: strict match

The strict matcher trims outer whitespace and otherwise requires exactly bang or click clack, including case and punctuation. It measures whether the model can participate without even harmless decoration.

First result

Run date: 2026-09-08

• 17 models
• 51 independent conversations
• 408 assistant responses
• 0 request errors
• 15 phrase matches
• 0 strict matches
• $0.6260 total OpenRouter cost reported by the API

Model	Acquisition by run	Median	Phrase accuracy
GPT-6 Astra	DNF, 5, 5	5	37.5%
GPT-5.6 Sol	3, DNF, DNF	DNF	25.0%
DeepSeek V4 Pro 0813	DNF, DNF, DNF	DNF	0.0%
DeepSeek V4 Flash 0731	DNF, DNF, DNF	DNF	0.0%
GLM 5.3 Flash	DNF, DNF, DNF	DNF	0.0%
GLM 5.3	DNF, DNF, DNF	DNF	0.0%
Kimi K3	DNF, DNF, DNF	DNF	0.0%
Claude Fable 5.1	DNF, DNF, DNF	DNF	0.0%
Claude Opus 5	DNF, DNF, DNF	DNF	0.0%
Claude Sonnet 5	DNF, DNF, DNF	DNF	0.0%
Gemini 3.8 Flash	DNF, DNF, DNF	DNF	0.0%
Gemini 3.7 Flash	DNF, DNF, DNF	DNF	0.0%
Grok 4.6	DNF, DNF, DNF	DNF	0.0%
Qwen3.8 Max 0902	DNF, DNF, DNF	DNF	0.0%
Hy4 Preview	DNF, DNF, DNF	DNF	0.0%
Local Qwen3.8 27B Q5_K_XL	DNF, DNF, DNF	DNF	0.0%
Local Ornith 1.5 9B Q5_K_M	DNF, DNF, DNF	DNF	0.0%

GPT-6 Astra was the only model with a winning median. GPT-5.6 Sol acquired the mapping once but failed the other two runs. Several losing models explicitly noticed or described the sequence, then continued to echo, explain, decorate, or invent around it.

The complete visible transcripts, request controls, canonical model revisions, routed providers, latencies, token counts, and costs are in results/2026-09-08.json.

Interpretation

The result supports a narrow observation:

Pattern recognition did not reliably become conversational fluidity.

It does not prove that models lack consciousness, cannot understand humans, or are incapable of general reasoning. It does not isolate training data as the cause. Assistant tuning, ambiguity, sampling, provider behavior, and the phrase pair itself are plausible contributors.

The result also does not invalidate large-task performance. It shows that task size and task difficulty are not ordered uniformly: rich scaffolding can help a model enter a familiar mode, while a tiny private convention can demand a more fluid interpretation of sparse human intent.

How this should be improved

This benchmark can and should be made stronger:

1. Randomized private mappings — generate unpublished phrase pairs with the same two-way structure to reduce contamination and memorization.
2. More repetitions — estimate variance rather than relying on three runs.
3. Human baselines — measure how often people infer the same convention and how quickly.
4. Explicit-control condition — compare implicit teaching with a direct rule such as “reply to X with Y.” This separates rule execution from intent inference.
5. Scoring sensitivity — publish strict, phrase-only, and semantic scores without choosing the matcher after seeing results.
6. Provider pinning — pin upstream providers and model revisions where possible.
7. Reasoning conditions — compare reasoning disabled, minimal, and default rather than mixing mandatory and optional reasoning models.
8. Alternative interaction shapes — test one-way mappings, three-item cycles, delayed recall, reversals, and recovery after a broken pattern.
9. Pre-registration — freeze prompts, stopping rules, normalization, and model selection before running.

Those improvements should follow the observation, not prevent it from being made. A tiny imperfect experiment can reveal a useful question. The next step is to tighten the experiment, not replace it with spectacle.

Run it

Requirements: Node.js 20 or newer and an OpenRouter API key.

export OPENROUTER_API_KEY='...'
npm run bench

If Oh My Pi already stores the credential, the runner can obtain it with omp token openrouter without printing or writing the key.

Run a subset or a single repetition:

node scripts/run-bench.mjs \
  --runs 1 \
  --models openai/gpt-6-astra,deepseek/deepseek-v4-flash-0731 \
  --output results/smoke.json

Local OpenAI-compatible models use LOCAL_LLM_BASE_URL, defaulting to http://127.0.0.1:8080/v1:

LOCAL_LLM_BASE_URL=http://127.0.0.1:8080/v1 \
  node scripts/run-bench.mjs --runs 1 --models qwen3.8-27b

The runner has no package dependencies. It fetches live OpenRouter metadata, runs each model sequentially, saves raw visible output, computes both scores, and prints a compact leaderboard.

Repository

.
├── index.html                  # static presentation
├── styles.css
├── models.json                 # model matrix
├── package.json                # standard run commands
├── scripts/
│   └── run-bench.mjs           # dependency-free benchmark runner
└── results/
    └── 2026-09-08.json         # complete first run

License

The benchmark protocol, runner, data, and site are released under the MIT License.