Comparing the default humor of LLM families. A joke request is maximally underspecified — the model chooses format, topic, register, and delivery entirely on its own — which makes "tell me a joke" a cheap, repeatable probe of model persona.
Each test lives in its own folder with its own preregistration. Designs,
measures, and hypotheses are committed before data collection; anything
that changes afterward goes in DEVIATIONS.md.
| Test | Question | Status |
|---|---|---|
test_1/ |
Unconstrained defaults + a content-pressure gradient (baseline / clean / dirty) across 2 families × 3 tiers, N=200/cell | data + dual judging complete (2026-06-11); human validation complete (2026-06-12) — format passed the §4.2 gate, compliance and raunchiness demoted to exploratory (see DEVIATIONS N4); format disputes resolved and confirmatory analysis complete (2026-07-05, DEVIATIONS N7) — H1 supported, H2/H4/H8 not supported; see test_1/RESULTS.md |
test_2/ (planned) |
Structure-constrained execution: "tell me a pun / a dad joke / a lightbulb joke / an absurd joke" — within-format style and execution quality, using test 1's baseline as control | not started |
pip install -r test_1/requirements.txt
cp test_1/.env.example test_1/.env # ANTHROPIC_API_KEY, OPENAI_API_KEYThen follow test_1/README.md. Raw data (test_1/data/) is gitignored;
JSONL outputs contain full raw API request/response pairs.
- Prereg or it didn't happen. Hypotheses are directional, falsifiable, and written to be fun to be wrong about.
- Receipts. Every call logs its full request and raw response.
- No silent fallbacks. If an API rejects a parameter, that's a decision point recorded in the prereg/deviations, never a quiet request mutation.
- Interested-party judges are flagged, dual-run, blinded, and human-validated. When the human validation fails a field, the field demotes — the rule fired in test 1 (DEVIATIONS N4), which is the system working, not failing.
- Instrument failures are findings. Judge degeneration, blinding defects, and category constructs that three coders read three ways are reported, not repaired in the dark (DEVIATIONS N1–N5).