Chess Agent Arena

A local arena where LLMs play chess — move by move, live — and you inspect every game afterward.

Chess Agent Arena started from a simple question:

What if the game itself was the agent harness?

Instead of a separate evaluation script, the model loop lives inside a playable browser chess game. Play against a model, let two models fight, watch the board update live, and inspect exactly what happened afterward. Chess is the first game because the rules are strict, the state is compact, and illegal moves are easy to catch.

This is a fun exploration project, not a rating authority. The bigger idea: a local arena where games become testbeds for agent behavior.

What it does today

• Play Human vs Model right from the browser.
• Run Model vs Model matches and watch them live.
• Use OpenRouter to try many models — including free ones when available.
• Save your OpenRouter key locally; pick a model from the list or type any model ID.
• Highlight legal moves; validate every move with chess.js so a model can't invent the board.
• Track live agent activity — tool calls, invalid moves, tokens, latency.
• Export PGN/FEN and keep replay, metrics, event, and model-call logs locally.
• Browse saved results without treating failed or incomplete runs as real final scores.

Live game — board + agent activity, moves, and export	Match setup — Human or Model for each color

Why chess?

Chess is a good first arena because it's unforgiving in useful ways:

• The model has to act on real state.
• A move is legal or it is not.
• Games can be replayed, and different models can face the same position.
• Provider errors, slow thinking, malformed replies, and bad moves are all visible.

That makes it a nice playground for comparing models — without pretending it's a formal benchmark.

The fun part

The app isn't a thin UI around an LLM call — it owns the match loop: board state, legal-move validation, turn handling, model requests, invalid-move handling, live updates, replay, artifacts, and result tracking. That's the part worth exploring. Chess is the first version; the same idea could grow into other games where agents must understand rules, make decisions, and live with the consequences.

Quickstart

Requirements: Node.js 22+, npm, and an OpenRouter API key for model matches.

git clone https://github.com/Null-Square/Chess-Agent-Arena.git
cd Chess-Agent-Arena
npm install
npm run dev

Open http://localhost:3000, then:

1. Open Configure Match → paste your OpenRouter key → Save.
2. Click Test model.
3. Choose Human vs Model or Model vs Model.
4. Pick models (or enter a model ID manually) and Start Match.

If port 3000 is busy:

npx tsx src/cli/index.ts dev --config configs/local.match.yaml --port 3001

Model arena

The most interesting mode is letting models play each other. A few experiments to try:

• Same model, different prompts.
• Fast model vs stronger model.
• Free model vs free model.
• Same opening position, different providers.

Free models can be rate-limited. When that happens the backend logs the provider error and the Results page marks the run as an issue instead of pretending the game ended normally.

Results & artifacts

Every run writes local artifacts under runs/<match_id>/ — game.pgn, final.fen, replay.json, metrics.json, events.jsonl, model_calls.jsonl, summary.md, and config.resolved.yaml.

The Results page stays honest: completed games show final scores, failed runs show the actual issue, incomplete runs aren't scored as final games, and matchups are grouped for comparison.

Useful commands

npm run dev          # start the local browser app
npm run validate     # validate the default config
npm test             # run tests
npm run typecheck    # run TypeScript checks
npm run build        # build server and web app

npx tsx src/cli/index.ts export-pgn runs/<match_id>   # export PGN from a saved run

Project shape

src/
  chess/       chess.js wrapper and legal-move enforcement
  runtime/     match loop and model-turn lifecycle
  model/       OpenRouter / OpenAI-compatible client
  logging/     events, metrics, PGN, replay, summaries
  server/      Express + Vite local server
  web/         React chess UI and results dashboard
  tests/       engine, runtime, provider, and UI behavior tests

Configs, model profiles, and prompts live in configs/, agents/, and prompts/. Most setup happens in the browser; the files are for repeatable experiments.

Privacy

• .env and .chess-agent-arena.secrets.json are git-ignored.
• Raw keys are redacted from provider errors.
• Run logs include model IDs, moves, usage, latency, and errors — not raw API keys.

Before publishing a fork, check for leaked keys:

git status --short --ignored=matching
rg -n "sk-or-v1-[[:alnum:]]+" -g "!node_modules" -g "!dist" -g "!dist-web" -g "!runs" .

If a real key was ever pasted into chat, an issue, or a commit, rotate it.

Roadmap

Working now: browser match setup · Human vs Model · Model vs Model · OpenRouter key save/test · free-model filtering & manual model IDs · legal-move highlighting & enforcement · live board · PGN/FEN export · local run artifacts · results page.

Not built yet: public multiplayer · accounts · cloud sync · built-in Stockfish analysis · match clocks / per-move time limits · games beyond chess.

Where it could go: time-limited matches, saved experiment presets, a better replay viewer, aggregate model-comparison stats, prompt-comparison reports, optional post-game analysis, and more games on the same agent-harness idea.

Contributing

Run checks before opening a PR:

npm run typecheck && npm test && npm run build

Contributions are welcome — especially around model reliability, replay UX, results, provider support, and making the arena more fun to experiment with.

---

An open-source experiment from NullSquare · more projects