AI Gateway Benchmark

Transparent benchmark templates for measuring an OpenAI-compatible AI gateway.

This repository helps developers measure:

• request success rate,
• P50 / P95 latency,
• first-token latency for streaming,
• error categories,
• cost assumptions,
• weekly trend snapshots.

The first version is public-safe and does not include private provider keys, internal routing details, or private infrastructure addresses.

Quickstart

Copy-Item .env.example .env
python .\scripts\benchmark_chat.py --runs 3

Environment

OPENAI_BASE_URL=https://your-gateway.com/v1
OPENAI_API_KEY=sk-example-xxx
OPENAI_MODEL=example-model

Output

The script prints JSON Lines so results can be appended to data/manual-runs.jsonl.

{"ok":true,"latency_ms":1234,"status_code":200,"model":"example-model"}

Public Reporting Rules

• Publish aggregate numbers only.
• Do not publish raw prompts containing private data.
• Do not publish real API keys.
• Do not publish private provider URLs, server IPs, or routing internals.
• If a run fails, keep the failure visible and explain the likely cause.

Suggested Weekly Metrics

Metric	Why it matters
Success rate	Shows basic reliability.
P50 latency	Shows typical user experience.
P95 latency	Shows tail behavior.
Error categories	Shows whether failures are user, provider, or gateway issues.
First-token latency	Shows streaming responsiveness.

License

MIT