An open-source workbench that evaluates, compares, and auto-optimizes LLM prompts.
Bring your own API key: one model answers, a second model scores the answer, and PromptVersus iterates toward the best prompt. Everything runs local-first, so your keys and data never leave your machine.
Prompt engineering is hard to measure. You tweak a prompt, the answer "feels" better, but you have no repeatable way to prove it — let alone improve it systematically. PromptVersus treats prompts as testable assets and closes the loop:
- Target model answers a standard question using your prompt.
- Judge model scores that answer with a structured rubric (per-dimension scores, summary, suggestions).
- The score reflects how well your prompt guided the model, measured against the rubric — surface fluency alone does not earn points.
- Optimizer model reads the deductions and rewrites the prompt, then the whole thing runs again — round after round — until the best version emerges.
A vague prompt scores low; a well-engineered one scores high. PromptVersus goes one step further and iterates toward a stronger prompt on its own.
- 🔄 Auto-optimization loop (flagship) — the core of PromptVersus: let the model improve your prompt for you. The target model answers, the judge scores and explains the deductions, and an optimizer model rewrites the prompt from that feedback. It repeats across multiple rounds and keeps whichever version scored highest across all of them. Round budget and manual abort are built in.
- 🎯 Two-model, prompt-aware evaluation — a target model produces an answer and a judge model returns a structured JSON verdict (total score, level, per-dimension breakdown, summary, and suggestions). The judge scores how well your prompt guided the model, so a fluent answer from a vague prompt does not get a free high score.
- 📊 Stability check — run the same prompt several times (repeatable testing) and see the score spread. Tells you whether a good result was real or just luck.
- ⚔️ Prompt A/B versus — run two prompts on the same question head to head and read the exact score delta.
- 🗂️ Prompt Library & Question Bank — your curated, reusable test suite. Save the prompts and questions you find effective into categorized repositories, then one-click import any of them straight into the workbench to test immediately. Pin your best ones to the top, search across categories, import/export as JSON/TXT, and restore items from the recycle bin. Over time your library becomes a battle-tested collection you can rerun any time.
- 🔌 Bring your own provider — works with any OpenAI-compatible endpoint, plus Anthropic, Gemini, and DeepSeek adapters. A built-in Mock Provider lets you try every feature with zero cost and no API key.
- 🔒 Local-first & private — prompts, questions, and evaluation history live in your browser's
localStorage. API keys are sent only with the outgoing evaluation request and are never persisted. - 📁 History & reports — every run and optimization is archived locally; filter, sort, and compare runs of the same prompt over time.
- Resilient & bilingual — an evaluation keeps running even if you navigate away and come back, and the whole UI switches between English and 简体中文.
Requirements: Node.js 18.18+ (or 20+).
git clone https://github.com/Nalnanananana/PromptVersus.git
cd PromptVersus
npm install
npm run devOpen http://localhost:3000. You can start immediately with the Mock Provider (no API key needed), or plug in a real provider in the config panel.
Your Prompt ──(system)──► Target Model ──answer──► Judge Model ──► Structured Score
Standard Question ──(user input)──┘ ▲ (0–100 + dimensions
Evaluation Focus ────────────────────────────────────────┘ + summary + tips)
- The standard question is treated as pure input material. What to do with it is decided entirely by your prompt.
- The evaluation focus is sent only to the judge model, never to the target model, so it cannot leak the rubric to the model being tested.
- The judge returns strict JSON. Malformed output is rejected outright, so a broken response never turns into a misleading score.
The auto-optimization loop wraps this cycle and drives it forward on its own:
┌──────────────────────────────────────────────────────┐
▼ │
Current Prompt ──► Evaluate (target + judge) ──► Score + deductions
▲ │
│ ▼
Optimizer Model ◄────────── rewrite from feedback ◄─── keep best-scoring version
(repeats until the round budget is reached; the highest-scoring prompt wins)
| Guide | Prompt Library |
|---|---|
 |
 |
-
API keys are entered in the config panel and sent only with the outgoing evaluation request. They are never written to
localStorage, the prompt library, or history. -
SSRF protection: the server-side evaluation route validates every provider Base URL and blocks loopback, private, and link-local addresses (including cloud metadata endpoints). To connect to a local gateway during development, set:
ALLOW_PRIVATE_PROVIDER_URL=true
Do not enable this on a publicly deployed instance.
-
This is a local-first tool. If you deploy it as a shared/public service, add network-level egress controls in addition to the built-in guard.
- Next.js 16 (App Router) · React 19 · TypeScript
- Tailwind CSS (Claymorphism design system)
- Zod for request validation
- Vitest + Testing Library
npm run dev # start the dev server
npm run build # production build
npm start # run the production build
npm test # run the test suiteAlready shipped: two-model evaluation, prompt-aware scoring, the auto-optimization loop, stability checks, prompt A/B versus, and the prompt/question repositories. Next up:
- Cross-model arena — send one prompt to several models at once and compare side by side.
- Cost caps & live spend display — hard per-run token/cost limits with a running total during optimization.
- RAG evaluation, Agent evaluation, and regression testing (community contributions welcome).
- Provider-level RPM/TPM rate limiting; one-click Vercel deploy plus a hosted demo.
Contributions are welcome. Please read CONTRIBUTING.md before opening a pull request. In short: open an issue to discuss non-trivial changes, keep PRs focused, add tests for new behavior, and make sure npm test and npm run build pass.
MIT © PromptVersus contributors