⚡ PromptDiff

Fast, CI-Native Regression Testing for LLM Prompts ("Git Diff for Prompts")

Catch silent quality regressions, schema breakages, latency spikes, and cost inflation before merging prompt changes.

🌐 Live Interactive Demo • 📖 Docs & API • 🚀 Quickstart • 🔍 Core Workflow • 🩺 Environment Doctor • 🐍 Python SDK • 📚 Recipe Catalog • 🧪 Pytest Integration • 📦 Installation

🌐 Try it Live in Your Browser: Test the interactive prompt diff playground, token cost calculator, and AST mutation visualizer without installing anything: latryee.github.io/promptdiff

• 💡 Why PromptDiff?
• ⚖️ Honest Comparison: PromptDiff vs Alternatives
• 🚀 Quickstart in 30 Seconds
• 🩺 Environment Doctor
• 🔍 How It Works: Pull Request Quality Gate
• 📚 Curated Recipe Catalog
• 🧪 Pytest Plugin Integration
• 🐍 Python SDK
• 📦 Installation & Modular Extras
• 🧩 Advanced & Extended Modules
• 🛡️ Production Engineering Standards & Quality Assurance
• 🏛️ Architecture Deep-Dive
• 🔒 Data Privacy & Local Storage Disclosure
• 🤝 Community & Contributing
• 📈 Star History
• 📄 License

Modifying system prompts or switching models often leads to unexpected side effects: broken JSON formatting, subtle hallucinations, increased latency, or ballooning token costs.

promptdiff brings standard software regression testing to prompt engineering:

• CLI & CI/CD First: Run lightweight local evaluations in seconds or gate pull requests in GitHub Actions.
• Deterministic Caching: SHA-256 keyed SQLite disk cache ensures identical runs cost $0 and execute in milliseconds.
• Accurate Token & Cost Gating: Model pricing registry with local tokenizers calculates exact financial and latency deltas.
• Hardened Subprocess Sandbox: Isolated code execution runner with resource limits and exploit-tested AST/memory sandboxing.
• Rich Reports: Standalone, zero-dependency interactive HTML reports and automated sticky PR comments (Explore Live Demo).

Selecting the right evaluation tool depends heavily on your team's workflow, runtime stack, and data sovereignty requirements:

• Choose PromptDiff if you are a Python/MLOps team that treats prompts as code in Git, wants pytest-native integration, requires 100% data sovereignty without external cloud dependencies, and needs fast PR regression gates.
• Choose promptfoo if you have a Node.js/TypeScript stack, want a rich browser-based red-teaming workspace, or need pre-packaged adversarial jailbreak test suites.
• Choose LangSmith if your primary requirement is distributed production trace visualization across multi-agent LangChain graphs.
• Choose Braintrust if you want an enterprise-managed centralized cloud evaluation platform with web-based team playground collaboration.

# 1. Install promptdiff core (lightweight, zero heavy ML dependencies)
pip install promptdiff-eval

# 2. Scaffold a starter evaluation project
promptdiff init my-evals
cd my-evals

# 3. Run regression tests offline (Zero API keys required)
promptdiff test prompts/system_v1.txt prompts/system_v2.txt \
  --inputs testcases.jsonl \
  --mock \
  --eval "latency,cost,similarity" \
  --assert "cost_delta <= 15%, latency_delta <= 20%" \
  --export-html report.html

Diagnose local environment readiness, LLM API keys, optional packages (tiktoken, sentence-transformers, mlflow, wandb), and disk cache engine with a single command:

promptdiff doctor

Integrate promptdiff directly into your CI/CD pipeline to block regressions before merging to main:

promptdiff test prompts/system_v1.txt prompts/system_v2.txt \
  --inputs datasets/testcases.jsonl \
  --model gpt-4o \
  --eval "json_validity,latency,cost,similarity,llm_judge,faithfulness,security" \
  --assert "cost_delta <= 10%, latency_delta <= 15%, similarity >= 0.75, faithfulness >= 0.85" \
  --fail-on-regression \
  --export-markdown report.md

promptdiff test prompts/system_v1.txt prompts/system_v2.txt \
  --inputs datasets/large_testcases.jsonl \
  --tags "critical,edge-case" \
  --limit 200 \
  --timeout 45.0 \
  --redact \
  --experiment-id "release-2026-q3" \
  --fail-on-regression

• --tags: Filter test cases by comma-separated tags (only executes matching cases).
• --limit: Stream and evaluate only the first $N$ matching test cases (ideal for smoke tests).
• --timeout: Enforce per-testcase execution timeout in seconds (default: 30.0s).
• --redact: Mask API keys (OpenAI, Anthropic, Gemini, AWS), Bearer/JWT tokens, and PII (emails, SSNs) from outputs, logs, and exported reports.
• --experiment-id: Track evaluations with a custom run ID for multi-run database telemetry.

For non-composite CI environments (Jenkins, GitLab CI, Buildkite, or custom GitHub Actions steps), use the standalone PR commenting script:

# Run regression evaluation exporting report JSON
promptdiff test prompts/system_v1.txt prompts/system_v2.txt \
  --inputs datasets/testcases.jsonl \
  --mock \
  --export-json report.json

# Post or update sticky Markdown evaluation comment on PR
python scripts/pr_commenter.py \
  --report report.json \
  --repo "$GITHUB_REPOSITORY" \
  --pr "$PR_NUMBER" \
  --token "$GITHUB_TOKEN"

Pull ready-to-use prompt templates, test suites, and tailored evaluators for your specific use case:

# List all domain recipes
promptdiff recipe list

# Pull a specific starter kit
promptdiff recipe pull rag-qa           # RAG Grounding & Faithfulness
promptdiff recipe pull json-extractor   # Strict Structured Output & Schema AST
promptdiff recipe pull sql-gen          # Natural Language to SQL
promptdiff recipe pull security-guard   # Prompt Injection & Extraction Defense

PromptDiff includes an official, native pytest plugin (pytest-promptdiff) for seamless test automation:

Define regression tests declaratively on test functions. Quality assertions are automatically verified by the plugin:

import pytest
from promptdiff.core.models import TestCase


@pytest.mark.promptdiff(
    v1="prompts/support_v1.txt",
    v2="prompts/support_v2.txt",
    dataset=[
        TestCase(id="tc1", vars={"query": "How do I reset my password?"}),
        TestCase(id="tc2", vars={"query": "Request billing refund"}),
    ],
    model="gpt-4o",
    eval="json_validity,latency,cost,similarity",
    assertions=["cost_delta <= 15%", "latency_delta <= 20%"],
    mock=True,
)
def test_support_prompt_regression(report):
    # 'report' fixture is automatically injected with the DiffReport
    assert report.verdict.passed
    assert report.verdict.cost_delta_pct < 15.0

For dynamic setups requiring conditional test generation:

# tests/test_prompts.py
import pytest
from promptdiff.core.models import TestCase


@pytest.mark.asyncio
async def test_support_prompt_regression(prompt_diff):
    report = await prompt_diff.compare(
        v1="prompts/support_v1.txt",
        v2="prompts/support_v2.txt",
        test_cases=[
            TestCase(id="tc1", vars={"query": "How do I reset my password?"}),
            TestCase(id="tc2", vars={"query": "Request billing refund"}),
        ],
        model="gpt-4o",
        mock=True,
    )
    assert report.verdict.passed, f"Regression detected: {report.verdict.failed_assertions}"

Run with standard pytest:

pytest tests/test_prompts.py

Use promptdiff programmatically inside Python applications or evaluation scripts:

import promptdiff
from promptdiff.core.models import TestCase

# Run regression evaluation
report = promptdiff.compare(
    v1="prompts/support_v1.txt",
    v2="prompts/support_v2.txt",
    dataset=[
        TestCase(id="tc1", vars={"query": "Reset password"}),
        TestCase(id="tc2", vars={"query": "Billing question"}),
    ],
    model="gpt-4o",
    mock=True,
    assertions=["cost_delta <= 15%", "latency_delta <= 20%"],
)

print(f"Passed: {report.verdict.passed}")
print(f"Cost Delta: {report.verdict.cost_delta_pct:.1f}%")

# Compress prompt tokens while maintaining quality
shrunk = promptdiff.shrink(
    prompt="Please kindly act as an AI and answer: {{query}}",
    dataset=[TestCase(id="1", vars={"query": "Help"})],
    mock=True,
)
print(f"Compressed Prompt: {shrunk.compressed_prompt}")

PromptDiff is built with a slim, featherweight core and modular extras so you only install what you need:

# Core CLI & CI runner (typer, rich, pydantic, httpx, jinja2, pyyaml, tenacity, numpy)
pip install promptdiff-eval

# Semantic dense embedding similarity (sentence-transformers)
pip install "promptdiff-eval[semantic]"

# Interactive split-screen Terminal UI (Textual)
pip install "promptdiff-eval[tui]"

# Streamlit telemetry web dashboard
pip install "promptdiff-eval[ui]"

# All optional components
pip install "promptdiff-eval[all]"

PromptDiff is built to enterprise MLOps standards with zero tolerance for unverified code or silent regressions:

Detailed system design documentation, architectural diagrams, mathematical formulations, and core engineering decisions are available in:

👉 Technical Architecture & System Design Deep-Dive (PORTFOLIO.md)

PromptDiff operates under an absolute local-first, zero-telemetry exfiltration guarantee:

• Local Persistence Only: Evaluation runs and token metrics are written to local SQLite storage (.promptdiff/telemetry.db). No prompt contents, outputs, or traces are ever sent to external cloud servers.
• Automated Retention Management: Automatically delete historical records older than $N$ days with --db-retention-days <N> or run promptdiff db prune --days 14.
• Sensitive Data & PII Redaction: Pass --redact (or redact=True in Python SDK) to automatically mask API keys (OpenAI, Anthropic, Gemini, AWS, Hugging Face), Bearer/JWT tokens, and sensitive PII (emails, SSNs, credit cards) in terminal outputs, logs, and exported artifacts.
• Ephemeral Storage: Run with --db-path ":memory:" for zero disk persistence.
• Complete security documentation and disclosure SLAs are available in SECURITY.md.

• Contributing Guide
• Code of Conduct
• Security Policy
• Changelog

Distributed under the MIT License. See LICENSE for more information.