PromptTrace

LLM observability for developers who care about cost, latency, and reliability.

PromptTrace gives your AI apps a black-box recorder — every model call is traced, analyzed, and surfaced in a real-time dashboard. Built because I needed it for my own projects and nothing else quite fit.

---

---

Why PromptTrace?

As soon as you're running LLM calls in production, you need answers to:

• 🔥 Which endpoint is burning the most money?
• 🐌 Where are the latency spikes happening?
• 💥 What's actually failing — quota, network, parse errors?
• 🔁 Are retries helping or just masking problems?

PromptTrace answers all of these, locally, with zero cloud dependency.

---

Features

• 📊 Real-time dashboard — KPI overview, trace table, failure cards, cost charts
• 🧩 Drop-in SDK — withTrace() and withRetry() wrappers for any provider
• 💰 Cost estimation — per-call USD estimates with versioned pricing
• 🔍 Error taxonomy — quota | permission | network | timeout | parse | unknown
• 🔒 Privacy-first — prompts are redacted and hashed by default, never leave your machine
• 📤 Export — JSON + Markdown reports
• ⚡ Quick Add — in-dashboard snippet generator for browser & Node integrations
• 🔄 Trace rotation — bounded log files, configurable retention

---

Quick Start

git clone https://github.com/WilliamK112/prompttrace
cd prompttrace
npm install && npm run build

# Seed with demo data and launch dashboard
node dist/index.js demo
node dist/index.js dashboard --port 4310
# Open http://localhost:4310

---

SDK Integration

Node.js / TypeScript

import { withTrace } from './sdk/trace.js';
import { withRetry } from './sdk/retry.js';

// Wrap any LLM call
const { result, retries } = await withRetry(
  () => gemini.generateContent(prompt),
  { maxRetries: 2, baseDelayMs: 400 }
);

await withTrace(
  {
    provider: 'gemini',
    model: 'gemini-1.5-flash',
    endpoint: 'story.generate',
    requestType: 'text',
    prompt,
    responsePreview: result.text,
    retries
  },
  async () => result
);

Browser (fetch wrapper)

Use the Quick Add panel in the dashboard to generate a ready-to-paste browser snippet for any provider and model — no config files needed.

---

CLI Reference

# Metrics overview
node dist/index.js analyze --range 7d

# Raw traces (filterable)
node dist/index.js traces --range 7d --status error --limit 50

# Failures report
node dist/index.js failures --range 7d

# Export report
node dist/index.js export --format md --range 7d
node dist/index.js export --format json --range 7d

# Launch live dashboard
node dist/index.js dashboard --port 4310

# Track OpenClaw command usage
node dist/index.js openclaw --endpoint openclaw.tool.exec -- openclaw status

# Rotate trace log (keep bounded)
node dist/index.js rotate --keep-lines 5000

---

Architecture

prompttrace/
├── src/
│   ├── sdk/
│   │   ├── trace.ts        # withTrace() — core instrumentation wrapper
│   │   ├── retry.ts        # withRetry() — backoff + retry metadata
│   │   ├── cost.ts         # Per-model USD cost estimation
│   │   ├── errors.ts       # Error taxonomy classifier
│   │   └── redact.ts       # PII redaction + prompt hashing
│   ├── collector/          # JSONL trace storage
│   ├── aggregator/         # P50/P95, cost rollups, by-endpoint breakdowns
│   ├── dashboard/
│   │   └── index.html      # Single-file React-free dashboard (dark, no dependencies)
│   └── cli/                # CLI commands (analyze, traces, failures, export, demo)
└── .prompttrace/
    └── traces.jsonl        # Local trace log (stays on your machine)

---

Privacy & Security

• Prompts are never stored in full — only a truncated, redacted preview and a SHA-256 hash
• No telemetry, no cloud — all data stays in .prompttrace/ on your machine
• Redaction masks emails, phone numbers, and API-key-like patterns automatically
• To enable full prompt logging (local only): PROMPTTRACE_FULL=1
• Delete .prompttrace/ at any time to wipe all traces

---

Deploy a Hosted Demo

Want to share a live dashboard (read-only, seeded with demo data)?

# 1. Generate demo traces
node dist/index.js demo

# 2. Deploy to Railway / Fly.io / Render
# The dashboard server serves on PORT env var automatically
npm run build && node dist/index.js dashboard

One-click Railway deploy:

---

Roadmap

• OpenTelemetry export (OTLP)
• Multi-project / team trace namespaces
• Python SDK (pip install prompttrace)
• Streaming token tracking
• Webhook alerts on failure spikes
• GitHub Actions CI integration

---

The Story

I built this while working on a multimodal AI project and realized I had no visibility into what my LLM calls were actually doing — which endpoints were slow, what was failing silently, and how much it was costing. Existing tools were either too heavyweight, required a cloud account, or didn't support the providers I was using.

So I built PromptTrace in a week. It now runs alongside every AI project I work on.

If you're building AI products, you'll hit these same problems. This is the tool I wish existed.

---

License

MIT

Author and Portfolio

This project is by William Kang (Ching-Wei Kang), a UW-Madison Computer Science and Data Science student building AI products, backend systems, and developer tools.

See the William Kang official profile at williamkang.com/william-kang.html.

See the canonical project profile at williamkang.com/william-kang-projects.html.

See the exact-name identity page at williamkang.com/ching-wei-kang.html.