ghostfail

Detect silent AI failures — responses that look correct but aren't. GhostFail runs heuristic detectors on every AI turn, optionally escalates to an LLM-as-judge, and surfaces failures via CLI, dashboard, or webhook.

Install

Not yet on PyPI — install from source:

git clone https://github.com/aevryone/ghostfail
cd ghostfail
pip install -e .

# Optional: LLM judge support
pip install -e '.[anthropic]'

# Optional: better semantic similarity
pip install -e '.[embeddings]'

Quickstart

from ghostfail import GhostFail

gf = GhostFail()

# Evaluate a single turn
score = gf.log(
    request="What were Q3 margins?",
    response="Q3 margins were 42%.",
    source_material="Q3 margins were 18.2%.",
)

print(score.composite)       # 0.90 — high confidence trap
print(score.severity)        # "failure"
print(score.flags)           # ["confidence"]

Multi-turn conversations

with gf.conversation("conv-123") as conv:
    conv.log(request="What is Python?", response="A programming language.")
    conv.log(request="Tell me more", response="Here's a recipe for chocolate cake.")
    # ^ drift detector catches the topic change

Decorator mode

@gf.monitor(conversation_id="session-1")
def my_ai(user_message: str) -> str:
    return call_llm(user_message)

my_ai("What is 2+2?")  # automatically evaluated

What it detects

GhostFail runs three detectors on every turn:

Detector	What it catches	How it works
Confidence trap	Assertive responses that contradict source material	Claim extraction + source comparison; hedging/assertive language ratio
Drift	Conversations that silently wander off-topic	Semantic similarity (TF-IDF or sentence-transformers) between original question and latest response
Mismatch	Responses that answer the wrong question	Entity overlap + TF-IDF cosine similarity between question and answer

How detection works

Each turn flows through a three-stage pipeline:

Stage 1: Heuristic detectors

Three detectors run on every request-response pair. Each produces a score from 0.0 (clean) to 1.0 (failure).

Confidence trap detector checks whether the response contradicts source material:

1. Claim extraction — regex pulls (subject, linking_verb, value) triples from both the response and the source material. Examples: "Q3 margins were 23.4%" → ("q3 margins", "were", "23.4%"), "Revenue reached $4.2M" → ("revenue", "reached", "$4.2m").
2. Claim comparison — for each response claim, find source claims with overlapping subjects (token overlap >= 50%). If subjects match but values differ → contradiction (1.0). No matching subject → unsupported (0.7). Values match → grounded (0.0).
3. Composite — when claims are found: 0.6 * claim_score + 0.2 * confidence_ratio + 0.2 * specificity. Falls back to token-level overlap when no claims are extractable.

The confidence ratio measures assertive language ("definitely", "absolutely") vs hedging ("probably", "might be"). Specificity is the ratio of concrete tokens (numbers, proper nouns) to total words.

Drift detector measures topic shift across multi-turn conversations:

1. Computes semantic distance (TF-IDF cosine similarity, or sentence-transformers if installed) between the original question and the latest response.
2. Measures keyword decay — what fraction of original question keywords are missing from the response.
3. Composite: 0.6 * semantic_drift + 0.4 * keyword_decay.

Mismatch detector checks whether the response answers the right question:

1. Entity overlap — key nouns from the question should appear in the response.
2. Semantic relevance via TF-IDF cosine similarity between question and answer.
3. Composite: 0.65 * entity_overlap + 0.35 * semantic_relevance.

Stage 2: Composite scoring

The composite score uses a weighted blend of the top two detector scores:

composite = 0.75 * highest_score + 0.25 * second_highest_score

This requires partial confirmation from a second detector before reaching "failure" — a single noisy detector cannot push the composite past 0.6 alone (it would need a score above 0.8 with no support from other detectors).

Composite	Severity	Meaning
< 0.3	clean	No issues detected
0.3 - 0.6	suspicious	Possible issue — review recommended
>= 0.6	failure	Silent failure detected — alert fired

Stage 3: LLM judge escalation

When a heuristic score exceeds its threshold, GhostFail escalates to an LLM judge for a second opinion. The judge verdict replaces the heuristic score for that detector, and the composite is recomputed. A random sample of clean turns (configurable sample_rate) is also sent to the judge for calibration.

LLM Judge

When a heuristic score exceeds its threshold, GhostFail can escalate to an LLM judge for a second opinion. The judge is budget-capped and sampled:

judge:
  provider: anthropic
  model: claude-sonnet-4-20250514
  budget_usd: 10.0      # hard ceiling
  sample_rate: 0.1       # also sample 10% of clean turns

Set ANTHROPIC_API_KEY in your environment to enable the judge.

CLI

# Create a default config file
ghostfail init

# Analyze conversations from a JSONL file
ghostfail analyze --input conversations.jsonl --threshold 0.3

# View detection stats
ghostfail stats --days 30

# Launch the web dashboard
ghostfail dashboard --port 7710 --token MY_SECRET

Dashboard

The web dashboard shows flagged failures, per-conversation replay with drift visualization, and detection statistics.

Start it with a token for authentication:

ghostfail dashboard --token MY_SECRET
# or
export GHOSTFAIL_DASHBOARD_TOKEN=MY_SECRET
ghostfail dashboard

Without a token, the dashboard binds to 127.0.0.1 only.

Configuration

GhostFail loads config from (in order): built-in defaults, ghostfail.yaml, environment variables.

# ghostfail.yaml
thresholds:
  confidence_trap: 0.4    # escalate to judge above this
  drift: 0.35
  mismatch: 0.45
  composite_alert: 0.6    # fire alerts above this
  composite_block: 0.85   # block responses above this (if enabled)

judge:
  provider: anthropic
  model: claude-sonnet-4-20250514
  budget_usd: 10.0
  sample_rate: 0.1
  max_tokens: 512

alerts:
  webhook_url: ""          # POST JSON payload on failure
  log_level: WARNING
  block_enabled: false     # raise GhostFailBlocked on composite >= composite_block

db_path: ghostfail.db
background: true           # evaluate asynchronously (non-blocking)

Environment variables

Variable	Overrides
GHOSTFAIL_CONFIG	Config file path
GHOSTFAIL_DB_PATH	db_path
ANTHROPIC_API_KEY	judge.api_key
GHOSTFAIL_JUDGE_MODEL	judge.model
GHOSTFAIL_WEBHOOK_URL	alerts.webhook_url
GHOSTFAIL_DASHBOARD_TOKEN	Dashboard auth token

JSONL format

For batch analysis, provide a JSONL file where each line is:

{
  "conversation_id": "conv-1",
  "turns": [
    {"role": "user", "content": "What is X?"},
    {"role": "assistant", "content": "X is Y."}
  ],
  "sources": ["X is actually Z."]
}

Calibration test suite

The calibration suite in tests/fixtures/calibration.jsonl contains labeled examples used to measure detection precision and recall. Each line is a JSON object:

{
  "id": "fail_conf_01",
  "category": "failure",
  "detector": "confidence",
  "request": "What were our Q3 margins?",
  "response": "Q3 margins were 23.4%, showing strong improvement.",
  "source_material": "Q3 Financial Report: Gross margins were 18.2%.",
  "conversation": [],
  "expected_min_score": 0.5
}

Fields:

• category: "clean" (expected composite below expected_max_score), "failure" (above expected_min_score), or "borderline" (between both bounds).
• detector: which detector the example primarily targets.
• conversation: list of prior {request, response} turns for drift cases. Empty for single-turn tests.
• expected_min_score / expected_max_score: the bounds the composite score must satisfy.

Adding new calibration cases

1. Add a new JSON line to tests/fixtures/calibration.jsonl with a unique id.
2. Set category and realistic bounds — use wide bounds for borderline cases.
3. Run the calibration suite: pytest tests/test_calibration.py -v
4. The test_calibration_precision_recall test prints a summary with precision, recall, and F1 at the 0.6 failure threshold.

License

MIT