RAG Evidence Checker

An offline-first CLI and JavaScript library that checks whether individual RAG claims are actually supported by the evidence they cite.

It is intentionally deterministic: the same JSON input produces the same JSON report without calling a model, uploading documents, or requiring an API key. The checker catches missing citation IDs, number mismatches, negation mismatches, and weak lexical coverage. It is a transparent CI baseline, not a replacement for domain-specific semantic evaluation.

Quick start

npm install
npm test
node ./bin/rag-evidence.js --input ./examples/passing.json --format pretty

Check an example containing evidence gaps. The command returns exit code 2 because unsupported claims were found:

node ./bin/rag-evidence.js --input ./examples/evidence-gaps.json

Input format

{
  "answer": "Optional original RAG answer",
  "evidence": [
    {
      "id": "policy-1",
      "source": "https://docs.example.com/policy",
      "text": "Refund requests are accepted within 30 days."
    }
  ],
  "claims": [
    {
      "id": "claim-1",
      "text": "Refund requests are accepted within 30 days.",
      "citations": ["policy-1"]
    }
  ]
}

Claims are explicit rather than inferred from prose so every result is auditable and can be traced back to the calling RAG pipeline.

Report

Each claim is classified as:

• supported: citations resolve and coverage is comfortably above the threshold.
• partial: citations resolve and pass the configured threshold, but coverage is close to the boundary.
• unsupported: citations are absent or unknown, numbers disagree, negation disagrees, or coverage is below the threshold.

The JSON report includes matched and missing tokens, unresolved citations, missing numeric values, reason codes, aggregate coverage, and an overall passed flag.

CLI options

--input <file|->
--output <file>
--format <json|pretty>
--min-score <0..1>
--fail-on <unsupported|partial|never>

Use --input - to read JSON from stdin. --fail-on unsupported is the default and is suitable for CI. Invalid input returns exit code 1; a valid report that violates the selected failure boundary returns exit code 2.

Library usage

import { checkRagEvidence } from "rag-evidence-checker";

const report = checkRagEvidence(input, { minimumScore: 0.55 });
if (!report.passed) console.error(report.claims.filter((claim) => claim.status === "unsupported"));

CI

The included GitHub Actions workflow runs the offline tests and verifies the passing sample. No secrets are required.

Security and privacy

• Documents remain on the machine running the checker.
• Source URLs are reported as provenance only; the CLI never fetches them.
• No model or external service is called.
• Treat reports as evidence for review, not proof that a claim is universally true.

License

Apache-2.0