| title | Rubriq |
|---|---|
| emoji | 🔬 |
| colorFrom | green |
| colorTo | gray |
| sdk | docker |
| app_port | 7860 |
| pinned | false |
| license | mit |
| short_description | Score LLM outputs locally; get repair prompts that fix them |
Paste the prompt you gave an LLM and the output you got back. Rubriq scores the output against evaluation parameters drawn from published research, quotes the evidence for every finding, and generates ready-to-paste replacement prompts that target the specific failures it found — adapted to the model you name (GPT, Claude, Gemini, or open-weights).
Everything runs locally. No API calls, no keys, no data leaving your machine.
- Classify — rule signals plus embedding prototypes decide the task type (summarization, code generation, extraction, Q&A, creative writing, reasoning, rewriting) and split pasted source material from the instruction.
- Score — each task type gets the parameters the literature assigns it: NLI-based faithfulness when source text exists (SummaC method), deterministic instruction-following checks (IFEval method), embedding relevance (RAGAS-style), readability against the stated audience, hallucination-risk signals, static code analysis, style statistics.
- Report — per-parameter verdicts with quotes from the output itself. Parameters that local methods cannot judge say so instead of guessing: open-world fact checking and subjective quality are labeled as judge-mode territory, and proxies are labeled proxies.
- Repair — the flagship. Every diagnosed failure generates a complete replacement prompt using the named model family's own documented mechanisms (XML tags and quote-first grounding for Claude, delimiters and literal phrasing for GPT, few-shot and positive instructions for Gemini, constraint repetition for open-weights models).
Every parameter and template traces to a named source — see RESEARCH.md.
python -m venv .venv
.venv/Scripts/pip install -e ".[dev]" # first run downloads two small models (~350 MB)
.venv/Scripts/uvicorn rubriq.api.main:app --reloadPOST /evaluate with {"prompt": ..., "output": ..., "model_name": "gpt-4o"}
returns the full report. GET /health shows engine and model status.
Models: BAAI/bge-small-en-v1.5 (embeddings) and
cross-encoder/nli-deberta-v3-small (entailment). Both run CPU-only.
.venv/Scripts/python -m pytest tests -qThe suite includes curated prompt/output pairs with known planted failures (invented facts, broken code, fabricated citations, ignored constraints) and asserts Rubriq finds each one — plus a documented-limitation test that keeps the honest boundary of the coverage proxy visible.
Step 1 (evaluation engine) complete. Frontend and deployment are Step 2.
An LLM-as-judge engine plugs in behind the same Scorer interface later.