shipgate

A drop-in eval gate for CI. Evals decide what ships.

Every AI feature looks fine right up until a prompt tweak, a model update, or a new edge case quietly breaks it — and nobody notices until a user does. Manual testing catches what you thought to check. shipgate catches the regression you didn't: point it at a config of cases + assertions and it computes a pass rate, prints a report, and exits non-zero when quality drops below a threshold or regresses against a saved baseline. Wire it into CI and a regression fails the build before it reaches production.

The deterministic assertions (citations, refusal, word-count, regex, equality) need no API key, so the gate runs on every commit for free. An optional llm_judge assertion uses an LLM when a key is present and degrades gracefully when it isn't.

A real run: a prompt tweak makes the assistant chattier and it drops its citation. The score (0.80) still clears the 0.75 threshold — but it regressed from the saved baseline (1.00), so the gate fails the build. A fixed bar would have shipped it; that red ✗ is the point.

Use it as a GitHub Action

# .github/workflows/eval-gate.yml
name: eval-gate
on: [push, pull_request]
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: enached134-ctrl/shipgate@v1
        with:
          config: evals/rag.yaml
          baseline: evals/baseline.json   # optional: fail on regression

The build turns red the moment your eval score drops. That red X is the point.

Or as a CLI

pip install pyyaml
python shipgate.py run evals/rag.yaml --threshold 0.9 --baseline baseline.json

The config

Each case carries the model's output and the assertions it must satisfy. A case passes only if all its assertions pass; the score is the fraction of cases that pass.

threshold: 0.9          # fail the build below this pass-rate
cases:
  - name: cites its source
    output: "The Eiffel Tower was completed in 1889 [2]."
    assert:
      - { type: regex, value: "\\[\\d+\\]" }   # must carry a [N] citation
      - { type: contains, value: "1889" }

  - name: abstains when the context can't answer
    output: "I can't answer that — the context does not contain that figure."
    assert:
      - { type: is_refusal, value: true }

In a real pipeline you generate output from your app first (a script, a notebook, your CI job) and hand the results to shipgate — it is the gate, not the runner, so it drops into whatever you already use.

Assertions

type	passes when
contains / icontains / not_contains	substring is / isn't present (ci = case-insensitive)
regex	the pattern matches
equals	output equals the value (trimmed)
is_refusal	output is (or isn't) an "I don't know / not in the context" abstention
min_words / max_words	length is within bounds
llm_judge	an LLM grades the output against a rubric (optional; needs a key)

The regression gate

A threshold catches "this is bad." A baseline catches "this got worse" — the failure mode that actually bites, because yesterday's 0.98 silently becoming today's 0.91 never trips a fixed bar.

python shipgate.py run evals/rag.yaml --update-baseline   # accept current score as the bar
python shipgate.py run evals/rag.yaml --baseline baseline.json --tolerance 0.02

The gate fails if the score drops more than tolerance below the baseline.

Optional LLM judge

- name: answer is grounded
  output: "The Eiffel Tower is 330 metres tall [1]."
  assert:
    - type: llm_judge
      rubric: "PASS if every claim is supported by an Eiffel-Tower context and carries a citation."

Set OPENAI_API_KEY or ANTHROPIC_API_KEY (and optionally SHIPGATE_JUDGE_MODEL) to grade it for real; with no key it's skipped and passes, so the gate never breaks a build just because a secret is missing.

Why this exists

Most "production AI" work is the unglamorous layer: deterministic gates instead of prompt-hope, structured outputs you can validate, retrieval that cites or refuses, and evals in CI so a regression fails a build instead of a customer. shipgate is that last piece, small enough to drop into any repo.

Built by Daniel Enache — see also groundcheck (a local groundedness judge you can wire in as an llm_judge) and mcp-vitals.

License

MIT