Plumb

A rule engine for testing the validity of AI outputs - deterministic checks and deterministic repair for structured model output, verified against the request that produced it.

Plumb answers one question: does this structured output faithfully represent what was actually asked for, and if not, can we fix it in code instead of asking the model again?

A small, stdlib-only Go library (module github.com/srinivasgumdelli/plumb, Go 1.21+). Not a guardrails framework, not an eval harness, not an observability platform: the fidelity-check pattern made explicit, testable, and dependency-free, for Go services that act on model output.

If you work in Python, read Prior art and honest comparison before anything else. Much of what follows can be assembled from Instructor and Guardrails AI, and you may not need a separate library at all.

This README teaches the model and gets you productive; every exported symbol carries a full doc comment, so go doc is the reference.

---

The problem: fidelity faults

As LLM output moves from prose to structured data that drives actions - tool calls, code, cloud specs, SQL, transactions, config - one failure class becomes dominant and dangerous:

A field that is syntactically valid, safe, well-typed, and semantically wrong relative to what was asked - executed faithfully, reported healthy.

The canonical archetype:

Prompt:   "a Postgres staging DB in Mumbai"
Output:   { "name": "db-mumbai", "tags": {"region":"mumbai"},
            "reasoning": "...staging database in Mumbai...",
            "region": "us-east-1" }     ← the one field that acts
Result:   real infrastructure created in us-east-1, status "healthy"

The model understood the intent - it echoed "Mumbai" three times - and still emitted the wrong region code in the field that matters. Models are reliable at extracting intent and unreliable at recalling exact enumerated tokens (region codes, instance types, IAM actions, table names, currency codes, SKUs, timezones). Schema validation passes, safety scanning passes, and every downstream system treats the output as ground truth. The fault is silent.

A quieter variant has the same ending: sometimes the request is the ambiguous part. "Replicate the Mumbai DB to Frankfurt" names two regions that disagree, and a checker that can only answer "matched" or "no signal" must abstain - which is indistinguishable from passing. Plumb's extractors report a third outcome, Ambiguous, so the conflict surfaces as a Flag.

Call it a fidelity fault. Nothing catches it by default, because the mature "checking AI" categories are aimed at other axes: safety guardrails ask whether the output is harmful, evals ask whether it is good offline on a test set, and observability records what the model did, after the fact. (Where Plumb sits has the full comparison.)

Two of those can be pushed into this lane with work, and the prior art section shows exactly how. The claim here is narrower than "uncovered": none of them is pointed at this axis, so the check exists only if you decide it should and then build it. Plumb's lane is making that decision cheap and explicit: runtime, per-request, semantic fidelity of structured output, with intervention.

The two load-bearing ideas

1. Fidelity, not isolation. A check receives both the typed output and the original request, so it can assert "the region field matches the location the user actually named." Schema validation and safety scanning cannot express this, because neither one sees the prompt.

2. Repair, not re-ask. The common reflex for a bad output is detect → re-prompt: slow, non-deterministic, unbounded, unauditable. Plumb's first move is detect → fix in code. Mapping "Mumbai" to ap-south-1 is a lookup, not a generation - deterministic, free, idempotent, explainable. Re-asking is a bounded, opt-in escalation (core Retry), never the default. Guardrails AI's on_fail="fix" shares this instinct; the difference is that repair is the default outcome here rather than one policy among six.

Install

go get github.com/srinivasgumdelli/plumb

Quickstart

Declare rules for the fields that matter, evaluate, branch on the report. regions is a gazetteer.Table compiled from your own terms - the worked example builds one.

type DeploySpec struct {
    Region string `json:"region"`
}

// One rule: the region field must match the location the request named.
// regions.Match does the extraction; the repair is automatic.
var rules = []plumb.Check[DeploySpec]{
    plumb.Fidelity("region", func(s *DeploySpec) *string { return &s.Region },
        regions.Match),
}

// Call this with the model's output already parsed into spec.
func handle(spec *DeploySpec, userPrompt string) error {
    report := plumb.Evaluate(spec, userPrompt, rules...)
    if report.IsBlocked() {
        return clarify(report.Blocks()) // fail closed - never act
    }
    return act(*spec) // already repaired; report.Corrections() says what changed
}

That's the whole loop. Prompt says "…in Mumbai…", the model wrote us-east-1, spec.Region is now ap-south-1, and the report records the repair. Warnings for a human gate come from report.Flags(); telemetry from the otel package.

Core concepts

Five nouns:

• Output (*T) - the typed, already-parsed result of a model call. Plumb operates on your Go struct, not raw JSON.
• Prompt (string) - the request that produced it. Everything else a check needs (existing state, budgets, catalogs, defaults) arrives via closure capture: write a constructor taking it as a typed parameter and returning the Check. The core never performs I/O to obtain it.
• Check - a named unit of logic inspecting (*T, string), returning zero or more findings. A check may mutate the output to repair it.
• Finding - one result: check name, severity, a dot/bracket path into the output ("region", "tables[users]"), a message, and for repairs the before/after values.
• Report - the aggregate, with helpers: IsBlocked(), Corrections(), Flags(), Blocks().

A check is the whole extensibility surface. Adding a guardrail = writing a check + a table test. There is no plugin system, DSL, or registry.

Every finding carries a severity, and each maps to one caller behavior: Correct means the check repaired the output in place, so proceed with the repaired value; Flag means proceed, but to a human; Block means never act. The section after the next one is the full table.

A worked example, end to end

The quickstart is the shortest loop that works. This is the same archetype at realistic size: one request, one rule set, one evaluation, all three severities landing, and the report actually handled. Nothing is elided and none of it requires opening examples/.

1. The request, and what the model produced

A customer asks for infrastructure in a named region:

"a Postgres staging DB in Mumbai with 4 TB of storage"

The model returns well-formed, schema-valid JSON. Parsed into the host's type:

// The structured output a provisioning request produces.
type DeploySpec struct {
    Name      string `json:"name"`
    Region    string `json:"region"`
    Tier      string `json:"tier"`
    StorageGB int    `json:"storage_gb"`
}

// Exactly what the model emitted for the request above.
var produced = DeploySpec{
    Name:      "db-mumbai-staging", // it clearly understood "Mumbai"
    Region:    "us-east-1",         // and still wrote the wrong code here
    Tier:      "standard",          // the request named no tier; the model chose
    StorageGB: 4096,                // over this host's ceiling
}

Nothing in that struct is malformed, unsafe, or off-enum. A schema validator passes it. Region is the field that acts.

2. The vocabulary

gazetteer ships no terms. You bring the words your customers write and the codes your provisioning API takes:

// regionTable compiles the host's own region vocabulary. Several terms may
// name one code, and the codes are terms too, so a request naming
// "ap-south-1" has stated its intent as plainly as one naming Mumbai.
func regionTable() (gazetteer.Table, error) {
    return gazetteer.NewTable(map[string]string{
        "mumbai":       "ap-south-1",
        "ap-south-1":   "ap-south-1",
        "frankfurt":    "eu-central-1",
        "eu-central-1": "eu-central-1",
        "virginia":     "us-east-1",
        "n. virginia":  "us-east-1",
        "us-east-1":    "us-east-1",
    }, []string{
        // Blocking terms: names that contain a term but mean something else.
        // Without this, "a DB in West Virginia" resolves to us-east-1 and the
        // check confidently repairs toward a region nobody asked for.
        "west virginia",
    })
}

That error is not decoration. NewTable refuses to compile a vocabulary that cannot behave - a term normalizing to nothing that would match every text, two terms collapsing to one key but naming different codes, a blocking term that blocks nothing - so the failure surfaces at startup instead of as a rule that silently never fires. Step 4 handles it.

3. The rule set

// Rules is the whole verification policy for DeploySpec. Everything the checks
// need beyond the output and the prompt - the vocabularies, the tier
// allowlist, the storage ceiling - arrives as typed parameters and is captured
// by closure. The core never fetches any of it.
func Rules(regions, tiers gazetteer.Table,
    allowedTiers map[string]bool, maxGB int) []plumb.Check[DeploySpec] {

    return []plumb.Check[DeploySpec]{
        // The flagship rule. Ambiguous requests Block rather than Flag here,
        // because a database in the wrong region is expensive to undo.
        plumb.Fidelity("region", func(s *DeploySpec) *string { return &s.Region },
            regions.Match, plumb.OnAmbiguous(plumb.Block)),

        // A bare Check literal: host policy no primitive can express. A nil map
        // reads false, so an unconfigured allowlist blocks everything.
        {Name: "tier.allowed", Inspect: func(out *DeploySpec, _ string) []plumb.Finding {
            if allowedTiers[out.Tier] {
                return nil
            }
            return plumb.Blocked("tier", "tier "+out.Tier+" is not offered here")
        }},

        // A bare Check that repairs in place, and records what it changed.
        {Name: "storage.max", Inspect: func(out *DeploySpec, _ string) []plumb.Finding {
            if out.StorageGB <= maxGB {
                return nil
            }
            was := out.StorageGB
            out.StorageGB = maxGB
            return plumb.Corrected("storage_gb", fmt.Sprint(was), fmt.Sprint(maxGB),
                "clamped to this host's ceiling")
        }},

        // The drift shape: the request was silent and the model chose. Keyed on
        // Unresolved specifically, so a request naming two conflicting tiers
        // stays the fidelity check's business instead of producing a second
        // finding for one cause.
        {Name: "tier.drift", Inspect: func(out *DeploySpec, prompt string) []plumb.Finding {
            if _, res := tiers.Match(prompt); res == plumb.Unresolved {
                return plumb.Flagged("tier",
                    "the request named no tier; this is the host default")
            }
            return nil
        }},
    }
}

Four checks, four of the shapes in Fidelity-fault shapes: one primitive and three struct literals, which is the usual ratio.

4. Wiring it up

// Built once at startup. A vocabulary is source code, so a fault in it is a
// bug: refuse to serve traffic behind a rule that can never fire.
func newRules() ([]plumb.Check[DeploySpec], error) {
    regions, err := regionTable()
    if err != nil {
        return nil, err
    }
    tiers, err := gazetteer.NewTable(map[string]string{
        "standard": "standard",
        "premium":  "premium",
    }, nil)
    if err != nil {
        return nil, err
    }
    return Rules(regions, tiers,
        map[string]bool{"standard": true, "premium": true}, 1024), nil
}

5. The chokepoint

// Provision is the single place the model's output is checked: after parsing,
// before anything is persisted or acted on.
func Provision(spec *DeploySpec, prompt string, rules []plumb.Check[DeploySpec]) error {
    report := plumb.Evaluate(spec, prompt, rules...)

    // Block: no safe deterministic fix exists. Fail closed - nothing is
    // provisioned, including the repairs the other checks already made.
    if report.IsBlocked() {
        return clarify(report.Blocks())
    }

    // Correct: repaired in place, and Was/Now is the audit trail. Recording
    // it is the invariant - a repair may be silent to the user, never to
    // operators.
    for _, f := range report.Corrections() {
        recordCorrection(f.Check, f.Path, f.Was, f.Now, f.Message)
    }

    // Flag: proceed, but not unattended. Each flag becomes a line on the
    // approval surface a human signs before the plan executes.
    if flags := report.Flags(); len(flags) > 0 {
        return submitForApproval(*spec, flags)
    }

    return act(*spec)
}

// Host-supplied sinks, stubbed here so the example compiles as written.
func recordCorrection(check, path, was, now, msg string)          {}
func submitForApproval(s DeploySpec, f []plumb.Finding) error     { return nil }
func clarify(blocks []plumb.Finding) error                        { return nil }
func act(s DeploySpec) error                                      { return nil }

6. What it produces

prompt   "a Postgres staging DB in Mumbai with 4 TB of storage"

field         model wrote     after Evaluate
region        "us-east-1"     "ap-south-1"
storage_gb    4096            1024
tier          "standard"      "standard"

report (evaluation order, which is check argument order)
  correct  fidelity.region  region      "us-east-1" -> "ap-south-1"
  correct  storage.max      storage_gb  "4096" -> "1024"
  flag     tier.drift       tier        the request named no tier; this is
                                        the host default

not blocked -> 2 corrections written to the audit log, 1 flag sent to the
               approval gate, and the plan a human approves is the REPAIRED
               spec, not the one the model wrote.

Change one thing - the customer asks for a tier this host does not offer - and the same rule set fails closed:

prompt   "a gold-tier Postgres DB in Mumbai"

report
  correct  fidelity.region  region  "us-east-1" -> "ap-south-1"
  block    tier.allowed     tier    tier gold is not offered here

blocked -> nothing is provisioned. The region repair still happened in the
           struct, and the Block is precisely the signal not to act on it.

7. The same rules under a stricter policy

The rules do not change; the policy on the arms where the extractor did not resolve does:

// The same field, at a host where a request the vocabulary cannot read must
// not pass silently, and a self-contradictory one must not pass at all.
strictRegion := plumb.Fidelity("region",
    func(s *DeploySpec) *string { return &s.Region },
    regions.Match,
    plumb.OnUnresolved(plumb.Flag),
    plumb.OnAmbiguous(plumb.Block))

The request says	Default policy	With the options above
"…in Mumbai"	Correct - repaired to ap-south-1	same
"…in Reykjavik" - a place this vocabulary does not carry	nothing. An empty report, byte-identical to a clean pass	Flag on region, naming the field whose value could not be traced to the request
"replicate the Mumbai DB to Frankfurt"	Flag, field left exactly as the model wrote it	Block - nothing proceeds until the request is resolved

The middle row is the one worth internalizing. No curated vocabulary is ever complete, so for a field whose value must be traceable to the request, the extractor recognizing nothing is the finding.

Severity → action

Every finding carries a severity that maps to an action. This is the spine of the library:

Severity	Meaning	What the caller does
Correct	The check found a fault and deterministically fixed it in place.	Proceed with the repaired output; record/surface the correction.
Flag	Suspicious, ambiguous, or a low-confidence default was applied.	Proceed to a human/approval gate with the warning visible.
Block	No safe deterministic fix, too risky to pass.	Fail closed - reject or escalate. Never act.

Block is the zero value of Severity - a hand-built Finding{} fails closed instead of reading as a silent repair. Correct is preferred over Flag over Block, whenever each is safe: a correction must be deterministic and unambiguous to qualify; anything speculative is a Flag. A Report is blocked if any finding is Block.

Repair invariants - a repair MUST be deterministic (same input, same output), idempotent (a second run fixes nothing), toward the stated intent (never toward a library default that overrides an explicit request), and recorded (every repair emits a Correct finding with Was/Now - never silent to operators, even when silent to users).

Fail-safe on ambiguity - fidelity checks never guess, and never abstain silently. A deterministic extractor returns a Resolution alongside its value, and each of the three is a different statement about the request:

Resolution	The request	Fidelity does, by default
Resolved	names exactly one value	repair the field toward it, emit Correct
Ambiguous	names a value the extractor could not resolve to exactly one - two candidates that disagree, or a single one it could not read	leave the field as written, emit Flag
Unresolved	names nothing the extractor recognizes	leave the field as written, stay silent

Unresolved is Resolution's zero value, so an extractor that returns without naming an outcome abstains rather than asserts. A resolution outside the three can only come from a broken extractor, and Fidelity neither trusts it (no repair) nor ignores it (that would be the silent pass this library exists to prevent): it leaves the field alone and emits a Block. The Ambiguous message names no candidate, so no guess leaks through the text.

Choosing the outcome policy

The two non-Resolved rows above are defaults, not the library's opinion about your risk - whether a request your extractor could not read should stop a deployment depends on a risk appetite the library cannot know - so Fidelity takes OnUnresolved and OnAmbiguous options. Step 7 of the worked example shows both, and what each changes.

OnUnresolved is the one that closes a silent-failure path, and SilentOnUnresolved() names the default so you can restore it after inheriting an override.

Two things an option cannot do. There is deliberately no SilentOnAmbiguous: the severity is yours, whether a detected problem reaches the report is not. And Correct is rejected, because nothing is repaired when the extractor does not resolve - a check built from a rejected option fails closed with one Block on every prompt rather than downgrading to something that looks like it works. See the FidelityOption godoc for the full contract, including how options compose and override.

Per-outcome policy is prior art rather than something new here: Guardrails AI has shipped it for years as each validator's on_fail setting, selecting among fix, filter, refrain, reask, noop and exception. What differs is the axis it keys on - plumb splits by why the check did not resolve (Unresolved = no evidence of intent, Ambiguous = evidence that does not settle on one value), rather than by a single pass/fail axis.

Writing checks

Anything the primitives don't cover is a plain Check struct literal - inspect the output and the prompt, return findings, optionally repair. The worked example has three, including one that repairs in place. Hand-roll a fidelity check (switching on your extractor's Resolution yourself, the way Fidelity does) only when the shape differs: when the repair spans more than one field, or the value is not a straight comparison.

Two flavors exist, and both are just Checks:

• Fidelity checks compare output to prompt - the differentiator. The recommended recipe for any code/enum/identifier field is extract-then-map: extract the intent signal deterministically from the prompt, map it to the canonical value through an owned curated table, compare, and repair on unambiguous mismatch.
• Invariant checks assert well-formedness beyond schema using data captured via closure - a referenced identifier exists in the catalog, a numeric field is within budget, a mutation is bounded. They don't need the prompt. sql.table_exists in sqlplan is one, end to end.

The rules layer: pattern / rule / data

The correctness policy for a type lives in one place, composed from library primitives:

Layer	Lives in	Example	Changing it costs
Pattern (generic primitive)	the library	Fidelity, Check[T]	a library release
Rule (instantiated primitive)	your rules.go	Fidelity("region", loc, regions.Match)	a host code change
Data (what rules compare against)	typed parameters of your rule constructors	region vocabulary, catalogs, budgets	no recompile

Primitives reach into your type through a typed accessor closure - no reflection, no struct tags, compile-checked:

func Fidelity[T any, V comparable](field string, loc func(*T) *V,
    intended func(prompt string) (V, Resolution), opts ...FidelityOption) Check[T]

V is any comparable type, not just string, so one primitive covers region codes, row limits, tiers and enums. It compares and repairs the field in its own type and renders the recorded Was/Now itself, so a numeric field needs no string round-trip in your rule set (examples/sqlplan uses it on an int row limit). opts is the per-field outcome policy from Choosing the outcome policy. The Rules constructor in step 3 of the worked example is a host's whole policy at this layer: configuration-shaped, still type-safe and unit-testable.

Why not a DSL / rules-as-config? The flagship rules are inherently code (extraction + lookup); an interpreter (CEL/Rego) breaks the stdlib-only, hot-path constraint; and you'd lose the compile-time guarantee that a rule references a field that exists. Externalize the data, not the logic. The primitive catalog stays small for the same reason: a new primitive earns a name only after the same shape has been hand-written across enough real rule sets.

Fidelity-fault shapes

Fidelity covers exactly one shape: a field whose correct value is a closed-set code recoverable from the prompt. Everything else is a Check struct literal, and "write a check" is not much of an answer on its own. Find your shape below and read the named check - every row points at a real rule, not a sketch.

Shape - and the fault it catches	Pattern	Severity	Real check
Closed-set code named in the request. The field holds a valid code, not the one the request named.	Fidelity plus a deterministic extractor	Correct; Flag when the request does not resolve to one value	worked example step 3; fidelity.limit over an int in sqlplan
Membership in a live catalog. The field names an identifier that does not exist.	Check literal over a captured catalog	Block	sql.table_exists in sqlplan
Absence / bounded mutation. A qualifier is missing, and its absence is what makes the action dangerous.	Check literal, with the request the only license to proceed without it	Block	sql.bounded_mutation in sqlplan
Drift from a host default. The request was silent and the model chose for you.	Check literal keyed on Unresolved	Flag - a human decides	tier.drift, worked example step 3
Policy allowlist. The field holds a value policy does not permit.	Check literal over an allowlist; a nil map reads false, so an unconfigured policy blocks everything	Block	tier.allowed, worked example step 3
Numeric clamp. The field exceeds a host ceiling.	Check literal repairing in place	Correct	storage.max, worked example step 3

The severity column follows from whether a deterministic repair exists: a code the request named can be looked up, so that shape corrects; a hallucinated table name cannot be guessed at, so that shape blocks. Only one of the six uses a shipped primitive - the library carries the pattern, you carry the policy.

The drift row is the subtle one, and where two checks can collide over one field. Its guard keys on Unresolved specifically - the request said nothing

• rather than on "the extractor returned no value", which is also true under Ambiguous. Get that wrong and a request naming two conflicting values produces two findings for one cause. Any check firing on the absence of a signal needs the same care.

Testing your rules

The rules are yours, so proving them right is yours too - and the failure worth defending against is a check firing when it should not. A hand-rolled table test that indexes into the findings asserts the ones it names and says nothing about the rest, so a rule that fires spuriously passes it.

plumbtest is a case runner where a case declares the complete outcome. Here it is over the worked example's rule set, with rules being what newRules returned in step 4:

import (
    "testing"

    "github.com/srinivasgumdelli/plumb/plumbtest"
)

var cases = []plumbtest.Case[DeploySpec]{{
    Name:   "wrong region repaired toward the location named",
    Prompt: "a premium Postgres DB in Mumbai",
    In:     DeploySpec{Region: "us-east-1", Tier: "premium", StorageGB: 512},
    Out:    DeploySpec{Region: "ap-south-1", Tier: "premium", StorageGB: 512},
    Findings: []plumbtest.Finding{
        plumbtest.Corrected("fidelity.region", "region", "us-east-1", "ap-south-1"),
    },
}}

func TestRules(t *testing.T) {
    plumbtest.Run(t, cases, rules...).AssertAllFired(t)
}

Findings is the whole finding set, not a subset: any check emitting something this case did not declare fails it, and there is no option to relax that. Out is the whole repaired output, so a Correct finding that records a flawless repair and writes the wrong value is caught even though the finding itself looks right. Every case also re-evaluates the repaired output and requires the second pass to change nothing - the Idempotent repair invariant asserted rather than assumed.

Run returns a *Coverage recording which checks fired across the whole corpus, and AssertAllFired is the half no individual case can supply. A check that never fires anywhere reads as coverage and is actually dead policy: every case passes, and nothing has ever proved the rule does anything. Failing on it is opt-in, because some checks legitimately fire rarely:

func TestRulesCoverage(t *testing.T) {
    cov := plumbtest.Run(t, cases, rules...)
    cov.AssertAllFired(t, "tier.drift") // no case here is silent about tier
}

The exemption list is held to the same standard as the rules: exempting a check that did fire is itself a failure, so a suppression nobody revisits cannot sit there masking a later regression, and exempting a name that is not a check in the corpus fails too.

Cases that vary the rule set's data

Checks get their data by closure capture, so a corpus that varies the data varies the rule set with it. A case can carry its own via Checks rather than being exiled to a separate table, and coverage still spans every rule set the corpus used:

var catalogCases = []plumbtest.Case[SQLPlan]{{
    Name:   "unconfigured catalog blocks every referenced table",
    Prompt: "list users",
    In:     SQLPlan{Query: "SELECT * FROM users", Tables: []string{"users"}},
    Out:    SQLPlan{Query: "SELECT * FROM users", Tables: []string{"users"}},
    Findings: []plumbtest.Finding{
        plumbtest.Blocked("sql.table_exists", "tables[users]"),
    },
    Checks: Rules(nil), // same policy, different data
}}

Asserting on a finding's message

Check, Severity, Path, Was and Now are compared exactly. Message is prose, so pinning it turns every wording change into a failure; assert the part you care about with MessageContains or MessageOmits. For an ambiguity Flag the contract is what it must not say, since naming a candidate would leak the guess the check refused to make:

var ambiguityFlag = plumbtest.Finding{
    Check:        "fidelity.limit",
    Severity:     plumb.Flag,
    Path:         "limit",
    MessageOmits: []string{"5", "10"},
}

Package tour

plumb                 core: Severity, Resolution, Finding, Check, Report,
                      Evaluate, Corrected/Flagged/Blocked, Fidelity and its
                      FidelityOption policy set, Retry. Stdlib only.
├── gazetteer         optional: Table, NewTable, Match - compiles YOUR terms
│                     into a deterministic, fail-safe extractor. Ships no
│                     vocabulary of its own.
├── otel              optional: SDK-free helpers rendering findings as
│                     telemetry attributes.
├── plumbtest         test support: Case, Finding, Run, Coverage - declarative,
│                     exhaustive-by-default tests for your own rule sets.
└── examples          deploy, sqlplan, vocabulary (see Reference examples)

Satellites depend on the core; the core depends on nothing beyond the standard library - no network, no model SDK, no config reading.

gazetteer

Three symbols: Table, NewTable, and (Table).Match. You bring the terms; NewTable compiles them into a fail-safe extractor of exactly the shape Fidelity consumes. There is no shipped vocabulary, because fidelity is domain-specific and a term list nobody in your domain reviewed is how a confidently wrong code gets into production. Step 2 of the worked example builds one; this is what it then does:

region, res := regions.Match("a Postgres staging DB in Mumbai")
// region == "ap-south-1", res == plumb.Resolved

_, res = regions.Match("a DB in West Virginia")
// res == plumb.Unresolved, code "" - blocked, so "virginia" never fires

_, res = regions.Match("replicate the Mumbai DB to Frankfurt")
// res == plumb.Ambiguous, code "" - two codes, and Match never picks

Contract: a non-empty code accompanies plumb.Resolved and nothing else. Conflicting codes are the only route to Ambiguous here: NewTable validates every term at construction, so a Table has no match it can read and then fail to parse. The Resolution itself is wider - it means "named something that does not resolve to exactly one value" - which is how an extractor that parses numbers out of free text reports a count it matched and could not read. Fidelity treats both the same way. Matching is case-insensitive, word-boundary aware ("Oregonian" never matches a term "oregon"), and longest-match-wins. NewTable normalizes every term the same way it normalizes input text, so "us-east-1", "Asia/Kolkata" and "São Paulo" are all legal terms and none can end up silently unmatchable. A Table is immutable once built, so concurrent Match calls are race-free.

otel

Renders findings as flat key/value attributes ready for any tracing SDK (deliberately SDK-free - convert each Attr with your library's constructor):

plumb.finding.check     = "fidelity.region"
plumb.finding.severity  = "correct"
plumb.finding.path      = "region"
plumb.finding.was / .now  (Correct findings only)

SeverityString is exhaustive: values outside the three severities render as invalid(<n>) so malformed findings surface instead of vanishing.

Retry

report := plumb.Retry(ctx, &plan, prompt, rules,
    func(ctx context.Context, out SQLPlan, blocks []plumb.Finding) (SQLPlan, error) {
        // Your bounded model retry: turn blocks' Path/Message into a
        // repair prompt, return a new candidate (or error to stop).
        return askModelAgain(ctx, out, blocks)
    }, 2)

Re-asking is the middle rung of the escalation ladder - repair in code first, fail closed last - and it is opt-in so the non-determinism stays under the caller's own budget. context is stdlib, so the core is still a pure leaf with Retry in it.

Re-ask fires only while the report is Blocked, and every failure path - a reask error, a canceled context, exhausted rounds - returns the latest report still carrying its Blocks. maxRounds <= 0 behaves exactly like Evaluate.

The one hazard worth knowing before you write a reask: it receives out by value, and the copy is shallow. T's slice, map and pointer fields alias into the caller's output, so writing through one mutates the caller's value immediately, even on a round that then errors. Build the candidate fresh and treat both out and blocks as read-only.

Where Plumb sits

Run it once, immediately after parsing into T, before the output is persisted or acted on - a single chokepoint:

generate → parse → **Plumb** → persist → plan → (human approve) → act

It composes with the neighboring categories: constrained generation runs before it, safety guardrails run in parallel on the harm axis, your check corpus doubles as an offline eval regression set, and findings stream into observability as structured per-fault signal.

Tool / category	Axis	Sees the request?	Repairs in code?	Runtime gate?
JSON Schema / Outlines	type/format	no	n/a	at generation
Instructor (+ Pydantic validators)	type/format, plus whatever you validate	yes, via validation context	yes, if you write it	yes
Guardrails AI / NeMo / LLM Guard	safety/harm	via metadata, if plumbed	yes (on_fail="fix")	yes
RAGAS / grounding	faithful-to-context	context, not request	no	usually offline
promptfoo / DeepEval	quality (offline)	in test cases	no	no (CI)
Langfuse / Phoenix / OpenLLMetry	observability	records	no	no
Plumb	fidelity + invariants	yes, by signature	yes, by default	yes

Read that table as a description of defaults, not capabilities. Two rows are close enough that the difference is ergonomic rather than architectural, and the next section says so plainly.

Relationship to evals

Evals and Plumb checks are the same kind of assertion run at different times. An eval assertion runs offline, over a fixed test set, and produces a score. A check runs inline, on the request in front of it, and produces an action.

The useful property is that the second subsumes the first: a check written for the runtime gate can also be run over a corpus as a regression test, but an eval assertion written against a fixture cannot be moved into the request path, because it depends on a reference answer that production does not have. One artifact, two jobs, and the substitution works in one direction only.

This is the one comparison in this section where Plumb does something a neighbor structurally cannot, rather than doing something it merely does by default.

Prior art and honest comparison

Two tools in the table above can be pushed into this lane, and it would be dishonest to imply otherwise.

Instructor plus Pydantic can do fidelity checks today. Pydantic validators receive a validation context and Instructor passes one through, so a validator can see the prompt, compare a field against it, and either repair in place or raise to trigger a bounded re-ask:

@model_validator(mode="after")
def region_matches_intent(self, info: ValidationInfo):
    want, ok = region_from_text(info.context["prompt"])
    if ok and self.region != want:
        self.region = want          # repair, or raise to re-ask
    return self

That is this library's flagship check, in Python, with no new dependency.

Guardrails AI already treats repair as first-class. Its validators carry an on_fail policy with fix, filter, refrain, reask, noop, and exception, which is a severity-to-action mapping with deterministic repair among the options, plus a hub of community validators.

So the accurate claim is not that this axis is uncovered. It is:

• In Python, the pieces exist and you assemble them. The result works. What you don't get for free is the taxonomy: a three-way severity split where Flag (proceed, but to a human) is distinct from Block (never act), a report you inspect and route on rather than an exception you catch, a standing rule that every repair is recorded with before/after values, and a two-valued extractor's blind spot - the snippet above cannot tell "no location named" from "two named, and they disagree", so the second case passes silently.
• In Go, no equivalent exists. go-playground/validator is struct tags with no prompt access and no repair. That gap is real, and it is also about 160 lines wide, which is roughly the size of this library's core. If you would rather write those 160 lines in your own repo, the design here is worth reading and the dependency is not compulsory.

What is genuinely load-bearing, and would need reimplementing either way: Block as Severity's zero value, so a malformed finding fails closed; the Correct/Flag/Block split mapping to three distinct caller behaviors; the requirement that every repair emits a finding with Was/Now; the three-way Resolution that makes an extractor's abstention visible instead of indistinguishable from a pass; and containing a panicking check as a Block rather than letting it escape. The for-loop is not the contribution. The discipline is.

And the rules are still yours. Nobody can ship you "region matches intent". Fidelity is domain-specific by construction, which is why no generic gateway will do this for you, and why gazetteer ships the matcher and no vocabulary: the terms are yours, from your own authoritative data.

Determinism guarantees

Evaluate runs checks in argument order, single-threaded, applying repairs in place as it goes - same input, same report, same repaired output, every time. Repairs are visible to later checks (usually desirable: a later invariant validates the repaired value). Findings returned without a check name are stamped with the originating check's Name.

Panics in checks are contained, never propagated. Checks are caller-written code, and a library whose job is to be a safety net must not become the crash surface of your request handler. If a check's Inspect panics - including a nil Inspect field - Evaluate recovers, converts the panic into a single Block finding naming that check, and carries on with every later check. That fails closed, because a check that could not complete cannot vouch for the output, and it degrades the report by exactly one check instead of destroying it. Repairs the panicking check already applied are not rolled back; the Block is your signal not to act on that half-repaired value. Retry inherits this, reaching checks only through Evaluate.

Reference examples

All three carry runnable Examples in godoc.

• examples/deploy - the worked example's shapes as a compiling, tested package behind a typed Platform parameter, with the drift check keyed on region rather than tier, which is where the two Flags could collide over one field.
• examples/sqlplan - text-to-SQL: catalog-membership Block, bounded-mutation Block, a Fidelity[SQLPlan, int] row-limit repair, and the Retry round-trip end to end.
• examples/vocabulary - three larger extraction vocabularies, split by provenance into the terms the vendor publishes and the aliases a human curated, with the blocking terms and the audits that keep a growing term list from acquiring a confidently wrong answer. The terms are illustrative; the audits are the part to copy.

Status

All four planned stages are implemented and merged: v0.1 core, v0.2 gazetteer + otel, v0.3 the bounded re-ask driver (now core Retry), v0.4 the text-to-SQL reference integration.

Gates. Every push and pull request runs gofmt, go vet, go build, golangci-lint against a pinned version and an explicit linter list, and the full test suite under the race detector behind a statement-coverage threshold - on both the module's declared floor (Go 1.21) and current stable. That is CI discipline, not evidence of field use; see below.

What has not changed is how much production evidence sits behind any of it: none. The library has zero real-world integrations. Everything here is exercised by this repo's tests and reference examples and by nothing else, so the design's weak spots are the ones a real integration would find and this repo cannot. Treat the gates above as a floor on obvious defects, not as a claim of maturity.

The API is pre-v1, unversioned, and still moving - the most recent round renamed Report.Blocked to Report.IsBlocked and removed gazetteer's shipped vocabularies, leaving the matcher and moving the terms to examples/vocabulary. Expect more of that until there is a v1 tag.

Why "Plumb"

A plumb line is a weight on a string. It does two jobs at once: it shows you what is out of true, and it is the reference you straighten against. A builder doesn't consult it and then guess - they move the wall until it matches the line.

That is the whole library. The prompt is the line, the structured output is the wall, and where they disagree the fix is to move the output back to the line, deterministically, in code. "Out of plumb" already means what a fidelity fault is: not broken, not unsafe, not malformed, just not true to the reference it was built against.