design reusable agent evaluation and improvement flywheel

[Sam123ben]

Outcome

Design the reusable DevAI quality platform that turns reviewed production signals into versioned per-agent datasets, evaluates candidate agent compatibility manifests, and promotes only candidates that pass deterministic, model-judged, safety, latency and cost gates.

This issue is design/repository work only. It does not authorize ingesting production traces into training, copying sensitive datasets, exposing production credentials to CI, model training, or production rollout.

Dataset taxonomy

Use an agent-scoped hierarchy, not one universal corpus:

• customer-support-agent: normal queries, tool selection, policy compliance and adversarial cases.
• ocr-agent: invoices, receipts, handwriting, poor-quality scans, multilingual/layout and prompt-injection cases.
• sre-agent: Kubernetes analysis, incident reasoning, safe remediation and permission boundaries.

Each immutable case contains input, expected outcome constraints, required tools, forbidden actions/claims, evaluation rules, required facts/citations/schema, safety boundaries, tags, provenance, consent/licence, classification, split and version. Exact prose is optional when rule-based outcomes are stronger.

Trace and feedback boundary

Define a versioned redacted trace envelope covering agent/prompt/model/tool/retrieval/policy/memory versions, model parameters, user request classification, retrieved-context references, tool calls/results, final-response reference, tokens, latency, cost, retries/errors, user feedback and automated scores. Raw sensitive request/context/tool/response values are not ordinary trace attributes.

Flow: capture -> classify/redact -> select success/failure/unusual cohorts -> human or governed automatic label -> approve into a new immutable dataset version -> candidate comparison -> protected release gate -> shadow -> canary -> promote/rollback. Never train automatically from every trace.

Evaluation levels

• Deterministic: schema, required fields/facts/tools/citations, forbidden claims/actions, permissions, code/SQL/math validation, OCR thresholds and hallucinated-source detection.
• Model judge: correctness, relevance, completeness, clarity, groundedness, tool selection and policy compliance using structured scores; calibrate against human-reviewed examples and never use as the sole gate.
• Production: lightweight policy/schema/error/latency/tool checks on 100%; deeper quality/groundedness on a budgeted sample; detailed evaluation on 100% of failures; stratified human review sample.

Versioned release manifest

Pin agent, system prompt, tool schemas, model/provider/parameters, retrieval/index, knowledge, safety policy, memory strategy, token limits, evaluation dataset and metric versions. Release sandbox -> offline evaluation -> shadow -> canary -> production with one-action rollback.

Initial proposed gates requiring baseline/owner approval: zero critical safety regression; required-tool accuracy >=98%; structured-output validity >=99%; agreed cohort groundedness floor; p95 within SLO; cost increase within approved percentage; no material language, tenant or document-class regression.

Storage and execution

• GCS: immutable agent datasets, evaluation artefacts and model bundles using the boundaries in design review: shared sandbox, evaluation and training boundaries document-intelligence#12.
• Global CNPG: agent/dataset/prompt/release manifests, lineage, metric definitions, experiments, approvals and audit/outbox state.
• Langfuse: safe trace exploration, prompt/experiment visibility and scores; not canonical raw dataset storage.
• Qdrant: rebuildable retrieval and optional bounded semantic-cache candidate index.
• Valkey: TTL caches, counters and coordination only.
• Existing event/outbox and telemetry platform first. Kafka or ClickHouse require measured throughput/retention/query evidence and a separate ADR; they are not launch defaults.

Acceptance

• Dataset schema and per-agent namespace/version contract reviewed.
• Trace allowlist/redaction, sampling and retention reviewed with privacy/security.
• Human review, poisoning/leakage, consent/licence and dataset-admission controls tested.
• Deterministic and calibrated judge suites produce structured reproducible results.
• Candidate and baseline run against the identical frozen dataset/metrics/runtime.
• Agent compatibility manifest, signing, promotion and rollback proven.
• PR/fork workflows have no protected corpus, model registry or production secret access.
• Production sampling budgets and regional residency are enforceable per tenant/product.
• Cache correctness is version-bound and authorization is rechecked on every hit.
• Dashboards cover cohort quality, safety, latency, cost, cache efficiency, drift and review rate.

Related: tesserix/document-intelligence#7, #11, #12; #392; tesserix/australis#20; tesserix/ai-agents#25.

[Sam123ben]

Owning integrations are now explicit: tesserix/ai-agents#26 defines signed agent manifests, prompt-cache-safe assembly and trace payload references; tesserix/australis#21 defines tenant-safe semantic/tool-result caching; tesserix/document-intelligence#12 defines sandbox/train/development-eval/sealed-test storage and identity boundaries. OCR-specific execution remains in #392.