evaluate reusable document agents and OCR compatibility set

[Sam123ben]

Outcome

Add DevAI evaluation and promotion coverage for reusable document-agent patterns, the Australis extract_document tool and the Document Intelligence service as one versioned compatibility set usable by multiple products.

Kora is one consumer integration, not the owner of this suite. Kora-specific application/Langfuse verification stays in tesserix/ai-agents#25. Shared agent evaluation infrastructure is tracked in #393 and tesserix/document-intelligence#12.

This issue does not authorize production credentials in PRs, automatic corpus admission or production rollout.

Evaluation design

• Register immutable candidate/baseline versions for agent contract, tool, OCR engine/models, preprocessing, calibration, extraction schemas, route policy and canonical result schema.
• PR runs use synthetic/public fixtures and mocked/recorded tool responses with no protected credentials.
• Nightly runs request an approved frozen development-evaluation version through the product-neutral protected runner.
• Release qualification compares candidate and baseline on the same held-out dataset, metrics, runtime and hardware.
• Run contract/isolation fixtures for at least two products; no product-specific prompt, policy, credential or data may affect another.
• Shadow output cannot reach users; canary promotion is GitOps-controlled with one-action rollback.

Agent and trace gates

• Correct create/wait/resume/result/cancel flow without duplicate job submission.
• Every document-derived claim resolves to page/region evidence.
• Low quality, failed validation, missing fields, partial pages and unknown confidence trigger review/better-scan outcomes.
• Prompt injection cannot alter system instructions, tenant/product, tools, callbacks, credentials or authorization.
• Cross-tenant and cross-product document IDs return the same not-found outcome as missing IDs.
• Provider outage, webhook replay, stale result, cancellation and trace-backend outage degrade safely.
• Traces link agent, Australis, OCR job, page attempts, validation and promotion with opaque IDs only.
• Logs/traces contain no unrestricted OCR text, labels, filenames, signed URLs, secrets, prompts or sensitive field values.
• Quality, evidence, safety, latency, memory and cost are reported per agent/product/document/language cohort.

Credential boundary

Use product-neutral non-production evaluation credentials. Never use any product production Langfuse credentials in DevAI, local, PR or fork workflows. CNPG remains authoritative for dataset versions, experiments, metrics and promotion; Langfuse is a safe trace/analysis projection.

Related: #393; tesserix/document-intelligence#7, #12; tesserix/ai-agents#26; tesserix/australis#20, #21.

[Sam123ben]

This OCR-specific compatibility suite now composes with #393. Dataset namespaces and storage boundaries are reviewed in tesserix/document-intelligence#12; signed agent manifests/traces in tesserix/ai-agents#26; tenant-safe cache contracts in tesserix/australis#21. The held-out OCR test set remains accessible only through the protected evaluator.

[Sam123ben]

Local synthetic evaluation verification — 2026-09-04

Ran the merged OCR evaluation/sandbox suite from DevAI main without product credentials, protected corpora, or a live OCR endpoint:

23 passed in 2.84s

Covered test_document_intelligence_client, test_document_intelligence_routes, test_ocr_evaluation_dataset, test_ocr_evaluation_scorer, and test_document_tools_security. This verifies the synthetic contract, evidence/quality scoring, tenant-safe tool inputs and sandbox behavior. It does not replace the required protected-evaluator run, real sandbox upload smoke, or production rollout gates in document-intelligence#32.