Is the returned ExactMatch Score object the right minimal external-consumer seam?

[Rul1an]

Hi AutoEvals folks,

We just added a small Assay-side sample around ExactMatch, and I wanted to share the narrow version rather than open with a broader integration ask.

The sample is here:
https://github.com/Rul1an/assay/tree/main/examples/autoevals-exactmatch-evidence

We kept it intentionally small. The probe runs ExactMatch on autoevals==0.2.0, stores the compared output / expected values separately as discovery context, and then reduces only the returned Score object. In the Python path we observed the useful public shape as name, score, empty metadata, and error=None.

For the Assay fixture, the canonical artifact keeps only the scorer name from name, the integer 0 / 1 score from score, and a target_kind that says this is an output-vs-expected comparison level rather than a stable target id. We deliberately leave raw compared values, metadata, error state, scorer config, Braintrust run/experiment wrappers, and broader AutoEvals scorer-family semantics out of the artifact.

The question is: does the returned Score object seem like the right minimal public surface for an external evidence consumer, or would you rather point consumers at a different returned/result boundary? If there is a better Python/TypeScript-stable seam, happy to tighten the sample around that.

Thanks for maintaining AutoEvals. ExactMatch is exactly the kind of small deterministic scorer that makes this boundary easy to reason about.

[vaibhavhariram]

The Score object's public fields (name, score, metadata, error) are part of AutoEvals' stable public API and safe to depend on for external integrations. You can rely on these fields remaining consistent across releases.

The structure is designed as the contract between evaluators and consumers: scorers return a Score with a numeric value (0–1), a descriptive name, optional metadata with details specific to each scorer, and error information if evaluation failed.

Your approach of extracting the fields you need (name, score, metadata) is exactly right—it's a minimal, robust interface for evidence consumers. We won't break this shape in future versions.

[Rul1an]

Thanks, this is exactly the confirmation we were hoping for.

We’ll keep the Assay sample anchored to the returned Score object and treat name, score, metadata, and error as the public scorer-result seam, while still reducing only the bounded fields we actually need for evidence receipts.

On our side that means we’ll keep raw compared values, scorer config, Braintrust run/experiment wrappers, and broader scorer-family semantics out of the v1 artifact.

Really appreciate the clear contract guidance. This gives us a much cleaner boundary to build around.