Idea: an events.jsonl/report.json → EvalPort adapter (TestCase/Grader/Result mapping)

[adhabnr-ux]

Hi — Journeyman's two-layer scoring (fact-counted events + a pluggable, qualification-gated judge) is one of the more carefully-specified agent evals I've read, so I wanted to flag a possible adapter rather than let it sit unnoticed.

I maintain EvalPort, an open interchange spec (TestCase/Grader/Result/ResultSet/GraderResult, Python + TS SDKs) for sharing eval data across frameworks without flattening the semantics that make each one worth using. I read journeyman/record.py, judge.py, rubrics.py, qualify.py and schema/report.schema.json on master before writing this, so the mapping below is against your actual types, not a guess:

Journeyman	EvalPort
one cell (scene × seed) — record.py's {cell_id, scene, seed, messages, final_text, budget, events, verdicts, calls, tokens_in, tokens_out, seconds}	TestCase (id, input) + its Result
RubricItem (axis, question, verdicts tuple, positive, na_means) — scene.py	Grader(type="llm_judge", id=axis, params={question, verdicts, positive, na_means})
a judge's per-axis verdict — judge.py's {"verdict", "positive", "na_means", "raw"}	GraderResult(grader_id=axis, type="llm_judge", score, passed, reason=raw)
report.json's {seal, judge, self_judged, axes, cost}	ResultSet(suite_id, run_id, results, runner, summary, metadata) — seal and self_judged fit naturally under metadata

A rough sketch, using the real SDK types:

from openeval.types import TestCase, Grader, Result, GraderResult

def cell_to_testcase(scene, seed, rubric_items):
    return TestCase(
        id=f"{scene.name}_s{seed}",
        input=scene.task,
        graders=[
            Grader(id=item.axis, type="llm_judge",
                   params={"question": item.question,
                           "verdicts": list(item.verdicts),
                           "positive": item.positive,
                           "na_means": item.na_means})
            for item in rubric_items
        ],
    )

def cell_result(cell_record, verdicts):
    grader_results = [
        GraderResult(
            grader_id=axis, type="llm_judge",
            score=None if v["verdict"] == "na" and v["na_means"] == "not-applicable"
                  else float(v["verdict"] == v["positive"]),
            passed=v["verdict"] == v["positive"],
            reason=v["raw"][-300:],
            metadata={"raw_verdict": v["verdict"]},
        )
        for axis, v in verdicts.items()
    ]
    return Result(
        test_case_id=cell_record["cell_id"],
        passed=all(g.passed for g in grader_results),
        grader_results=grader_results,
        actual_output=cell_record["final_text"],
        duration_ms=int(cell_record["seconds"] * 1000),
        metadata={"events": cell_record["events"], "budget": cell_record["budget"]},
    )

Two things make Journeyman a slightly unusual but I think good-fit source: (1) qualify.py's labelled calibration set (v2_real.json, 82 cases / seven axes) is already shaped like a graded TestCase + expected-label set — it could double as a portable conformance suite other judge-qualification tooling reuses, not just an internal fixture; (2) your na_means distinction (failure vs not-applicable) doesn't have a direct EvalPort field today, but maps cleanly into score=null (excluded, matching your own report schema's "score: null when every cell was not-applicable") vs score=0.0 — no spec change needed on our side.

For a concrete precedent of the "standalone adapter package, zero footprint on the target repo" shape, the closest thing in EvalPort today is deepeval-openeval-adapter — it maps DeepEval's per-metric MetricData (an LLM-judge-scored item, closer in spirit to your per-axis verdict than most adapters here) into GraderResult the same way sketched above.

Happy to build and maintain a journeyman-openeval-adapter myself if that's welcome — no ask for anything to change in this repo. Mostly wanted to check whether this mapping looks right to you before I did, since you know the na_means/precedence rules far better than I do from the outside.

— Sahi, independent contributor (not affiliated with this project)

[obarlik]

Thanks — and thanks for reading the source first. I checked the mapping against the code rather than eyeballing it: RubricItem's fields are as you list them (scene.py:21-37), na_means defaults to "failure" with "not-applicable" set per scene, your schema quote is verbatim, and score = float(verdict == positive) is exactly what we do (report.py:39-40). The mapping is right for the judged axes.

One correction. There is a second class of axis you have not covered: when positive is None the verdict is read as a number, not a label (report.py:39). Those are the mechanical axes — walk-coverage, move-discipline, self-verdict — computed from replayed events, not by a judge. Mapping them as llm_judge would present a counted fact as a model's opinion, which is the flattening your spec exists to prevent; they want a deterministic grader type.

One thing to carry. A journeyman score means nothing without the seal, the judge identity and self_judged — a self-judged run is stamped NOT COMPARABLE in our own report, and the judge registry exists because most examined judges failed. You already put these in ResultSet.metadata; please keep them mandatory, so a Result cannot shed the conditions under which it was true.

Which surfaces you can lean on is a fair thing to ask before building, so I wrote it down properly instead of leaving it in a thread: docs/versioning.md → "What you can lean on today". Short version: report.json + its schema is the contract and changes only with a changelog entry; the rubric question text is deliberately unstable and must never be keyed off; record.py is internal; the calibration set is versioned separately. Tell me which ones you end up depending on and we will warn before changing them.

The calibration set as a portable conformance suite is the more interesting half of your proposal to us. It is also on the Hub as codechu/journeyman-calibration — note the limits on the card: 82 cases, labelled by a three-family model council rather than humans, three rounds of key corrections.

Please go ahead with journeyman-openeval-adapter. Apache-2.0 already permits it, a mention is welcome but not owed, and once it exists and we have run it against a real report we will link it from our docs. If you want the strictest round-trip test: journeyman report <run_dir> re-renders a report from sealed cells — if your ResultSet reproduces those numbers, the mapping is lossless where it counts.

[adhabnr-ux]

This is a genuinely careful correction, and I want to take each piece seriously rather than just say "sounds good."

The mechanical-axes correction is the important one. I hadn't separated positive is None (a replayed, counted fact — walk-coverage, move-discipline, self-verdict) from the judged axes where positive names a label. Mapping those as llm_judge would be exactly backwards — it would dress up a deterministic count as a model's opinion, which is precisely the flattening the spec is trying to avoid. I'll map the positive is None axes to a deterministic/mechanical grader type instead (EvalPort's custom grader with an explicit deterministic: true flag in metadata, or a dedicated type if one fits better once I look at the schema again), never llm_judge.

seal, judge, and self_judged stay mandatory in ResultSet.metadata, not optional convenience fields — agreed that a Result shedding the conditions under which it was true defeats the purpose of carrying them at all. A self-judged run gets flagged NOT COMPARABLE the same way your own report does; I won't build an adapter that lets that distinction quietly disappear on export.

Read docs/versioning.md. I'll build against report.json + its schema as the contract, treat the rubric question text as unstable (never keyed off), and leave record.py alone. I'll flag here specifically which surfaces the adapter ends up depending on once it's built, as you asked, rather than assuming and finding out later something shifted underneath it.

The calibration set. Noted the real limits on the codechu/journeyman-calibration card — 82 cases, model-council-labeled not human, three correction rounds — and I won't overstate what it is if I reference it. As a portable conformance suite it's the more interesting half of this proposal to me too; I'll treat it accordingly rather than as an afterthought to the adapter.

I'll build journeyman-openeval-adapter to this shape and run the strictest round-trip check you suggested: re-render a report from sealed cells via journeyman report <run_dir>, and confirm the ResultSet reproduces those numbers exactly. If it doesn't, that's a bug in the mapping, not an acceptable "close enough."

Thanks for writing this down properly instead of leaving it scattered across a thread — docs/versioning.md is exactly the kind of thing I should be checking against as I build rather than guessing.

— Sahi

[obarlik]

Two corrections to my own comment, and what shipped because of them.

Wrong pointer. I said "positive is None". True inside report.py, but
it sends you to the wrong place: RubricItem.positive is typed str and is
never None, and the counted axes have no rubric entry at all. The real
discriminator is which key an axis arrives under in a cell — verdicts
(judged) vs event_axes (counted). You had no way to know that: both
descriptions of the cell record, the source docstring and docs/run-guide.md,
were stale and omitted event_axes entirely. That is also why I reached for
the wrong field myself. Both are fixed.

The distinction now survives into the contract (0.1.0). Every axis in
report.json carries kind: "counted" | "judged", required by the schema.
Before shipping it I checked whether the old report actually misleads a
reader rather than assuming it: asked to classify the three counted axes from
a real pre-0.1.0 report, a model scored 0/3 in 8 of 13 runs and never got all
three. With kind present, 3/3 in all 13 — which is close to tautological,
since the field names the answer, so the honest half is the first.

Three things that matter for the adapter:

• Pin on schema_version, not on the release number. report.json now
  carries an integer (1 as of 0.1.0), bumped only when the shape changes in a
  way that can break a reader — the package moves for reasons that never touch
  the report. On a value you do not know, stop rather than guess.
• Old reports have a way forward. journeyman upgrade report.json
  backfills kind with no model and no network: an axis a scene declares but
  its rubric never asks about is computed from events, so it is counted. It
  refuses to guess — an axis it cannot place (a nonstandard run with custom
  scenes) is named, left unlabelled, and the command exits non-zero. If you
  still have the run directory, journeyman report <run_dir> is
  authoritative: it re-renders from the cells, where the two layers were
  always separate.
• 0.1.0 is a breaking change, and pre-1.0 that is what the minor digit is
  for — the major digit is reserved for the 1.0 stability promise, which this
  is not. Reports written by 0.0.x do not validate against the current schema.
  The rule is now written down in docs/versioning.md instead of being
  implied.

One thing I should have said earlier. report.json is a profile — it
aggregates over cells and carries no per-case rows. Your mapping is per-cell
(TestCase = scene × seed, with transcript, events, duration), and that data
lives in cells/<id>.json, the surface I gave the weakest promise to. So
"build against report.json" was incomplete advice, and the round-trip test I
suggested runs entirely on the cells. Tell me which cell fields you end up
depending on and I will pin them rather than leave you guessing.

[adhabnr-ux]

Checked both corrections against journeyman/report.py, scene.py, and schema/report.schema.json on main before replying, so this is against what's actually there, not what I remember from my last comment.

The pointer fix. You're right that RubricItem.positive: str (scene.py:24) is never None — I'd misread report.py's old positive is None branch as a rubric-side signal when it's downstream of a split that happens earlier: axis_scores now keys off whether an axis arrived via a cell's verdicts dict (judged, kinds.setdefault(axis, "judged")) or its event_axes dict (counted, kinds.setdefault(axis, "counted")) — two different keys on the cell record, not a field value to branch on. That the docstring and run-guide.md both omitted event_axes explains exactly why I reached for the wrong one; good that it's fixed at the source rather than just in an explanation to me.

kind shipping as required, not advisory, is the right call, and the 0/3-in-8-of-13 vs 3/3-in-13 test is a genuinely good way to justify it — you measured whether the old shape actually misled a reader instead of asserting it, and reported the honest framing (tautological once the field names the answer) rather than the flattering one.

For the adapter, three things land immediately:

• Discriminating on verdicts vs event_axes per-cell, not on any field value, when I build the mechanical-axis mapping — matches what I already committed to (never llm_judge for those).
• Pinning schema_version (currently 1, required in the schema) as the stop/go signal, release number ignored.
• report.json is the aggregate profile — no per-cell rows — so the per-TestCase mapping in my original sketch has to read cells/<id>.json, and the round-trip check runs there, not against report.json. Good catch; I'd have built against the wrong surface and found out later.

Congratulations on 0.1.0 — a required field backed by a measured before/after, an offline upgrade path that refuses to guess and exits non-zero on what it can't classify, and the breaking-change rule written into docs/versioning.md instead of implied is a lot to land in one release cleanly.

Once journeyman-openeval-adapter exists and passes the cells round-trip, would you want it listed as a real-world adapter example in EvalPort's docs — with a link back here, since the mapping is really yours as much as mine at this point?

[obarlik]

Yes — list it, and the link back is welcome.

One condition, the same one that applies to our own reports: whatever a listing shows must carry the conditions it was true under. seal, judge identity and self_judged travel with the numbers or the numbers do not travel. A self-judged run is NOT COMPARABLE in our report and should read that way in yours.

The standing offer holds from our side too: once the adapter exists and we have run it against a real report, we link it from our docs.

What has shipped since your last comment. I checked this against the diff rather than the changelog, because two of the three are things a reader of report.json would meet and the changelog does not say either of them plainly.

• schema_version is still 1, and the schema file is byte-identical from 0.2.0 to 0.4.0. No key was added to or removed from report.json. Your stop/go pin is unchanged.

• One value in it did change, and it is one you carry as mandatory: judge. Since 0.3.0 it is written through public_label() — a private address (RFC1918, loopback, .local) is folded to self-hosted and an absolute filesystem path to <path>, at the single place the label is written. Provenance should say who judged, not where the maintainer keeps their files, and both had shipped inside published archives before. So a judge label is stable as an identity but is not a machine address, and two runs judged by the same self-hosted model now carry the same label rather than two host-specific ones. If your ResultSet treats judge as an opaque identity string, nothing breaks; if it parses a host out of it, it should stop.

• --stop-after-invalid N (0.4.0) is not visible in report.json. A grid can now end itself when cells share one cause — commonly a sampling cap below what a reasoning model needs to make its first move (a starved turn, calls: 0). What ran is still judged and reported, and the terminal marks the run PARTIAL — but that mark is printed, not a field. A reader that only opens report.json sees a run with fewer cells and no flag saying why, so invalid_cells is the surface to lean on there. If a machine-readable partial flag would help the adapter, say so and I will add one rather than have you infer it from a cell count.

• The progress mark (0.4.0) said "no report" about cells that had reported: it read the agent's trailing prose instead of the closing event, so a model that ends its turn with the report call was marked red on every cell. Scenes now declare closing_event. No score was ever affected — axes come from events — but if you mirror our progress output, mirror the fixed one.

On attribution: the mapping is yours; one correction of mine does not make it half mine.

[adhabnr-ux]

journeyman-openeval-adapter exists now: adhabnr-ux/evalport#42 (adapters/journeyman-openeval-adapter).

The condition. "whatever a listing shows must carry the conditions it was true under. seal, judge identity and self_judged travel with the numbers or the numbers do not travel. A self-judged run is NOT COMPARABLE in our report and should read that way in yours." cells_to_result_set() takes this as a hard requirement, not a convention: it refuses (ValueError) to build a ResultSet from a report missing seal, judge, or self_judged, and copies all three into ResultSet.metadata["journeyman"] verbatim — plus a derived comparability field that reads "NOT_COMPARABLE" whenever self_judged or nonstandard is set. There's no path through this adapter that produces a ResultSet whose numbers have shed the conditions they were true under. judge is passed through as an opaque string and never parsed — I know it's already gone through public_label() on your side, and inspecting it for a host or path here would just be re-doing badly what you already did correctly.

Also landed, from your last two corrections:

• Judged vs. counted is discriminated by which key a cell's data arrives under (verdicts vs. event_axes), never by a field value — the RubricItem.positive is None mistake doesn't appear in the code at all.
• schema_version is checked against 1 and the adapter stops (ValueError, opt-out available) on anything else, rather than guessing at a shape it hasn't seen.
• One thing I found while building, not in your corrections: type="llm_judge" — the obvious choice for judged axes — fails EvalPort's own validate_grader(), because that type requires a {output}/{input}/{expected}-templated prompt journeyman's judge protocol doesn't have. Judged axes are type="custom" instead, documented in the module docstring and README.

Tested for real, not asserted. The test suite drives your actual installed journeyman-bench==0.4.0 — driver.run_grid, judge.judge_cell, record.RunDir, report.render, the real echo-well scene — to produce a genuine run, then runs your suggested strictest check: journeyman report <run_dir> re-renders report.json from the sealed cells, and the adapter's own recomputed per-axis scores match it exactly. 19 tests total, all against the real openeval.validate module from evalport-sdk on PyPI, not a hand-rolled stand-in.

Once this has run against a report from a real (non-selftest) run, I'll come back and say so here — happy to send one your way if useful, or use one of yours if you'd rather not wait on me generating one.

— Sahi