Proposal: an evalport-openeval-adapter for LexBench-Browser tasks/results

[adhabnr-ux]

Proposal: an evalport-openeval-adapter for LexBench-Browser tasks/results (portable interchange, not a replacement for anything here)

Hi — I maintain EvalPort, an Apache-2.0 open interchange spec (TestCase / Grader / Result / ResultSet) for portable LLM eval data, with a Python/TS SDK and 37 framework adapters merged so far (DeepEval, Ragas, MLflow, LangSmith, Braintrust, plus a FinanceBench adapter for a static benchmark dataset — same shape this would be). Not affiliated with this project; just read the actual schema/code before writing this.

Why I think this is a genuinely close fit, not a generic pitch:

I pulled the real field names from browseruse_bench/data/LexBench-Browser/task.jsonl, data_info.json, browseruse_bench/schemas/agent_result.py, eval_result.py, and EVALUATION_PROTOCOL.md — not the docs. A LexBench-Browser task already looks almost like an EvalPort TestCase with a rubric-based llm_judge grader attached:

LexBench-Browser (task.jsonl)	EvalPort TestCase	Notes
id (int)	id (str(id))	schema requires string
query	input	direct
target_website, domain, difficulty, task_type, reasoning_type, language, robustness_tags, login_required, risk_control	tags + metadata["lexbench"]	no schema field covers these individually — preserved, not dropped
reference_answer.steps, .key_points, .common_mistakes	context	reference trajectory / rubric hints, not a literal expected string
—	expected_output	intentionally left unset — there's no single gold string, only a scored rubric
reference_answer.scoring (total, items[].name/score/description) + score_threshold	one llm_judge Grader	rubric items get folded into the judge prompt; score_threshold/scoring.total becomes the pass threshold

That last row is the real design decision, and it mirrors what your own EVALUATION_PROTOCOL.md already specifies: official runs use gpt-5.4 with a stepwise judge strategy. So the adapter wouldn't invent a grading approach — it would encode the one this repo already runs, as one llm_judge grader per task with the scoring.items rubric embedded in params.prompt and params.threshold = score_threshold / scoring.total.

On the results side, AgentResult + EvalResult (from browseruse_bench/schemas/) map onto EvalPort's ResultSet.results[] almost field-for-field:

Source	EvalPort Result
AgentResult.task_id	test_case_id
AgentResult.answer	actual_output
AgentResult.metrics.end_to_end_ms	duration_ms
AgentResult.metrics.steps, .usage, agent_metadata, action_history	metadata
EvalResult.predicted_label == 1	passed
EvalResult.evaluation_details.score / scoring.total	grader_results[0].score (clamped [0,1] — EvalPort requires it, your score field doesn't cap it)
EvalResult.evaluation_details.reasoning	grader_results[0].reason
EvalResult.failure_classification	grader_results[0].metadata["lexbench_failure_classification"]
AgentResult.error / agent_done != done	Result.error ({"type": "runner_error", ...})

I'd write this the same way financebench-openeval-adapter handles a static benchmark-plus-results shape (closest precedent — read that README, it documents exactly this kind of "dataset, not a live SDK" mapping honestly, including what does not round-trip), and the same package layout deepeval-openeval-adapter uses for the trajectory/tool-call side (tools_called/expected_tools as name-only arrays is a real schema constraint on both sides, not a guess).

Rough sketch, using the real field names above (untested pseudocode, not a claim this is finished):

# lexbench-openeval-adapter/to_openeval.py
def task_to_testcase(task: dict) -> dict:
    scoring = task["reference_answer"]["scoring"]
    rubric = "\n".join(f"- {i['name']} ({i['score']}/{scoring['total']}): {i['description']}"
                        for i in scoring["items"])
    return {
        "id": str(task["id"]),
        "input": task["query"],
        "context": task["reference_answer"]["steps"] + task["reference_answer"]["key_points"],
        "tags": [task["domain"], task["difficulty"], task["task_type"], task["language"],
                  *task["robustness_tags"]],
        "metadata": {"lexbench": {k: task[k] for k in
                      ("target_website", "login_required", "risk_control", "reasoning_type")}},
        "graders": [{
            "id": f"lexbench_judge_{task['id']}",
            "type": "llm_judge",
            "params": {
                "model": "gpt-5.4",  # per EVALUATION_PROTOCOL.md's official judge
                "prompt": f"Score the agent's trajectory against this rubric:\n{rubric}\n"
                          f"Common mistakes to penalize: {task['reference_answer']['common_mistakes']}",
                "threshold": task["score_threshold"] / scoring["total"],
            },
        }],
    }

# lexbench-openeval-adapter/results_to_openeval.py
def agent_eval_to_result(agent_result: dict, eval_result: dict) -> dict:
    scoring_total = eval_result["evaluation_details"]["benchmark_details"].get("scoring_total", 100)
    raw_score = eval_result["evaluation_details"]["score"]
    return {
        "test_case_id": agent_result["task_id"],
        "actual_output": agent_result["answer"],
        "duration_ms": agent_result["metrics"]["end_to_end_ms"],
        "passed": eval_result["predicted_label"] == 1,
        "grader_results": [{
            "grader_id": f"lexbench_judge_{agent_result['task_id']}",
            "type": "llm_judge",
            "score": max(0.0, min(1.0, raw_score / scoring_total)) if raw_score is not None else None,
            "passed": eval_result["predicted_label"] == 1,
            "reason": eval_result["evaluation_details"].get("reasoning", ""),
            "metadata": {"failure_classification": eval_result.get("failure_classification")},
        }],
        "metadata": {"steps": agent_result["metrics"]["steps"],
                      "usage": agent_result["metrics"].get("usage")},
    }

What this would not claim to solve: it doesn't touch bubench run/bubench eval internals, the leaderboard, or community/results/submission.json — those stay exactly as they are. This is purely an optional, standalone to_openeval()/from_openeval() package (own repo, own tests against the real validator, zero footprint here) so a task.jsonl + experiments/.../result.json pair could round-trip into a format other eval tooling (Promptfoo, Inspect AI, etc.) can also read, the same way the FinanceBench and DeepEval adapters already do for their respective ecosystems.

If this is useful, I'd rather build it as a real, tested PR against adapters/ in the EvalPort repo (linked above) than ask for anything to change here — just wanted to check first whether the mapping actually makes sense to people who know this dataset and its scoring model better than I do, and whether there's prior art or a reason this wouldn't be worth the effort (e.g. if the rubric-per-task shape is expected to change before v2.0 — I saw LexBench-Browser2.0 already exists as a data directory).

— Sahi, independent contributor (not affiliated with this project)