Correct evaluator and canonical replay contracts - #3
Merged
Merged
Conversation
# Conflicts: # src/webarena_verified/core/evaluation/data_types/base.py # src/webarena_verified/core/evaluation/data_types/markdown_string.py # tests/api/test_corrected_task_contracts.py # tests/dataset/test_known_contract_corrections.py
Author
独立防作弊审查结论已按“是否降低门槛、修改答案或为特定浏览器放行”重新审查最终 diff,并加入对抗用例。结论:最终 head 审查中确实发现并修复了三类会产生假阳性的共享问题:
另外纠正了 task 760 把 Allentown 机场当作市区起点的问题;这是评测数据错误修正,不是放宽条件。Browser Eval 的旧报告仍绑定旧评测版本,不会把旧轨迹结果冒充为新合同结果。 验证:目标测试 45 passed;核心、数据完整性与 API 2355 passed / 3 skipped;Ruff、ty、diff check 通过。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
变更
last_event_only与数组类型的通用网络评测语义。这些修改属于所有执行器共用的评测器与 canonical dataset。浏览器轨迹、Rainbond 部署地址、运行证据和报告继续由
browser-eval维护。PR 关系
lex-main已包含 Fix final-page event evaluation and five verified task contracts #1 和 Correct verified task and network evaluation contracts #2,本 PR 以该最新基线为父提交继续修正。lexmount/webarena-verified当前唯一开放的评测修复 PR。验证
需要 Docker、外部站点或 Hugging Face 数据的测试未计入本地可运行套件。