feat(benchmark): add reviewed LHTB generic CLI heartbeat runner - #4504
Conversation
Signed-off-by: shangzh0 <97216392+shangzh0@users.noreply.github.com>
Signed-off-by: shangzh0 <97216392+shangzh0@users.noreply.github.com>
Signed-off-by: shangzh0 <97216392+shangzh0@users.noreply.github.com>
Signed-off-by: shangzh0 <97216392+shangzh0@users.noreply.github.com>
动机复审 exact head 改动思路这条路径是 具体改动P1(阻断)外层 scheduler timeout 不能保证 fresh wake/Codex 已终止
当 worker 已运行一段时间后再启动一个最长 4800 秒的 wake,5080 秒总 deadline 可能先到:外层 这会允许 agent 在 verifier 或下一 phase 开始后继续改工作区,并且 Harbor 的 tmux-descendant freeze 无法可靠捕获已经 reparent 的独立 session,属于评分完整性问题。最小修复是让 worker 自己拥有总 deadline,把每次 wake timeout 截断为剩余预算,并在任何 SIGTERM/异常路径中 drain/kill 当前 wake 的完整进程树;在确认后代退出前不能把外层 124 当作正常完成。回归测试应在临近总 deadline 时启动长 wake,并断言 worker 返回前其后代 PID 已消失。 P2 可配置 network 名称没有传入 Compose
P2 Harbor patch 的兼容性只检查 marker,可能接受旧版或半安装状态
P3 smoke 可选择 separate-verifier 任务,但不会准备其本地镜像
对主干的风险P1 会改变被评分工作区的时间边界,可能导致 verifier 与未终止 agent 并发,是当前不能合并的 blocker。两个 P2 会让同一份公开配置在不同宿主机状态下得到不同网络行为,破坏 runner 的可复现性;P3 影响干净环境的 smoke 入口。默认关闭方面,Harbor patch 仍由 验证结果:当前 我的整体评价方向和 #4464 review 的修订是正确的,terminal packet 修复可以保留,LHTB 适配器的职责划分也基本清晰。但 exact head 尚不能视为评测安全:先修复并测试 descendant drain,再处理 network/patch 的真实 readback,才有足够证据证明“agent phase 已结束”和“preflight 所验证的网络就是实际运行网络”。 English verdict: REQUEST_CHANGES — exact head |
huangruiteng
left a comment
There was a problem hiding this comment.
动机
这个 PR 独立替代已经关闭的 #4464,做两件事:一是给 LHTB 46 任务补一个可复现的 generic_cli 心跳 treatment,二是修掉上一轮 review 在 scripts/external_scheduler_worker.py 上指出的真实缺陷。缺陷本身不难理解:worker 原来自己维护 TERMINAL_ACTIONS = {stop_until_explicit_resume},于是 producer 的另一种停止动作(peer 协调阻塞时的 return_to_owner_until_material_change)对 worker 不算终止;同时停止包会省略 cold_path_detail,而 worker 却先解析 cadence 再判断终止,于是"已经宣布结束的循环"要么继续被唤醒、要么直接报错。对跑长任务的操作者来说,这意味着配额被无意义地烧掉,而且每轮都是没有合法收尾的空转。
改动思路
作者没有往那个字面量集合里再加一个动作名——那只会把"谁定义停止"这件事留在消费侧,而是改成消费 producer 自己的指令:scheduler_hint.unchanged_poll.local_scheduler == stop 时立刻返回终止决策,cold_path_detail 不再参与判断。这一点我对着 producer 侧核过:loopx/control_plane/scheduler/scheduler_hint.py:249-256 的共享 stop-hint 构造器,以及 1257-1267 的 goal 停止/配额暂停路径,都会发出同一个指令,所以两个停止动作都真正终止。benchmark 侧则是新建 benchmark/LHTB/:一次唤醒一个全新的 codex exec --json(不使用 resume),续跑状态放在任务工作区和 trial 本地 registry;Codex/Python/Node/LoopX 都预先 staged,任务容器不下载;并且按上一轮要求复用 benchmark/swe-marathon/agents/codex_offline.py,不再自己留一份副本。
具体改动
scripts/external_scheduler_worker.py:删掉消费侧的动作集合,改成 producer 指令判定;停止包不再需要冷路径 cadence 细节。benchmark/LHTB/:run.sh(环境、网络、preflight、Harbor 启动)、configs/heartbeat-generic-cli.yaml(46 任务模板)、agents/codex_loopx_heartbeat.py(Harbor 生命周期与 Goal 建立)、runtime/wake_once.py(每轮唯一 Turn + 全新 codex exec)、scripts/(preflight、render_config、summarize_results、build_verifier_images)、harbor_patch/(可选 model-only 网络补丁)、verifier-images/(两个任务自带 verifier 镜像)。- 验证面:
tests/test_external_scheduler_worker.py与examples/external-scheduler-worker-smoke.py被扩展,而不是新加一份并行的 smoke。
关键代码讲解
scripts/external_scheduler_worker.py:45 LOCAL_SCHEDULER_STOP_DIRECTIVE与:97的短路返回:停止判定改为读 producer 指令,且在任何 cadence 解析之前返回TickDecision(terminal=True, after_limit="stop_tick_loop")。我用独立探针喂了两个 producer 停止动作,带指令时都返回terminal=True。- producer 侧对照:
loopx/control_plane/scheduler/scheduler_hint.py:249-256(共享 stop-hint)与1257-1267(goal_stopped/quota_paused)都发出unchanged_poll.local_scheduler: stop,所以上一轮那个"只覆盖一个停止动作"的问题在这条链上是真的关掉了,不是靠再补一个字面量。 benchmark/LHTB/run.sh:49-51,136:PYTHONPATH="$CODE_DIR/agents:$SHARED_CODEX_AGENT_DIR:...",并在文件缺失时直接die;scripts/preflight.py:159再校验同一条路径。head 上benchmark/LHTB/agents/只剩codex_loopx_heartbeat.py,之前那份重复的codex_offline.py确实没有了。benchmark/LHTB/runtime/wake_once.py:每个心跳唤醒一次全新codex exec,续跑靠任务工作区与 trial 本地 registry,而不是codex exec resume,这与 treatment 声明一致。
对主干的风险
共享面的改动只有 35 行,且没有引入新的 CLI 选项、协议字段或持久化状态。我跑到的证据:pytest -q tests/test_external_scheduler_worker.py 6 项通过;examples/external-scheduler-worker-smoke.py 12 项检查通过(含 test_end_to_end_terminal_stops_immediately,无需 sleep 即退出);独立探针还确认了反例——把指令从停止动作里拿掉时,worker 会以原有的 cold_path_detail ... missing 报错 fail closed,而不是静默继续轮询,这条耦合是安全的。公开边界上也干净:.env.example 用的是 /absolute/path/to/... 占位和 RFC 5737 的 192.0.2.x 文档网段,runs/、reports/、.env 都在 .gitignore 里保留 .gitkeep,没有凭据、原始轨迹或本地绝对路径进入提交。
一条非阻塞 P3 与一条残余风险:
- verifier 基础镜像只按可变 tag 固定(
benchmark/LHTB/verifier-images/nbody/Dockerfile:1、.../langchain/Dockerfile:1)。两个镜像都继承自第三方个人命名空间的时间 tag(zli12321/lhtb-*:20260615),仓库与 preflight 都没有记录 digest。判定任务是否通过的是 verifier 环境,上游重新打 tag 就能在不改本仓库一行代码的情况下改变它。建议把解析出的 digest 写进 README 并在scripts/preflight.py里断言(或直接按 digest 固定),这样重建要么可复现、要么直接失败。 - 终止语义现在完全依赖 producer 发出指令。这是有意的单一权威,且已由扩展测试覆盖两个已发出的停止动作;但若将来新增第三种停止动作而忘了带指令,worker 会走 fail-closed 报错路径(不会静默空转)。保持"每个 producer 停止路径都带指令"的断言即可。
另外,本环境没有 LHTB checkout、Harbor 与任务容器,因此这个 treatment 的端到端运行没有复现,这一项在我的结论里保持 unverified;harness 自身的 preflight 会在缺前置条件时快速失败。
English verdict: APPROVE — exact head 82ec4fd6e307d682f282b96170ad8f31a7a0ecea of #4504. The shared-code change is 35 lines and it removes a duplicate authority instead of extending a literal: terminality now comes from the producer's scheduler_hint.unchanged_poll.local_scheduler = stop, and I verified on the head that both producer stop paths emit that directive (loopx/control_plane/scheduler/scheduler_hint.py:249-256 and 1257-1267), so the earlier review's "only one of two stop actions is terminal" blocker is genuinely closed. An independent probe confirms both stop actions return terminal=True with the directive and fail closed with the pre-existing cold-path error without it, so the coupling is safe rather than silent. pytest -q tests/test_external_scheduler_worker.py passes 6 tests and examples/external-scheduler-worker-smoke.py passes 12 checks, including an end-to-end terminal stop with no sleep. The superseded revision's duplicated benchmark/LHTB/agents/codex_offline.py copy is gone: only codex_loopx_heartbeat.py remains, and reuse is guarded by run.sh:49-51,136 (PYTHONPATH plus fail-fast) and scripts/preflight.py:159. The public boundary is clean (placeholder /absolute/path/to and RFC 5737 192.0.2.x values only; runs/, reports/ and .env are gitignored). One non-blocking P3: the two verifier images inherit third-party date tags with no recorded digest, so the scoring-relevant verifier base can move without a repository change — record the digests in the README and assert them in preflight. Residual risk: terminality now depends entirely on the producer emitting the directive, which the extended tests cover for both emitted stop actions. No product runtime, quota, todo, authority or persistence behaviour changes, and no live LHTB task run was executed here.
huangruiteng
left a comment
There was a problem hiding this comment.
动机
这个 PR 独立替代已经关闭的 #4464,做两件事:一是给 LHTB 46 任务补一个可复现的 generic_cli 心跳 treatment,二是修掉上一轮 review 在 scripts/external_scheduler_worker.py 上指出的真实缺陷。缺陷本身不难理解:worker 原来自己维护 TERMINAL_ACTIONS = {stop_until_explicit_resume},于是 producer 的另一种停止动作(peer 协调阻塞时的 return_to_owner_until_material_change)对 worker 不算终止;同时停止包会省略 cold_path_detail,而 worker 却先解析 cadence 再判断终止,于是"已经宣布结束的循环"要么继续被唤醒、要么直接报错。对跑长任务的操作者来说,这意味着配额被无意义地烧掉,而且每轮都是没有合法收尾的空转。
改动思路
作者没有往那个字面量集合里再加一个动作名——那只会把"谁定义停止"这件事留在消费侧,而是改成消费 producer 自己的指令:scheduler_hint.unchanged_poll.local_scheduler == stop 时立刻返回终止决策,cold_path_detail 不再参与判断。这一点我对着 producer 侧核过:loopx/control_plane/scheduler/scheduler_hint.py:249-256 的共享 stop-hint 构造器,以及 1257-1267 的 goal 停止/配额暂停路径,都会发出同一个指令,所以两个停止动作都真正终止。benchmark 侧则是新建 benchmark/LHTB/:一次唤醒一个全新的 codex exec --json(不使用 resume),续跑状态放在任务工作区和 trial 本地 registry;Codex/Python/Node/LoopX 都预先 staged,任务容器不下载;并且按上一轮要求复用 benchmark/swe-marathon/agents/codex_offline.py,不再自己留一份副本。
具体改动
scripts/external_scheduler_worker.py:删掉消费侧的动作集合,改成 producer 指令判定;停止包不再需要冷路径 cadence 细节。benchmark/LHTB/:run.sh(环境、网络、preflight、Harbor 启动)、configs/heartbeat-generic-cli.yaml(46 任务模板)、agents/codex_loopx_heartbeat.py(Harbor 生命周期与 Goal 建立)、runtime/wake_once.py(每轮唯一 Turn + 全新 codex exec)、scripts/(preflight、render_config、summarize_results、build_verifier_images)、harbor_patch/(可选 model-only 网络补丁)、verifier-images/(两个任务自带 verifier 镜像)。- 验证面:
tests/test_external_scheduler_worker.py与examples/external-scheduler-worker-smoke.py被扩展,而不是新加一份并行的 smoke。
关键代码讲解
scripts/external_scheduler_worker.py:45 LOCAL_SCHEDULER_STOP_DIRECTIVE与:97的短路返回:停止判定改为读 producer 指令,且在任何 cadence 解析之前返回TickDecision(terminal=True, after_limit="stop_tick_loop")。我用独立探针喂了两个 producer 停止动作,带指令时都返回terminal=True。- producer 侧对照:
loopx/control_plane/scheduler/scheduler_hint.py:249-256(共享 stop-hint)与1257-1267(goal_stopped/quota_paused)都发出unchanged_poll.local_scheduler: stop,所以上一轮那个"只覆盖一个停止动作"的问题在这条链上是真的关掉了,不是靠再补一个字面量。 benchmark/LHTB/run.sh:49-51,136:PYTHONPATH="$CODE_DIR/agents:$SHARED_CODEX_AGENT_DIR:...",并在文件缺失时直接die;scripts/preflight.py:159再校验同一条路径。head 上benchmark/LHTB/agents/只剩codex_loopx_heartbeat.py,之前那份重复的codex_offline.py确实没有了。benchmark/LHTB/runtime/wake_once.py:每个心跳唤醒一次全新codex exec,续跑靠任务工作区与 trial 本地 registry,而不是codex exec resume,这与 treatment 声明一致。
对主干的风险
共享面的改动只有 35 行,且没有引入新的 CLI 选项、协议字段或持久化状态。我跑到的证据:pytest -q tests/test_external_scheduler_worker.py 6 项通过;examples/external-scheduler-worker-smoke.py 12 项检查通过(含 test_end_to_end_terminal_stops_immediately,无需 sleep 即退出);独立探针还确认了反例——把指令从停止动作里拿掉时,worker 会以原有的 cold_path_detail ... missing 报错 fail closed,而不是静默继续轮询,这条耦合是安全的。公开边界上也干净:.env.example 用的是 /absolute/path/to/... 占位和 RFC 5737 的 192.0.2.x 文档网段,runs/、reports/、.env 都在 .gitignore 里保留 .gitkeep,没有凭据、原始轨迹或本地绝对路径进入提交。
一条非阻塞 P3 与一条残余风险:
- verifier 基础镜像只按可变 tag 固定(
benchmark/LHTB/verifier-images/nbody/Dockerfile:1、.../langchain/Dockerfile:1)。两个镜像都继承自第三方个人命名空间的时间 tag(zli12321/lhtb-*:20260615),仓库与 preflight 都没有记录 digest。判定任务是否通过的是 verifier 环境,上游重新打 tag 就能在不改本仓库一行代码的情况下改变它。建议把解析出的 digest 写进 README 并在scripts/preflight.py里断言(或直接按 digest 固定),这样重建要么可复现、要么直接失败。 - 终止语义现在完全依赖 producer 发出指令。这是有意的单一权威,且已由扩展测试覆盖两个已发出的停止动作;但若将来新增第三种停止动作而忘了带指令,worker 会走 fail-closed 报错路径(不会静默空转)。保持"每个 producer 停止路径都带指令"的断言即可。
另外,本环境没有 LHTB checkout、Harbor 与任务容器,因此这个 treatment 的端到端运行没有复现,这一项在我的结论里保持 unverified;harness 自身的 preflight 会在缺前置条件时快速失败。
English verdict: APPROVE — exact head 82ec4fd6e307d682f282b96170ad8f31a7a0ecea of #4504. The shared-code change is 35 lines and it removes a duplicate authority instead of extending a literal: terminality now comes from the producer's scheduler_hint.unchanged_poll.local_scheduler = stop, and I verified on the head that both producer stop paths emit that directive (loopx/control_plane/scheduler/scheduler_hint.py:249-256 and 1257-1267), so the earlier review's "only one of two stop actions is terminal" blocker is genuinely closed. An independent probe confirms both stop actions return terminal=True with the directive and fail closed with the pre-existing cold-path error without it, so the coupling is safe rather than silent. pytest -q tests/test_external_scheduler_worker.py passes 6 tests and examples/external-scheduler-worker-smoke.py passes 12 checks, including an end-to-end terminal stop with no sleep. The superseded revision's duplicated benchmark/LHTB/agents/codex_offline.py copy is gone: only codex_loopx_heartbeat.py remains, and reuse is guarded by run.sh:49-51,136 (PYTHONPATH plus fail-fast) and scripts/preflight.py:159. The public boundary is clean (placeholder /absolute/path/to and RFC 5737 192.0.2.x values only; runs/, reports/ and .env are gitignored). One non-blocking P3: the two verifier images inherit third-party date tags with no recorded digest, so the scoring-relevant verifier base can move without a repository change — record the digests in the README and assert them in preflight. Residual risk: terminality now depends entirely on the producer emitting the directive, which the extended tests cover for both emitted stop actions. No product runtime, quota, todo, authority or persistence behaviour changes, and no live LHTB task run was executed here.
我的整体评价
整体 APPROVE。这个 PR 的价值不在体量,而在于把"谁定义停止"收敛回 producer:worker 删掉自己那份动作字面量、改为消费 unchanged_poll.local_scheduler=stop,我在 head 上核到 producer 的两条停止路径都发出该指令,所以上一轮 review 的阻塞项是真的关掉,而不是绕过去;指令缺失时 worker 会 fail closed 报错,而不是静默继续空转。benchmark 侧是本 PR 的主交付,位置正确(研究工件区、opt-in、不参与 CI 执行),并按上一轮要求删掉重复的 offline adapter,改用 PYTHONPATH 复用加 preflight 校验。公开边界干净,.env.example 只有占位符与 RFC 5737 文档网段,运行产物目录都被 gitignore。唯一 P3 是 verifier 基础镜像只固定可变 tag、没有 digest 记录,而 verifier 环境决定任务是否通过,建议按 digest 固定或在 preflight 断言;本环境没有 LHTB checkout 与 Harbor,treatment 端到端运行保持 unverified。
Summary
benchmark/LHTBgeneric_cliheartbeats through one freshcodex exec --jsonper wake, withoutcodex exec resumereplan_after_completed_todos=3scheduler_hint.unchanged_poll.local_scheduler=stopdirective before parsing omitted cadence detailThis supersedes #4464 and starts from current upstream
mainatf4ed58de9e44ff58e1a3e46731c520309e486e2f.Owner review addressed
The blocking and non-blocking findings from #4464 are included in this revision:
TERMINAL_ACTIONSliteral set. It derives terminality from the producer'sunchanged_poll.local_scheduler=stopdirective, so bothstop_until_explicit_resumeandreturn_to_owner_until_material_changeterminate without requiringcold_path_detail.terminal,after_limit, andunchanged_limit, and runs both producer stop actions end to end with no sleep and exit code 0.run_workerstate and exit behavior.benchmark/LHTB/agents/codex_offline.pyis removed; the runner explicitly importsbenchmark/swe-marathon/agents/codex_offline.py, and preflight plus a real Harbor import probe verify that resolution.Validation
python3 -m pytest -q tests/test_external_scheduler_worker.py(6 passed)python3 examples/external-scheduler-worker-smoke.py(12 checks passed)ruff check benchmark/LHTB scripts/external_scheduler_worker.py tests/test_external_scheduler_worker.py examples/external-scheduler-worker-smoke.pypython3 -m compileall -q benchmark/LHTB scripts/external_scheduler_worker.py tests/test_external_scheduler_worker.py examples/external-scheduler-worker-smoke.pybash -n benchmark/LHTB/run.sh benchmark/LHTB/scripts/build_verifier_images.shLoopxHeartbeatCodexinherits the shared swe-marathonCodexOfflinebenchmark/LHTB/run.sh preflightagainst the LHTB checkout: 46/46 tasks, 22 offline/24 online, 44 shared/2 separate verifier tasks, both scheduler stop actions, shared adapter, fresh-exec contract, Web Search disabled, and replan threshold readback all passedloopx canary premerge --from-git-diff --git-diff-base origin/main --tier standard: direct checks passed; 10/10 catalog canaries passed; 8/8 risk-profile smokes passed; public-boundary scan passed; 0 failures; expectedbenchmark_sensitivemanual-review hold retainedBoundaries
Future-facing scope check
The related bounded refactor was applied: terminality now has one producer-owned directive instead of a second consumer action vocabulary, and offline Codex staging has one existing adapter owner. A wider scheduler schema migration or benchmark framework extraction was intentionally deferred because neither is required for this runner or the reported stop-loop defect.