Skip to content

fix(bench): exclude wait duration from repetitive-loop fingerprint - #19

Open
AmirF194 wants to merge 1 commit into
Purewhiter:mainfrom
AmirF194:fix/14-loop-detect-wait-duration
Open

fix(bench): exclude wait duration from repetitive-loop fingerprint#19
AmirF194 wants to merge 1 commit into
Purewhiter:mainfrom
AmirF194:fix/14-loop-detect-wait-duration

Conversation

@AmirF194

@AmirF194 AmirF194 commented Aug 4, 2026

Copy link
Copy Markdown

Root cause: _action_fingerprint() in bench_env/runner/base.py hashes an action's full data payload, including WAIT's value field, which holds the wait duration in seconds. Controller.run()'s repetitive-loop guard truncates an episode only when the last loop_threshold fingerprints are all identical. An agent that varies the wait length on each call (choosing its own duration, or scaling it) produces a distinct fingerprint every time, so consecutive WAIT actions never compare equal and the guard never fires. The episode then runs to max_steps instead of truncating on a stuck agent, exactly the behavior reported in #14 and confirmed by the maintainer's own 2026-06-26 comment.

Fix: drop the value key from the fingerprint specifically when action_type is WAIT, so repeated waits are recognized as the same action for loop-detection regardless of their durations. Every other action type keeps its full payload in the fingerprint (an ANSWER's value is real content, not noise, and still needs to compare unequal across different answers).

Verified in a clean python:3.11-slim container:

  • New regression test (bench_env/tests/common/test_loop_detect.py) drives Controller.run() with an agent that emits WAIT with a different duration each step: fails on unpatched main (runs to max_steps, no REPETITIVE_LOOP) and passes with this fix (truncates at the third repeated wait).
  • Full offline suite (pytest bench_env/tests/ -m "not live"): 2634 passed on this branch, same 3 pre-existing failures and 1 collection error present on unmodified main (a node-dependent test, a flaky-import error, and two locale-timing judge assertions), none touched by this change.
  • Did not run: the live suite (-m live, needs a running simulator) or the JS/web test suite, neither of which this diff touches.

Closes #14

_action_fingerprint() hashed the full action payload, including WAIT's
"value" duration field. An agent that varies the wait length on every
call (e.g. picks a duration itself) produces a different fingerprint
each step, so Controller.run()'s loop guard never sees two identical
fingerprints and REPETITIVE_LOOP never trips; the episode runs to
max_steps instead of truncating early.

Drop the duration from the fingerprint specifically for WAIT actions
before hashing, so repeated waits are recognized as the same action
regardless of how long each one slept. Other action types are
unaffected.

Fixes Purewhiter#14
@AmirF194

Copy link
Copy Markdown
Author

Checking in on this one. Happy to rebase, split the diff, or adjust scope if that would help it move.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The loop_detect function needs improvement

1 participant