fix(bench): exclude wait duration from repetitive-loop fingerprint - #19
Open
AmirF194 wants to merge 1 commit into
Open
fix(bench): exclude wait duration from repetitive-loop fingerprint#19AmirF194 wants to merge 1 commit into
AmirF194 wants to merge 1 commit into
Conversation
_action_fingerprint() hashed the full action payload, including WAIT's "value" duration field. An agent that varies the wait length on every call (e.g. picks a duration itself) produces a different fingerprint each step, so Controller.run()'s loop guard never sees two identical fingerprints and REPETITIVE_LOOP never trips; the episode runs to max_steps instead of truncating early. Drop the duration from the fingerprint specifically for WAIT actions before hashing, so repeated waits are recognized as the same action regardless of how long each one slept. Other action types are unaffected. Fixes Purewhiter#14
Author
|
Checking in on this one. Happy to rebase, split the diff, or adjust scope if that would help it move. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause:
_action_fingerprint()inbench_env/runner/base.pyhashes an action's fulldatapayload, including WAIT'svaluefield, which holds the wait duration in seconds.Controller.run()'s repetitive-loop guard truncates an episode only when the lastloop_thresholdfingerprints are all identical. An agent that varies the wait length on each call (choosing its own duration, or scaling it) produces a distinct fingerprint every time, so consecutive WAIT actions never compare equal and the guard never fires. The episode then runs tomax_stepsinstead of truncating on a stuck agent, exactly the behavior reported in #14 and confirmed by the maintainer's own 2026-06-26 comment.Fix: drop the
valuekey from the fingerprint specifically whenaction_typeisWAIT, so repeated waits are recognized as the same action for loop-detection regardless of their durations. Every other action type keeps its full payload in the fingerprint (anANSWER'svalueis real content, not noise, and still needs to compare unequal across different answers).Verified in a clean
python:3.11-slimcontainer:bench_env/tests/common/test_loop_detect.py) drivesController.run()with an agent that emitsWAITwith a different duration each step: fails on unpatchedmain(runs tomax_steps, noREPETITIVE_LOOP) and passes with this fix (truncates at the third repeated wait).pytest bench_env/tests/ -m "not live"): 2634 passed on this branch, same 3 pre-existing failures and 1 collection error present on unmodifiedmain(anode-dependent test, aflaky-import error, and two locale-timing judge assertions), none touched by this change.-m live, needs a running simulator) or the JS/web test suite, neither of which this diff touches.Closes #14