Skip to content

fix: retry transient get_entities failures during observation - #413

Open
JackHopkins wants to merge 2 commits into
mainfrom
fix/observation-robustness
Open

fix: retry transient get_entities failures during observation#413
JackHopkins wants to merge 2 commits into
mainfrom
fix/observation-robustness

Conversation

@JackHopkins

Copy link
Copy Markdown
Owner

Problem

During a 4-node, 64-step eval, one rollout died ~1h in with "Error getting observation: Error getting entities while getting observation" and --fail-on-error cancelled the other three healthy epochs. Two compounding issues:

  1. The wrapper in FactorioGymEnv.get_observation() discarded the underlying exception message, so the eval log contains no diagnosable cause.
  2. A single transient failure (the full 41-program action sequence replayed clean on a fresh server, twice) is fatal to the whole epoch — and with fail-on-error, to the whole run.

Fix

  • Retry get_entities up to 3 times (0s/2s/5s backoff) before failing the observation, logging each attempt.
  • Include the underlying error text in the raised exception so real failures are diagnosable from the log.

Verification

  • Lint/format pass with CI's ruff; environment imports clean; tests/gym_env/test_server_reuse.py passes (3/3)
  • Failure mode reproduced-by-absence: replaying both failed epochs' exact program sequences (47 and 41 programs) against live servers produced zero get_entities failures, confirming transience

A single transient get_entities failure (engine busy, RCON hiccup) killed
an entire multi-hour eval: one rollout errored ~1h in and fail-on-error
cancelled the other three healthy epochs. The action sequence replayed
clean, confirming the failure was transient. Retry up to 3 times with
short backoff, and include the underlying error in the raised message —
the previous wrapper discarded it, making the failure undiagnosable from
the eval log.
score() after a step could fail transiently (e.g. KeyError 'player'
from a corrupted RCON response) and kill the epoch. Apply the same
retry-with-backoff used for get_entities, via a shared _call_with_retry
helper used by both sites.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant