Cross-agent benchmark over a seeded environment family - #8
Open
gkirgizov wants to merge 11 commits into
Open
Conversation
The perlin food field was hardcoded at threshold=1.0, octaves=8 in both Env._init_data and jx.env.init_state, so the resource landscape was the one thing an environment family could not vary. Adds food_scale (abundance), food_octaves (patchiness) and food_threshold to Dynamics and EnvParams, mirrored per "two engines, one physics". On EnvParams they are static: init_state consumes them on the host, so sweeping them builds a different init bank rather than vmapping, exactly like init_agent_ratio. food_scale is a plain multiplier and provably leaves the spatial pattern alone (same nonzero cells, sum scales exactly), which is what makes it a clean abundance axis independent of octaves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
canonical.py pins the one config training and evaluation both import, so the two cannot drift apart. It is deliberately not a registered DynamicsId: presets are world flavours, this is a benchmark definition that also pins seed ranges and episode length. Its numbers come from the rejax cross-check so results stay comparable with docs/JOURNAL.md. family.py is a one-axis-at-a-time sweep around that baseline rather than a grid -- a cross product of six axes would be hundreds of configs whose differences could not be attributed to anything. 19 scenarios x 6 seeds = 114 worlds, over abundance, patchiness, food noise, comm noise, crowding and wave flow. The family's seeds are disjoint from both TRAIN_SEEDS and EVAL_SEEDS, so every number it produces is out-of-sample. Two bounds are encoded from measurement rather than taste: food_noise stops at 0.5, above which clipping the multiplicative draw at zero starts *adding* food and the harder world is measurably a richer one; and the octaves axis was checked for an abundance confound (16.7-18.5 total food across octaves 2..32, against a per-seed spread of +/-3.0, so it is neutral within noise). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both are trained only on TRAIN_SEEDS and scored on disjoint worlds, so the benchmark measures generalization to unseen food fields rather than recall of a memorized one. Three seeds each, and all of them are kept. docs/JOURNAL.md records a seed spread of +/-1.2 to 2.7 on this benchmark at effect sizes around +2, and two single-seed conclusions that had to be retracted; a single checkpoint would be an anecdote. This run makes the case concretely -- PPO seed 0 scores 2.985, barely above the do-nothing bar, while seeds 1 and 2 score 5.801 and 5.250. Held-out returns: PPO 4.68 +/- 1.24, evolution 6.73 +/- 0.80. The PPO figure reproduces the cross-check's own 4.68 +/- 1.22 for this config, which is the evidence that this harness scores the same thing docs/JOURNAL.md does. Trained at lr 1e-3, the rate the journal settles on after retracting 1e-2 as a single-seed artifact -- note HyperParams.learning_rate still defaults to the retracted value and its docstring still argues for it. LFS tracking is scoped to models/ rather than a global *.msgpack rule so stray checkpoints elsewhere in the tree are not silently captured. The files are small (845 B evolved, 49 KB PPO); LFS here is for the format, not the size. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
runner.py is the single path every reported number goes through. Until now each of the four private return helpers in tests/behavior made its own assumptions about episode length, alive-masking and averaging; a cross-agent comparison built on them would have measured the harness. What it scores is total unshaped energy gain over a fixed horizon -- the one quantity every path in the repo already agrees on, and the reason die.rl keeps raw_return separate from its shaped objective. Rollouts use step_env rather than the auto-resetting step, so a long-running world's score cannot be topped up with a fresh world's food-rich early steps. policies.py adapts the trained checkpoints to the same functional protocol the scripted JAX ports use. The Agent interface they normally wear is xarray-shaped and cannot be jitted or vmapped, which would put a Python loop around every world step of a 19 x 6 x 8 sweep. Cross-validated three ways, all through code paths independent of the trainer: inaction scores 2.748 against the journal's published 2.68-2.76, and replaying the checkpoints reproduces the training script's held-out numbers to three decimals (5.802 vs 5.801, 7.078 vs 7.078). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two figures because there are two questions. The heatmap is diverging and centred on the do-nothing baseline for each scenario, because the meaningful quantity here is not how much energy a policy gathered but whether it gathered more than a policy that never moved -- agents pay to act, so on a sequential ramp a return of +2 looks respectable while actually being a loss. The curves show degradation along each axis with a band for the spread across world seeds. The three training seeds of each learned agent collapse into one series with a visible band rather than three lines. Reading a seed difference as an effect is this benchmark's documented recurring failure mode, and a band makes the spread impossible to overlook. Colours follow the validated categorical order and were checked rather than chosen by eye: the six slots pass the lightness, chroma, CVD-separation and normal-vision gates on a light surface (worst adjacent CVD dE 9.1, worst normal-vision dE 19.6). Three slots fall under 3:1 contrast, which obligates relief, so every curve is directly labelled and the tidy CSV is the table view. Plotting reads the CSV rather than re-running the sweep, so figures can be reworked without paying for the rollouts and the numbers in a figure are provably the numbers measured. The documented replay commands are verified to run, and pass --agent-ratio 0.2 explicitly: die replay otherwise defaults to 0.1 and a different dynamics preset, so a default rollout is not the world the numbers came from. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The hand-written baselines only ran on the xarray engine, so benchmarking them against the learned policies meant a Python loop around every world step. These ports run in the same jitted, scanned rollout. The port forces one structural change: GradientAgent and PhysarumAgent mutate momentum and heading inside forward(), and hidden mutable state cannot cross a jit boundary, so the carry becomes an explicit pytree. Obs gains `coords`, the real-valued positions `cells` is rounded from. Not redundant: the scripted agents sense at an offset of 0.03, which is *smaller* than the rounding error cells has already discarded (0.5/(width-1) = 0.033 at 16x16), so a position reconstructed from cells would pick a different sensing cell than the original does. `cells_at` is split out of `agent_cells` so both engines share one rounding rule rather than growing a second copy. The rejax flat vector deliberately does not carry coords. That vector is what rejax's stock MLP baseline consumes directly, so widening it would add 512 inputs on the 16x16 world and silently invalidate the calibration in docs/JOURNAL.md; no policy on that path senses at an offset. Parity is the deliverable, and it is exact where the agents are deterministic. Physarum's random turn is handled by computing the draws on the JAX side and feeding the same array to the numpy agent, so the whole trajectory is comparable rather than just the deterministic subset. Comparisons are per-channel because deposit is ~100x larger than dx/dy and a pooled tolerance would let a completely wrong movement channel pass. Brownian is distributional only: the two uniform streams are unrelated. One deliberate divergence, tested rather than hidden: these mask actions by alive, while ConstAgent and GradientAgent write into dead slots. It changes no physics, but AGENTS.md's rule is to mask every per-agent computation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
10 agents x 19 scenarios x 6 seeds = 1140 rollouts, 72s. Three chart defects found by rendering the figures and looking at them, which the palette validator cannot catch because none of them are about colour: - The heatmap scaled to its maximum, and wave-flow's +53.3 flattened the other eighteen columns into indistinguishable gray. Now scaled to a robust quantile with outliers saturating; every cell carries its exact value, so saturation costs no information, and the colourbar says how many cells overflowed. - food_flow was drawn as a response curve despite having one value, producing isolated dots on an invented x-range. Single-valued axes are categorical and belong to the heatmap; they are now excluded from the curves. - Six series routinely end within a few percent of each other, so end-of-line labels overlapped and misattributed curves. They are now nudged apart in order-preserving fashion, after the y-scale settles, so a label never swaps sides with its own line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
HyperParams.learning_rate still argued that 1e-2 was "best on every seed tried", which docs/JOURNAL.md retracts: that came from a single-seed sweep, and across three seeds 1e-3 wins for per_agent too (4.39 +/- 1.41 vs 3.21 +/- 2.70). The docstring is the misleading part, so only it changes. The default value stays: moving it would silently shift every calibration measured against it, and that is a deliberate decision rather than a docs fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
die.utils.xy2polar takes the angle of x + np.multiply(1j, y), which is not arctan2(y, x): the complex multiply contributes copysign(0, y) to the real part, so (-0.0, +0.0) collapses to (+0.0, +0.0) and reports angle 0 where IEEE reports pi. Reachable, not theoretical. grad_clip discards weak gradients by multiplying by a boolean, preserving the sign bits of the direction it just threw away, and Physarum's undetermined_grad fires on angle 0 but not pi. Measured at seed 7 on the 16x16 world: at step 1, 28 cells sit in 0 < norm < grad_clip, 22 carry a negative zero, and 13 disagree between conventions -- a clipped gradient pointing left reads as a decided heading while one pointing right does not. jx.scripted._angle already reproduced this deliberately, so that the JAX Physarum is the same agent the xarray benchmark numbers describe. What was missing is a test saying so. Without it, "cleaning up" xy2polar to arctan2 would surface as an unexplained multi-step parity drift instead of a named coupling. A second test asserts the quirk still exists, so the first cannot start passing vacuously. The underlying convention is arguably wrong and is left alone: changing it means changing both engines together and recalibrating tests/behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The environment family was static-modulo-noise plus monotonic depletion: food was eaten and never came back, and the one non-stationary scenario (wave flow) moved resource without creating any. Nothing in the repo could express a food source that depletes and reappears. Three mechanisms, deliberately distinct, because each does something the others provably cannot: - **Wave flow is now clipped at zero.** The wave term is a signed cosine mixture, and the channel it writes to is *food* -- a negative level inverts everything downstream, since feeding on a negative cell drains the agent and the env's own deduction adds resource back. Measured before the clip: the 16x16 world went from +16.1 total food to -9.6 within five steps, cells reaching -1.24. No parameter choice avoids this; the steady state is scale*wave/decay, so any decay > 0 drives troughs negative. - **Logistic regrowth** renews resource in place, deterministic and therefore in the physics core where it stays mirrored and bit-comparable. It cannot revive a cell at exactly zero -- the term is proportional to what is already there. - **Stochastic respawn** creates resource where there is none, which is the half regrowth cannot do. It needs an RNG, so it sits in the interface layer beside perturb_medium, keeping the core RNG-free. Rates are calibrated against measured consumption rather than picked: under physarum over 40 steps the non-renewable baseline runs 16.15 -> 10.18, and regrow_rate=0.02/capacity=0.5 and respawn_rate=0.005/amount=0.15 each land within 1% of where they started. An uncalibrated rate makes a "dynamic" world merely a richer one -- the failure mode already recorded for food noise. The regrowth/respawn distinction is measured, not just asserted: over 30 steps regrowth holds the occupied-cell count at exactly 123 while respawn grows it. One mistake worth recording: regrow_food originally skipped its work when the rate was zero, which is a Python branch on a traced leaf and raised TracerBoolConversionError across 18 tests as soon as params arrived as a jit argument. EnvParams' own docstring names this trap. The fix was removing the branch, not making the field static -- it is a continuous scalar that only multiplies, so it belongs on the traced side and must sweep under vmap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…variant The family gains three dynamic axes -- regrow_rate, respawn_rate and a swept wave_decay -- taking it from 19 scenarios to 27. wave_decay is swept rather than left at one point: it was previously a single scenario producing by far the largest effects in the family, which is the worst thing to under-sample. Learned agents are now trained twice, on the static world and a renewable one, so results separate "cannot handle a changing resource" from "never saw one". The renewable training config is renewal-only, and that was a correction. It first combined regrowth, respawn *and* the wave, on the reasoning that a policy shown one mechanism would have a blind spot. Measured, that was wrong in an instructive way: the mechanisms are not comparable in reward scale. A wave world pays a do-nothing baseline of ~70 against ~3.5 for a renewable one, so a combined objective is ~95% wave. The resulting checkpoints were wave specialists -- +152 over baseline on wave scenarios, and *worse than the static-trained agents* on the renewable ones they existed to handle (-4.0 against +2.1). Retrained without the wave, the evolved agent recovers to +3.9..+7.0 on exactly those scenarios. "Food moves" and "food depletes and returns" are different questions and now get different worlds. A caveat kept on the record rather than buried: the two training paths do not see identical physics. Evolution rolls out through the physics core while PPO goes through the interface layer, and stochastic respawn lives in the interface to keep the core RNG-free -- so the evolved renewable agents train under regrowth but never under respawn. Palette extended to the full validated eight-slot order, re-run through the validator: all eight pass on the adjacent pairlist (worst CVD dE 9.1, worst normal-vision dE 19.6). Eight is the cap, so a ninth series would need a facet. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compares every agent — scripted, evolved, and RL-trained — on one common footing across a seeded family of environments, spanning scarce→rich→noisy→dynamic worlds. Branches off
rl-rejax(now merged to main), so this targetsmain.What's here
src/die/jx/scripted.py) so all agents run in one jitted rollout. Pinned against the xarray originals with exact multi-step parity where deterministic (gradient/physarum at noise 0, max abs diff ~1e-8), distributional parity for brownian. Includes a test pinning a deliberate signed-zero angle-convention coupling.src/die/bench/): 27 scenarios × 6 disjoint world seeds, one axis varied at a time from a calibrated baseline. Every number is out-of-sample.benchmarks/,src/die/bench/plots.py): tidy CSV, a diverging heatmap centred on do-nothing, and per-axis response curves with seed error bands.Headline results (advantage over inaction)
deposit=4.5cost, not locomotion (deposit=0→ physarum +7).Validation
🤖 Generated with Claude Code