Tiered event-driven observation protocol: ~2 ms full-state tensors for RL training - #414
Open
JackHopkins wants to merge 11 commits into
Open
Tiered event-driven observation protocol: ~2 ms full-state tensors for RL training#414JackHopkins wants to merge 11 commits into
JackHopkins wants to merge 11 commits into
Conversation
observation_diff.lua maintains a change buffer in storage: build/mine/die events append upsert/remove records immediately, and an on_nth_tick(47) round-robin slice scan reconciles mutations that fire no event (script-set direction, status transitions, entities created without raise_built). Clients poll obs_diff_drain() over RCON in O(changes) instead of rescanning every entity, and obs_diff_full_sync() provides the initial snapshot. At 1000 entities and game.speed 10, a steady-state drain takes ~1.8 ms (~580 obs/s) vs ~514 ms for get_entities(), and ~3 ms under 50-entity churn. benchmark_event_diff.py reproduces the measurements, including a client-side reconciliation consistency check. Note: this is wired into open_world only. default_lab_scenario embeds its level script inside the packed level (blueprint.zip); its on-disk control.lua is never loaded, so enabling the diff there requires re-exporting the scenario.
… resource reconciliation Extends the event-diff mod into a full-game-state protocol with four tiers by change rate: - Terrain (event-driven): on_chunk_generated emits each new chunk's water bitmask (32x32 bits hex), ore tiles with bucketed amounts, trees, rocks/cliffs, and enemy structures into a terrain buffer, so continual chunk generation streams to the client with no full rescans. - Structural (event-driven): build/mine/die/rotate as before; tree/rock deaths and on_resource_depleted feed the terrain buffer; research completions feed the entity buffer. - Drift (reconciled): the on_nth_tick(47) pass re-rows 50 entities and rescans 4 resource chunks, catching eventless mutations. - Hot fields (quantized): rows carry energy (MJ), crafting progress (deciles), recipe, inventories (log2 buckets), and fluids (log2), so an active factory only emits records on bucket transitions. obs_all_drain() returns both buffers in one RCON round trip. Measured at 1000 entities with 200 furnaces smelting at speed 10: combined poll 12 ms p50 (~84 obs/s, ~10 B payload steady-state), initial map-gen terrain burst 8097 records / 97 KB drained in 180 ms, 121 mid-episode chunks streamed as 66 KB, scripted ore depletion surfaced in 0.3 s, client state exactly matches authoritative scans. Reconciler costs ~70 UPS of the 553-UPS box64 baseline (tunable via ENTITY_SLICE).
TensorClient extends TieredClient to maintain a (17, 96, 96) float32 spatial grid (3-tile cells: entity type counts, status, direction sin/cos, energy, progress, inventory fullness, water, trees, rocks, ore, nests) plus an 8-dim global vector, updated incrementally from diff records via subtract-old/add-new additive contributions. observation() returns zero-copy flat views suitable for PufferLib-style MLP training. Measured at ~2600 entities with 200 furnaces smelting at speed 10: end-to-end poll (RCON drain -> reconcile -> tensor) 11 ms p50 (~87 obs/s); the reconcile+tensor step is 0.013 ms, so the pipeline is entirely RCON/tick-bound. Mid-episode attach (terrain full sync + entity snapshot, 9898 records / 184 KB) applies to the tensor in 9 ms; a full rebuild (the window-recenter path) takes 9 ms. Incremental grid verified consistent against the contribution map.
tests/observation_diff/ spins up its own Factorio container on a random localhost port (torn down after the session) and shadows the repo-level autouse fixture, so the suite never connects to shared cluster servers. 14 tests cover: RCON API presence, initial terrain population, advancing header tick, build/destroy event flow, exact client-vs-authoritative entity consistency, reconciliation of eventless direction changes, discovery of entities created without raise_built, hot-field quantization sparsity under active smelting, resource-depletion reconciliation, tree removal events, mid-episode chunk streaming, full-sync equivalence for a fresh client, and incremental-vs-rebuilt tensor equality. Also adds a periodic discovery sweep to observation_diff.lua (every 32 reconcile passes): entities created without raise_built fire no event and were previously invisible to the entity reconciler, which only re-rows already-tracked entities. The eventless-creation test would have caught this; now it passes.
The grid tensor's additive 3-tile cells lose object identity, adjacency, and sub-cell position, which interlinked-factory construction needs. TensorClient now maintains a second view of the same client state: a (2048, 16) float32 entity table plus mask, one row per entity with exact tile coordinates, direction sin/cos, type/recipe/item/fluid vocab ids, exact energy, progress, and item counts. Slots are free-list allocated with no compaction, so a row index is stable for an entity's lifetime and policies can reference specific entities across steps. Wire change in observation_diff.lua: rich_row now returns (row, sig) - rows transmit EXACT values (energy joules, progress percent, raw counts) while the quantized signature still gates emission, so drains stay sparse and client snapshots are exact with staleness bounded by bucket width. Tests: exact position/count delivery (including the subtlety that build events fire before same-command inserts, covered by reconciliation), slot stability across removals, and rebuild/dict occupancy equivalence; 17 pass. E2E poll with both views: 1.7 ms p50 (~508 obs/s), apply cost 7 us.
The wire row gains the per-entity fields the FLE object model (fle/env/entities.py) exposes: exact health, tile_width/height footprint, inserter/drill drop_position and pickup_position (exact coordinates, the adjacency data interlinked construction needs), electric_network_id, temperature, and inventory-role indices (I<invidx>.<item>:<count>). All static or sig-gated, so drain sparsity is unchanged (payload p50 9 B under active smelting). The table grows to (2048, 38) with named feature constants: position, direction, type/recipe vocab ids, status, energy, progress, health, footprint, drop/pickup offsets relative to the entity, electric network id, temperature, first fluidbox, 8 (item id, exact count) slots merged across inventories and ranked by count, plus total and distinct counts. New tests: drop/pickup offsets round-trip exactly against server truth for a rotated inserter, and a 9-item chest fills 8 slots in count order while totals still count everything; 19 pass.
The drain header now carries the character's exact position each poll (cached LuaEntity ref in storage, revalidated on death; falls back from connected player to any character entity). Script-created FLE agent characters fire no movement events, so live header reads are the reliable source. Client-side: the spatial grid window recenters onto the player with a 24-tile dead zone - walking inside the zone costs nothing, and crossing it triggers a grid-only rebuild (~9 ms) that leaves entity-table slots untouched. observation() now returns the entity table with player- relative x/y (exact, since the player position is exact) while absolute coordinates remain available internally for world-coordinate actions; the global vector carries the exact player position. New test creates a character far from origin and verifies the header reports its position, the grid recenters onto it, a chest outside the old origin window appears in the egocentric grid, and the table view's relative coordinates reconstruct the server's absolute truth; 20 pass.
Adds an egocentric section to the tensor benchmark: creates a character, then measures full polls (drain + apply + observation()) while the player is stationary, walking (6 tiles/poll, recenter every ~5 polls), and sprinting (60 tiles/poll, recenter every poll). Also isolates the observation() view cost and teleports the player back to the factory before the sanity checks so the window is populated when verified.
TensorClient takes a table_rows constructor argument (default unchanged); the stress run uses 32k rows. benchmark_stress.py generates 625 chunks, then builds fueled smelting cells (furnace + burner-inserter + chest + belt) in stages to ~20k entities across 500x260 tiles, measuring steady drains, UPS, drift latency, cold-attach cost, and forced-recenter polls at 5k/10k/20k. Measured (box64, speed 10): steady-poll payload stays ~15 B at every scale - the p50 latency growth (10 -> 20 -> 39 ms) tracks the UPS decline (444 -> 260 -> 127) from simulating 5k smelting furnaces, not data volume. Drift latency scales as predicted by ENTITY_SLICE (12 s at 5k, 143 s at 20k). Cold attach is dominated by server-side row building (14.5 s at 20k) while client apply+rebuild stays ~100 ms each. The final consistency check pauses the world first: this map's biters destroyed thousands of factory entities mid-run (all tracked via death events) and a live comparison lags by in-flight kills; paused, client and server match exactly.
observation() now yields a fixed (view_k, 38) view of the view_k entities nearest the player, nearest-first with player-relative coordinates, plus a matching mask - a factory-size-independent observation (924 KB total with the grid) instead of scaling the table view with world size. The full absolute table still tracks everything at table_rows capacity for actions, consistency checks, and re-selection as the player moves. Distance ordering means view rows are not lifetime-stable, so self.view_units maps each view row back to its unit_number (maintained via a slot->unit reverse map). Selection is argpartition + gather: 0.2 ms at 5k live entities, ~1 ms at 30k. Tests: live k-nearest test (character among close chests; a far chest is tracked but excluded from the view) plus offline ordering/padding/removal coverage; egocentric test updated to locate rows via view_units; 21 pass.
Uses a real FactorioInstance so actions run through the existing tool Lua (verified semantics, unmodified) against the tiered protocol. Measures per action: tool-call latency, one observation poll after it, and act-to-visible latency - the wall time until the effect appears in client state. Measured (warm server, ~600 UPS at speed 10): every action's effect arrives on exactly the tier the protocol predicts. move_to is visible in ~6 ms (live header); insert_item and rotate_entity in ~90 ms (one reconciler pass = 47 ticks); place_entity in ~2.5 s (discovery sweep = 32 passes, because place_entity's create_entity omits raise_built and fires no event). A move_to + observe cycle runs ~30 steps/s, dominated by character walking time, not the protocol. Findings encoded in the script: FLE build reach (10 tiles) requires moving to each site; rotate_entity recreates entities for some types and FLE's Direction enum does not map 1:1 onto engine e.direction for inserters, so rotation visibility is detected as a direction change at the position; a warmup gate avoids measuring box64 dynarec cold starts (10-50x slower in the first minute after container start).
|
Thanks for letting me know 🙏 |
Contributor
|
I think the CI failures stem from missing worker isolation. Should be fixed in #411 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a tiered, event-driven observation protocol that delivers a full, MLP-trainable game-state tensor in ~2 ms per poll (vs ~580 ms for
get_observation()at 1,000 entities), with exact client/server consistency.Protocol (
fle/cluster/scenarios/open_world/observation_diff.lua): twostoragechange buffers drained over RCON in O(changes).on_chunk_generatedstreams each new chunk's water bitmask, ore tiles (bucketed amounts), trees, rocks/cliffs, and enemy structures — continual chunk generation is handled by construction.on_resource_depleted,on_research_finished.raise_built(as FLE'splace_entitydoes).Client (
tests/benchmarks/benchmark_tensor_obs.py):TensorClientreconciles drains into fixed-shape float32 views, updated incrementally (~20 µs/poll):(17, 96, 96)tracking the player character (24-tile dead-zone recentering),(2048, 38)— exact positions/health/footprint/drop-pickup offsets/electric network/temperature/8 inventory slots, nearest-first, player-relative (view_unitsmaps rows to unit numbers),(10,)incl. exact player position. Total 236,554 floats (~924 KB), constant across factory sizes.Benchmarks (Apple M4 Max, box64, speed 10)
get_observation()@1k entitiesENTITY_SLICE)Running the tests
Running the benchmarks
Notes / limitations
open_worldonly:default_lab_scenarioloads its level script from the packed level insideblueprint.zip, so its on-disk control.lua is never read; enabling the protocol there means re-exporting that scenario or injecting the same Lua viaLuaScriptManager.entities / ENTITY_SLICE x 47 ticks / UPS— 12 s at 5k, 143 s at 20k. Spatially-prioritized reconciliation is the natural next step.