slay_rl

Native batched combat RL for Slay the Spire (Ironclad), built on gamerpuppy/sts_lightspeed.

The CPU simulator owns persistent full runs. Existing heuristics make every non-combat choice; a masked policy controls every combat decision. Winning a combat returns a terminal PPO transition and advances the same run to its next battle. Dying returns a terminal transition and starts the next deterministic run.

What is implemented

• C++ CombatBatch with stable zero-copy NumPy views, externally supplied contiguous buffers, and GIL-free stepping.
• Deterministic per-slot/full-run seeds and automatic combat-to-combat advance.
• Fixed 1,169-action mapping validated by Action::isValidAction.
• Packed int16[3217] observations, including the uint8[1169] legal mask.
• Compatibility-isolated schema v2 with int16[2048] features, a routed-head byte, a 147-byte bit-packed mask, and uint16 global action IDs.
• Direct state-aware legal generation (81 normal, 64 single-select, and bit/popcount multi-select checks) instead of 1,169 generic validations.
• Native OpenMP actor pools, CUDA-registered shared buffers, multi-stream actor batches, split 81/64/1,024 policy heads, and associative-scan GAE.
• Separate zero-copy rewards, terminals, truncations, and action masks.
• Persistent spawned CPU workers with two shared buffer slots per worker.
• Pinned asynchronous H2D staging and compact D2H action transfers.
• GPU-resident packed rollouts, GAE, returns, shuffling, and PPO minibatching.
• BF16 inference/training with measured eager/torch.compile selection.
• Atomic periodic checkpoints with optimizer, topology, versions, and RNG state.
• Raw, topology, observation-copy, run-generation, and collect/update benchmarks.
• Coverage for action blocks, selection modes, lifecycle, determinism, masking, checkpointing, and finite PPO losses.

Build and test

The build compiles against an existing sts_lightspeed checkout. Its default location is /home/dsnoble/projects/sts_lightspeed; override it with STS_LIGHTSPEED_DIR.

./scripts/build_native.sh
./scripts/test_native.sh

For a sanitizer build:

cmake -S native -B build/native-asan -G Ninja \
  -DSTS_LIGHTSPEED_DIR=/path/to/sts_lightspeed \
  -DSLAY_SANITIZE=ON
cmake --build build/native-asan --parallel
ASAN_OPTIONS=detect_leaks=0 \
LD_PRELOAD="$(gcc -print-file-name=libasan.so)" \
SLAY_COMBAT_BUILD_DIR=build/native-asan \
./venv/bin/python -m unittest discover -s tests -v

ASAN_OPTIONS=detect_leaks=0 \
LD_PRELOAD="$(gcc -print-file-name=libasan.so)" \
SLAY_COMBAT_BUILD_DIR=build/native-asan \
./venv/bin/python -m slay_rl.stress --actions 1000000

Train

Create the isolated Python 3.14 / CUDA 13.0 environment and verify the GPU:

./scripts/setup_gpu_env.sh

The CUDA backend never falls back to CPU:

./.venv-gpu/bin/python -m slay_rl.combat_train \
  --backend cuda-mp --device cuda \
  --workers 16 --envs-per-worker 64 \
  --rollout-steps 128 --minibatch-size 8192 \
  --duration 10m --save-every 20000000 \
  --metrics-path benchmarks/latest_cuda.jsonl

The schema-v2 backend performs one honest PPO epoch and records completed decisions separately from optimizer-consumed samples:

./.venv-gpu/bin/python -m slay_rl.combat_train \
  --backend cuda-async-v2 --device cuda \
  --workers 2 --envs-per-worker 4096 --actor-threads 4 \
  --inference-bucket 4096 --rollout-steps 64 \
  --minibatch-size 131072 --update-epochs 1 --hidden-size 64

The fail-closed acceptance benchmark is:

./.venv-gpu/bin/python -m slay_rl.combat_train \
  --backend cuda-async-v2 --throughput-gate \
  --warmup-duration 30s --benchmark-duration 60s

The July 28, 2026 formal result is in benchmarks/cuda_async_v2_gate.json. It passed native simulation, PPO, accounting, legality, lag, finite-loss, memory, and temperature checks, but did not pass the 2.0M/s end-to-end gate (1.125M/s overall). Consequently, the guarded ten-minute curriculum was intentionally not run. It can only be started after a report with "passed": true exists:

./.venv-gpu/bin/python -m slay_rl.combat_train \
  --backend cuda-async-v2 --ten-minute-curriculum \
  --checkpoint checkpoints/cuda_async_v2_ten_minute.pt

Resume either a format-v2 checkpoint or the preserved legacy checkpoints/ten_minute_bot.pt warm start:

./.venv-gpu/bin/python -m slay_rl.combat_train \
  --backend cuda-mp --resume checkpoints/ten_minute_bot.pt \
  --total-steps 10000000

For correctness comparisons, the old synchronous trainer is explicit:

./venv/bin/python -m slay_rl.combat_train \
  --backend cpu --num-envs 64 --iterations 10

--puffer-workers now fails with a migration explanation. PufferLib 4 is source-built around a selected environment; its revision is pinned in requirements-puffer4.lock, but puffer4 remains gated until the source integration beats cuda-mp by at least 10% with equivalent held-out results.

The complete topology grid can be run with --autotune; it measures actor threads (8, 12, 16), total environments (1,024, 2,048, 4,096, 8,192), inference buckets (256, 512, 1,024, 2,048, 4,096), horizons (64, 128, 256), and minibatches (8K, 16K, 32K, 64K, 128K).

Evaluate fixed held-out seeds across Ascensions 0/5/10/15/20:

./.venv-gpu/bin/python -m slay_rl.evaluate \
  checkpoints/cuda_ten_minute.pt \
  --output benchmarks/corrected_reward_baseline.json

Benchmark

./venv/bin/python -m slay_rl.benchmark \
  --batch-sizes 1 16 64 --steps 512 \
  --topology --workers 1 2 4 8 16 --envs-per-worker 16 \
  --train --save benchmarks/latest.json \
  --save-default configs/topology.json

The measured default and full local results are in configs/topology.json and benchmarks/topology.json. See docs/benchmarks.md for interpretation and docs/observation.md for the packed schema.

Stress the shared double-buffer protocol independently of Torch:

./venv/bin/python -m slay_rl.shared_stress --actions 1000000 \
  --workers 16 --envs-per-worker 32

The previous whole-run value-learning modules remain available as slay_rl.train and slay_rl.baseline.

Schema-v4 end-to-end training

Schema v4 trains only player-visible macro decisions. Combat weights are imported from the proven schema-v2 checkpoint, context is bypassed, and every combat tensor remains frozen during macro PPO. The permanent complete-run return is ±1 + 0.5 * min(floor, 51) / 51 + 0.01 * completed_combats; it is paid once and copied without discounting to every macro transition in that run.

The current measured production result, exact checkpoint metadata, sweep history, live-export procedure, known pitfalls, and remaining promotion gate are recorded in docs/schema_v4_handoff.md.

Collect the two-million-decision, complete-seed-split demonstration set and clone/value-pretrain it:

./.venv-gpu/bin/python -m slay_rl.e2e_train collect \
  --decisions 2000000 --demos checkpoints/schema_v4_demos_routes --device cuda
./.venv-gpu/bin/python -m slay_rl.e2e_train clone \
  --demos checkpoints/schema_v4_demos_routes \
  --checkpoint checkpoints/schema_v4_bc.pt --device cuda

The full command profiles 4x512, 8x256, and 16x128, gates BF16, clones the 256/512/768 capacity variants, and executes the specified four-hour successive-halving schedule:

./.venv-gpu/bin/python -m slay_rl.e2e_train sweep \
  --demos checkpoints/schema_v4_demos_routes \
  --sweep-dir checkpoints/schema_v4_sweep --device cuda

The sweep command records its winner but does not automatically deploy it. The current production checkpoint was promoted separately after paired evaluation, isolated-combat, and export-parity checks. A post-shop-guard real-game smoke run is still required before treating the live promotion gate as complete.

Play the real game

The installed CommunicationMod adapter uses the Torch-free export of checkpoints/schema_v4_production_2h.pt for supported macro and combat screens. Unsupported states use a guarded fallback, and protocol-only transitions are forced when CommunicationMod exposes a state that is not a separate simulator decision.

Because Slay the Spire runs inside Steam Linux Runtime Soldier (Python 3.7), CommunicationMod launches the Torch-free exported policy:

/home/dsnoble/projects/sts-ai/venv/bin/python /home/dsnoble/projects/sts-ai/live_bot_py37.py

The installed configuration is generated from configs/communication_mod_policy.properties. Launch ModTheSpire with BaseMod and CommunicationMod enabled; runAtGameStart=true starts an Ironclad run automatically. The exported network is generated by scripts/export_live_policy.py and preserves greedy PyTorch decisions without requiring Torch in Steam's container. Live decisions are recorded in /home/dsnoble/projects/sts-ai/policy_bot.log. Set BOT_SEED, BOT_ASCENSION, BOT_STOP_AFTER_RUN, or BOT_POLICY_ASSETS in the launch environment to override their defaults. BOT_STOP_AFTER_RUN defaults to 1: the driver starts or resumes one run and then remains idle at game over instead of starting another. On a loss, the driver sends no proceed command, so the death/score screen stays visible until you leave it manually.

Restart ModTheSpire after changing either live driver. The most recent real-game incident was a shop cancel/enter loop caused by CommunicationMod splitting one simulator shop state into SHOP_SCREEN and SHOP_ROOM; the installed driver now has a one-shot transition guard. Run one post-restart smoke game and confirm there are no fallbacks, invalid commands, or repeated screen loops. See docs/schema_v4_handoff.md for the incident details and exact installed-file backups.