feat(valve-stack): Valve stack replication — pure-PE Proton 11.0 (Phases 1–3) - #82
Merged
Conversation
…on_11.0 source cloned, pure-PE gate corrected Phase 3 bootstrap status report (branch phase3/pure-pe-proton11): - container prereqs: no podman/docker installed; apt candidate 4.9.3; user in sudo group; install blocked on sudo password (user action) - SDK image correction: proton_11.0 pins steamrt4/sdk/x86_64: 4.0.20260331.220802-0 (NOT soldier/sdk as the task spec said); verified anonymously pullable via Docker Bearer token flow - source workspace: /home/wer/devis/steamflow-phase3/proton, proton_11.0 @ 0745bfb, 23 submodules, 3.9G (shallow clone, disk 19G free) - pure-PE gate corrected from source: Valve's Makefile always creates i386-unix/ with ONLY the 64-bit wine64 loader (--enable-win64, PE-only 32-bit side); classic-wow64 discriminator = 32-bit ELF .so files in i386-unix/ (verified against official 11.0 depot on disk)
…E path identified Phase 3 execution (branch phase3/pure-pe-proton11): - podman rootless solved: uidmap installed (user), slirp4netns missing -> ~/.local/bin/podman wrapper injects --network=host (host networking sufficient: container only fetches gecko/mono/xalia over HTTPS) - image_copy_tmp_dir -> /home/.cache/podman-image-tmp (root / is 31G, was filling during image unpack) - steamrt4 SDK 4.0.20260331.220802-0 pulled (~7G) - configure.sh OK with --container-engine=~/.local/bin/podman - make module=winex11.drv module: MODULE_EXIT 0 (after regenerating stale src-wine/include/wine/server_protocol.h via perl tools/make_requests) - CRITICAL: Valve default build = classic-wow64 (obj-wine-i386 is a full --host=i686-linux-gnu ELF-32 tree) — NOT pure PE; the i386-multilib-free goal needs ARCHS without i386-unix (single 64-bit tree with --enable-archs=i386-windows,x86_64-windows). Build-config patch to Proton's Makefile.in, no wine source changes. - -j16 hung the 14G-RAM box (kaldi parallel gcc); -j8 completed
…e scan Pure-PE WoW64 proof completed on phase3/pure-pe-proton11: Build-config patch (proton/Makefile.in, 3 edits, no wine source changes): - ARCHS drops i386-unix -> single 64-bit tree configures --enable-archs=x86_64,i386 --enable-win64 (both PE halves cross-compiled) - kaldi i386<->x86_64 serialization edges gated on i386-unix in ARCHS - module64 aliased to module32 (single tree builds both PE halves; the concurrent recipes raced on shared archives: libntdll.a file truncated) Result: make -j8 module=winex11.drv module -> MODULE_EXIT 0 Gate scan (obj-wine-x86_64/dlls/winex11.drv): - x86_64-windows/winex11.drv = PE32+ ; i386-windows/winex11.drv = PE32 - only .so is winex11.so = ELF 64-bit (unix side) - ELF 32-bit files: 0 ; i386-unix dirs: 0 (whole build tree) Disk: freed 10G->14G by removing dead i386-unix obj/dst trees (kept i386-windows PE ones: dxvk/vkd3d/vkd3d-proton/dxvk-nvapi). p2-* research dirs untouched. Patch artifact: docs/architecture/patches/proton-11.0-makefile-pure-pe.patch (apply with: patch -p1 < ... on a proton_11.0 checkout)
…ity harness, VERSIONS.txt visibility, native seed_prefix Phase 2 of the valve-stack directive (docs/architecture/valve-stack-replication.md): - items 1-2 (prior session): steam-cdn zstd/PICS fixes, test-diff harness (src/parity.rs), native Rust Proton ABI (src/runner/proton_abi.rs), headless test commands (src/headless.rs) - item 3 (tail): utils::write_runner_versions_txt writes VERSIONS.txt at runner extraction (GitHub tarball install, steam-cdn install_game, headless test_download_proton) — harvests tarball component version files + stamps RUNNER_VERSION; kills the found(bundled) display bug for flat-layout Proton trees; never overwrites an authoritative existing VERSIONS.txt - seed_prefix wired into prepare_prefix: fresh game prefixes seeded natively from runner files/share/default_pfx (no external Python init scripts) - i386-multilib permanently REJECTED (host is pure 64-bit) — recorded in dev book - test-diff 883710 re-run: 0 missing / 2 matched (PROTON_FORCE_LARGE_ADDRESS_AWARE, PROTON_USE_WINED3D resolved); EXTRA bucket = 2022-log format asymmetry - tests: 93 lib + 6 integration, all green
…s-compile in pure-PE build
…ymlinks deploy_dll_symlinks checked dest_path.exists(), which follows symlinks and returns false for dangling links (e.g. a link to a runner dir that was renamed/removed). The stale link was never removed, and the subsequent symlink() call failed with EEXIST, aborting PreparePrefix. Switch the existence check to symlink_metadata() (succeeds on dangling links) so stale links are removed and re-pointed like any other symlink.
PerGame mode seeds the per-game prefix (compatdata/<appid>/pfx) from the game runner's default_pfx via seed_prefix. The background Steam client was then hardcoded to the configured Steam Runtime Runner (default wine-tkg, classic-wow64), which cannot initialize inside a pure-PE-seeded prefix: wine-tkg's loader expects wow64.dll (unix-side WoW64 shim) which pure-PE wine deliberately does not ship → 'init_wow64: could not load wow64.dll' → Steam exits 53 and the launch aborts at PreparePrefix. Use the game's active runner for the background Steam in PerGame mode so the client matches the wine family that owns the prefix. Shared mode keeps the configured runtime runner (it owns the master prefix). Verified: test-launch 883710 resolves WINEPREFIX to compatdata/883710/pfx, Steam spawns without exit 53, and the game launches under the pure-PE runner.
…staged, E2E-verified on RE2 - phase3-pure-pe-proton11.md: bootstrap status → final record (build, blocker fixes, staging gates, live E2E, known limitation, open items all closed) - valve-stack-replication.md: Phase 3 INITIATED → CLOSED; header + open questions updated; SDK corrected to steamrt4 (branch pin), not soldier
…atch Shared prefix mode cannot host two different Wine/Proton runners in one WINEPREFIX (wineserver protocol mismatch -> 'wine client error: version mismatch'). effective_game_proton no longer force-equals the game runner to the Steam Runtime runner, so a Shared config with runtime runner = wine-tkg and game runner = steamflow-proton-11.0-purepe would crash at launch. Add effective_prefix_mode(ctx) / effective_prefix_mode_impl: apply the configured mode (per-game user config -> launcher default) and, when Shared AND the Steam Runtime runner path differs from the game runner path, fall back to PerGame with a visible warning. Thread the effective mode through every prefix decision: steam_wineprefix_for_game (new Option param), background-Steam spawn, CEF-enforcement prefix, symlink-deployment stage, registry-fixup stage and DLL-provider detection. UI management buttons and launch_custom_exec keep the configured mode (None). Tests (src/infra/runners/tests.rs): Shared + mismatched runners -> PerGame; matching runners -> Shared; no runtime runner -> Shared. Full lib suite (96) + launch_verification_tests (3) pass.
A prefix seeded by an older runner keeps absolute symlinks into that runner's lib/wine tree (system32/*.dll, syswow64/*.dll). When the runner dir is renamed/removed (e.g. swapping the vendored cachyos copy for the pure-PE build), every builtin link dangles and ANY wine — game runner or background Steam — dies before launch with: wine: could not load kernel32.dll, status c0000135 (exit 53) seed_prefix only seeds when system.reg is absent, and deploy_dll_symlinks only covers game-DLL providers, so the broken prefix was never repaired. Add utils::repair_dangling_prefix_symlinks(): walks system32/syswow64, re-points dangling lib/wine links at the ACTIVE runner's equivalent file, and drops links the active runner doesn't ship. prepare_prefix runs it on every launch (no-op scan on healthy prefixes). Regression test covers repoint / drop / untouched-healthy / idempotency.
Owner
Author
Follow-up commit (2026-08-12): prefix self-heal for dangling builtin DLL symlinksUser-facing bug this fixes: background Steam died 1s after spawn with
Fix: Verified: new unit test (repoint/drop/untouched-healthy/idempotent) + live: after repairing the RE2 prefix (1,306 re-pointed, 16 dropped), the exact background-Steam invocation boots Also: the app-level exit-53 message is now only reachable when the Windows Steam install is genuinely broken — the stale-prefix case self-heals first. |
…ime runner Exit-1 regression after the pure-PE split (RE2 launch, 2026-08-13): PerGame mode spawned background Steam with the GAME's runner (purepe) in the per-game prefix. The Windows Steam client CANNOT boot under purepe (documented 2026-08-12: CEF GPU-process crash + 32-bit network-init stall — two independent purepe defects), so the client died ~2s after spawn (exit 1, no Steam logs) and the launch aborted at PreparePrefix. The stale-wineserver guard also targeted the per-game prefix, and the 'is Steam running' check looked at the per-game prefix — so even with the user's wine-tkg client running in the master prefix (via Manage), SteamFlow spawned a doomed duplicate. Fix — the client ALWAYS belongs to the master prefix under the Steam Runtime runner (wine-tkg), in Shared AND PerGame mode: - PerGame mode: prefix_steam_dir/steam_wineprefix now resolve to the master Steam dir + master prefix (the per-game prefix still receives the client-file deployment for STEAM_COMPAT_CLIENT_INSTALL_PATH). - steam_runner: always the configured Steam Runtime runner (wine-tkg), never the game's runner. - Stale-wineserver guard: targets the GAME prefix (effective_game_prefix), so it can't kill the running master client. - Deployed client files (steam.exe, steamclient*.dll) are now REFRESHED from master when a stale real-file copy exists (the per-game prefix had a Feb-14 steam.exe that self-exits with code 1; only symlink-or-copy-if- missing meant stale copies persisted forever). Verified: test-launch 883710 → background Steam under wine-tkg, ready signal in 6s, effective_steam_wineprefix = master_steam_prefix/pfx, game launches under purepe, swapchain created, session result: Success.
Owner
Author
Follow-up (2026-08-13): background Steam exit-1 fix — client always in master prefix under runtime runnerYour reported bug: Play button → Root cause (two compounding):
Fix (commit b5f5c0a): the client ALWAYS runs in the master prefix under the Steam Runtime runner (wine-tkg) — Shared AND PerGame. The per-game prefix keeps only the client-file deployment (for Verified live: |
…window self-exit b5f5c0a parked the background Steam client in the MASTER prefix while games run in per-game prefixes (pure-PE). Wine named pipes are per-wineserver, so the game's steamclient.dll cannot reach the client's pipe → SteamAPI_Init fails → RE2 self-exits ~3s after the window appears (clean MFShutdown, no crash) and Portal 2 shows 'Steam must be running' even while Steam runs. b5f5c0a's 'verified' check (Steam ready in 6s + swapchain) used the same shallow 2s alive-check that masks the exit. Revert the placement to 8a56ed2 (client spawns with the game's runner in the per-game prefix — same wineserver as the game), keep b5f5c0a's client-file refresh (the real exit-1 fix was the stale Feb-14 steam.exe), and keep the stale-wineserver guard on the game prefix. launch_verification: two-phase check — 2s fast-fail for instant crashes, then a sustained-liveness window to 8s (500ms polls) so a graceful 3-5s-after-window self-exit is reported failed_after_spawn with the real lifetime instead of 'Success'. New test test_launch_verification_graceful_self_exit_caught covers the 4s case the old 2s check missed.
SteamClient::sync_master_session_to_prefix() copies the master client's
auth state into a per-game prefix's Steam dir before the headless client
spawns there:
- config/loginusers.vdf (account registry, AutoLogin/RememberPassword,
fresh Timestamp) — copied wholesale
- config/config.vdf Authentication block (RememberedMachineID JWT) —
merged, preserving the target's other per-prefix keys
- ssfn* sentry files (legacy) — copied when present
- HKCU\Software\Valve\Steam section of user.reg (AutoLoginUser, ...) —
per-key overlay, preserving target keys/subkeys and EOL style
Guards: no-op when the master client has no session; never downgrades a
per-game login as fresh as (or fresher than) master's; non-fatal.
Wired into the PerGame spawn path (wine_tkg.rs) right before the client
spawn, only when no client is already running in the target prefix.
Verified: 8 new unit tests (VDF/registry merges, freshness guard, e2e
sync) + full suite 105 lib tests green; live run confirmed the 620 prefix
receives master's loginusers.vdf timestamp (1786715012), the fresh
RememberedMachineID JWT, and the registry login keys.
DOES NOT achieve a logged-in per-game client end-to-end. The purepe-hosted
client asserts ClientAPI_InitGlobalInstance (client_api.cpp:601, "missing
a 32-bit dependency of steamclient.dll") on every boot and stays anonymous;
under wine-tkg it connects to Steam CM but the copied machine token is
rejected ("Invalid Password"). Open architecture question recorded in
docs/architecture/phase3-pure-pe-proton11.md §Session-auth sync (2026-08-14).
…(2026-08-14) Sync machinery committed separately (feat 1543484). This section states the CONCLUSION (per-game client process required; pointer-only model disproven by b5f5c0a), the two Portal 2 blockers with evidence (purepe client_api.cpp:601 assert dumps; wine-tkg CM connect + Invalid Password token rejection), the MachineGuid seed_prefix doc/code mismatch as an unfixed bug, and the single next step (MachineGuid cross-prefix injection test under wine-tkg).
…uct decision (2026-08-14) Replaces the "single next step" with the executed result and corrected framing: - MachineGuid cross-prefix injection test EXECUTED, verdict: NOT sufficient. Three-run table (620's own GUID -> server-side Invalid Password; master's GUID injected -> local 'cached creds not available', never attempts; own GUID + interactive login -> 'OK' + fresh JWT persisted). MachineGuid affects which failure mode the client hits but is not the auth binding; seed_prefix() doc/code mismatch stays open as a lower-priority real bug. - Working auth mechanism: one-time interactive login per prefix (client's own key store cannot be file-synced). Fresh machine-bound JWT iss r:0001_28A6B125_2FF1F vs master's r:0012_28A6B9E0_68319; AutoLogin=1 persists. Portal 2 now boots past SteamAPI_Init, loads p2-rtx chain on RTX 3070. - PITFALL recorded: disk-full (100%) causes post-login client crash 'Failed to create thread (error 0x3e6)' threadtools.cpp:3870; JWT never persists. Backlog item: defensive free-space check, NOT fixed this session. - Next blocker #1: Shared/PerGame wineserver routing gap in the custom-exec path (launch_custom_exec passes None to steam_wineprefix_for_game, bypassing effective_prefix_mode's runner-mismatch guard -> Shared config resolves to master prefix; per-game logged-in client never detected). Decision logic verified in code; fix NOT yet implemented. - Next blocker #2: winproc 1400 wine-11 get_window_thread() regression (pre-existing, patch unbuilt) - game exit code 5 at Remix bridge handshake. - OPEN PRODUCT DECISION recorded (not resolved): one-time manual login per prefix as shipped default vs opt-in advanced/isolated session mode.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the Valve Stack Replication directive (Phases 1–3): SteamFlow now replicates Valve's compatibility stack — pinned 11.x pure-WoW64 runners, native Rust launch-semantics parity, and a from-source pure-PE Proton 11.0 build — instead of maintaining a parallel wine-master-tracking stack.
Phase 1 — pinned 11.x pure-WoW64 stack (verified 2026-08-11)
wine-tkg-staging-git-11.13(pure new-WoW64) via the headless pipeline: Steam API ownership gate passes, D3D12/vkd3d-proton initializes, continuous rendering.Phase 2 — launch-semantics parity (complete 2026-08-11)
steamflow test-diff <appid>env-parity harness (src/parity.rs): parses nativePROTON_LOG=1output and diffs against SteamFlow'seffective_env.json(MISSING / EXTRA / MISMATCHED / MATCHED). RE2: 0 missing, 2 matched after fixes.src/runner/proton_abi.rs): port of Valve'sprotonscript +default_pfxlaunch semantics — compat config assembly,PROTON_*/WINE_*env rules, base DLL overrides, native prefix seeding. No Python at game launch.VERSIONS.txtat extraction (killsfound(bundled)).Phase 3 — pure-PE WoW64 Proton 11.0 from source (CLOSED 2026-08-12)
proton_11.0source with the branch-pinned steamrt4 SDK (not soldier) via rootless podman → staged ascompatibilitytools.d/steamflow-proton-11.0-purepe(1.4 GB)..somultilib dependency removed.docs/architecture/patches/proton-11.0-makefile-pure-pe.patch.Runner fixes included
wow64.dll; wine-tkg Steam → exit 53).symlink_metadatato prevent EEXIST on dangling DLL symlinks.Files
src/runner/proton_abi.rs,src/parity.rs,src/headless.rs,docs/architecture/valve-stack-replication.md,docs/architecture/phase3-pure-pe-proton11.md,vendor/steam-cdn/src/utils/zstd.rs(ZSTD chunk support for modern Proton depots)Test Plan
cargo test— 96 unit + 37 integration tests, 0 failurescargo build --release— clean (local, 2m09s); CIrust.yamlalso builds + packages.debsteamflow test-launch 883710→ RE2 renders (windowed, 1926×1112)steamflow test-diff 883710→ 0 missing / 0 mismatched / 2 matchedDocs
docs/architecture/valve-stack-replication.md— directive, phases, findingsdocs/architecture/phase3-pure-pe-proton11.md— full Phase 3 record (build, gates, E2E evidence)