Make the memory preflight OOM guard configurable (auto: on for APUs, off for dGPUs) - #697
Open
ghazni101 wants to merge 2 commits into
Open
Make the memory preflight OOM guard configurable (auto: on for APUs, off for dGPUs)#697ghazni101 wants to merge 2 commits into
ghazni101 wants to merge 2 commits into
Conversation
The preflight OOM guard (kv_slots::preflight_alloc: deployment-target ceiling + MemAvailable headroom, plus the CLI bench-sweep headroom check) assumed GPU memory always comes from system RAM. That is true on unified-memory APUs (Strix Halo, no swap), where an overshoot is a global OOM that kills the desktop — and is exactly why the guard exists. On a discrete-GPU dev box an overshoot is a plain failed hipMalloc, and reviewer feedback points out agents running multiple daemons/serves there get refused loads the hardware would survive. Add a typed, process-scoped schema key memory.oom_guard (default true, env compat HIPFIRE_OOM_GUARD) so the operator can opt out: hipfire config set memory.oom_guard false # or: HIPFIRE_OOM_GUARD=0 Both production guard sites honor the knob at the single preflight_alloc choke point and in preflight_headroom_for_model; a disabled guard prints a one-line stderr note so a skipped refusal is visible in logs. Default behavior is unchanged: the guard stays on unless explicitly disabled, and scripts/run-bounded.sh remains the hard cgroup backstop either way.
Follow-up to 375f446. The guard's default is now the three-state `auto` (bool spellings still parse) instead of always-on, so dGPU dev boxes run unguarded by default and unified-memory boxes stay protected without anyone setting anything: auto: unified-memory APU arch (gfx1035/1036/1103/1150/1151/1152) → guard ON — GPU allocations come out of system RAM discrete-GPU arch (gfx90x/10x/1100-02/1200-01, CDNA) → guard OFF — an overshoot is a failed hipMalloc, not an OOM unrecognized arch → guard ON — fail safe; extend the table when support lands no GPU arch known in this process (CLI side, pre-init) → host swap state decides: swap → OFF (overcommit degrades, does not kill), no/unreadable swap → ON (fail safe) Mechanics: - hipfire-config: `memory.oom_guard` flips to process_auto_bool_field (AutoBool rule, default "auto"); new OomGuardMode + is_unified_memory_arch tables + /proc/meminfo SwapTotal probe + oom_guard_effective(arch) resolver that logs the auto decision once with its reason. Tables live here because rdna-compute cannot depend back on this crate. - rdna-compute: arch_caps records the DETECTED (not HIPFIRE_TARGET_ARCH- overridden) arch at Gpu::init — first init wins; kv_slots::preflight_alloc resolves through it. The refusal checks are split into preflight_checks so unit tests stay deterministic regardless of host config. - hipfire-cli: bench-sweep headroom check resolves with arch=None (host swap signal). Verified: config list shows default `auto`, HIPFIRE_OOM_GUARD=0 and `config set memory.oom_guard auto` both resolve; hipfire-config and kv_slots suites green; all touched crates compile.
ghazni101
force-pushed
the
patch/oom-guard-multislot
branch
from
September 4, 2026 08:14
ed7b3a0 to
e7e17cf
Compare
Kaden-Schutt
added a commit
that referenced
this pull request
Sep 7, 2026
…l, only host headroom gated by memory.oom_guard)
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #696.
The memory preflight OOM guard assumed GPU memory always comes from system RAM (true on unified-memory APUs like Strix Halo, false on discrete GPUs). This made it refuse valid loads on dGPU dev boxes with no way to opt out — biting multi-daemon dev workflows and bench sweeps where
MemAvailableis temporarily depressed.Changes
Two commits:
1.
feat(config): make the memory preflight OOM guard configurableAdds typed, process-scoped config key
memory.oom_guard(defaulttrue, env compatHIPFIRE_OOM_GUARD). Both guard sites (kv_slots::preflight_allocand the CLI bench-sweeppreflight_headroom_for_model) honor the knob. A disabled guard prints a one-line stderr note. Default behavior unchanged.2.
feat(config): oom_guard auto mode — enable only on unified-memory APUsDefault flips from always-on to
auto, resolved by deployment class:hipMalloc)Explicit
true/falseoverridesauto. The auto decision is logged once to stderr with its reason.Files
crates/hipfire-config/src/lib.rsOomGuardMode, arch classification tables,oom_guard_effective()resolver,AutoBoolconfig field, testscrates/rdna-compute/src/kv_slots.rspreflight_allocresolves throughoom_guard_effectivecrates/rdna-compute/src/arch_caps.rsprocess_gpu_arch()— records detected arch atGpu::initcrates/rdna-compute/src/dispatch.rsnote_process_gpu_archat initcrates/hipfire-cli/src/main.rsarch=NoneAGENTS.md,docs/CONFIG.md,docs/env-vars.mdVerification
hipfire-configandkv_slotsunit test suites passconfig listshows defaultauto;HIPFIRE_OOM_GUARD=0andconfig set memory.oom_guard autoboth resolve correctly