Skip to content

fix(dflash): roll back partial weight loads - #652

Closed
HUSRCF wants to merge 1 commit into
warpfront:masterfrom
HUSRCF:fix/dflash-partial-load-rollback
Closed

fix(dflash): roll back partial weight loads#652
HUSRCF wants to merge 1 commit into
warpfront:masterfrom
HUSRCF:fix/dflash-partial-load-rollback

Conversation

@HUSRCF

@HUSRCF HUSRCF commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Fixes the partial DflashWeights::load leak reported in #640. The HTTP streaming and long-context throughput sections remain separate follow-ups.

Summary

  • replace the fallible DflashLayerWeights { ... } literal with a per-field partial owner;
  • roll back top-level tensors, completed layers, the in-progress layer, DFlash2 convolution weights, selector projection, and ensure_mq_signs failure;
  • use WeightTensor::free_all so AWQ/Paro sidecars are included;
  • reuse the same layer teardown helper for normal unload and rollback;
  • add a thread-local deterministic load-failure hook and a dflash_smoke --fail-load-after N diagnostic.

The successful load order, tensor names, shapes, quant dispatch, and forward path are unchanged.

Validation

  • cargo check --locked -p hipfire-runtime
  • cargo test --locked -p hipfire-runtime --lib dflash — 19 passed
  • cargo check --locked -p hipfire-runtime --example dflash_smoke --features deltanet
  • gfx1100/ROCm 7.14 failure injection after 2, 10, 25, and 55 successful load steps:
    • every arm returned the injected failure;
    • retained HIP allocator heap stayed constant at 153,092,096 bytes rather than increasing with failure depth;
    • every arm immediately reloaded and fully unloaded the complete draft in the same process (LOAD_ROLLBACK_RETRY PASS).
  • real serve path, Qwen3.8-27B target + MQ4 DFlash draft, greedy:
    • output: The capital of France is Paris.
    • τ=3.5; empty=0, runaway=0, attractor=0, runtime errors=0
    • prompt MD5 bbf2d0483ecacaa2eb7296f9c8083f95

Remaining affected-machine validation

The reporter's original 32 GB gfx1201 configuration should rerun MQ6-Pro at max_seq=262144. This PR remains Draft until that OOM → AR-fallback path confirms that the failed draft load no longer starves the fallback.

@Kaden-Schutt

Copy link
Copy Markdown
Collaborator

Superseded by #691, now on beta (aad88151d): same DflashWeights::load rollback plus the scratch ctors, the generic builder, emitter rollback, and one ctx-cap predicate. Hardware-verified in gate runs 33902688013 / 33905366422.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants