SR-43 reject-at-load is post-instantiate: an over-cap declared memory min is fully allocated (~5.3GB under --memory 64KB) before rejection — --memory cap bypassed (DoS)

[avrabe]

Summary

The SR-43 reject-at-load on oversized memory min check (#420) runs post-instantiate, so a
module declaring a large memory min is fully allocated before the rejection fires — the
--memory cap it is supposed to enforce is transiently bypassed. A hostile/faulty module can force
kilnd to allocate multiple GB of RAM despite a tiny --memory cap, then exit "rejected". This is a
memory-exhaustion DoS.

This is the documented follow-up the #420 commit flagged ("rejecting before eager min allocation
needs a pre-instantiate engine API (follow-up)") — filing to track it, with a concrete measurement.
Verified on kiln HEAD 25da5f9 (v0.4.3).

Root cause (source)

kilnd/src/lib.rs:

860   // Instantiate
862   .instantiate(module_handle)          // <-- Memory::new allocates the declared `min` HERE
...
875   // SR-43: reject-at-load when the module's DECLARED memory min exceeds the cap
878   // NOTE: this runs post-instantiate, so the declared `min` was
879   // already allocated by Memory::new; rejecting before eager `min`
880   // allocation needs a pre-instantiate engine API (follow-up).
881   let declared_min = mem.0.ty.limits.min;
882   if declared_min > cap_pages { ...reject... }   // <-- too late, memory already committed

The --memory cap's runtime limiter (runtime_max_pages, SR-44) guards grow, but the eager
initial min allocation goes through Memory::new at instantiation, which the post-instantiate
check cannot prevent.

Reproduction (executed)

(module (memory 40000) (func (export "start") (drop (memory.size))))   ;; 40000 pages = ~2.56 GB min

$ /usr/bin/time -l kilnd --memory 65536 --function start huge.wasm     ;; 64 KB cap
✗ Execution failed: [Resource][E03F8] Module declared memory min exceeds --memory cap (rejected at load)
        0.28 real
        5298307072  maximum resident set size        # ~5.3 GB allocated for a "rejected" module

Contrast: a normal (memory 1) module peaks at ~55 MB RSS. So the "rejected" module resident-allocated
~5.3 GB under a 64 KB --memory cap before the reject fired. On a memory-constrained host this
OOMs/swaps — the exact resource exhaustion --memory is meant to prevent.

Impact

• The --memory cap is bypassable for the initial declared min: reject-at-load prevents execution
  but not allocation. Peak allocation is bounded only by the module's declared min (up to 2^32
  pages = 256 GB), not by --memory.
• Distinct from kilnd --memory <bytes> does not bound runtime guest memory: only gates a static module_size*2 admission estimate (memory.grow escapes the cap; --fuel is enforced but --memory is not) #411/kilnd resource reporting is fabricated: --memory-profile always reports 0, Peak memory = module_size*2, Fuel consumed = module_size/10 (static estimates, not real usage) #412 (those were the runtime-grow cap + reporting, now fixed) and from the SR-44
  table cap. This is specifically the declared-min eager-allocation path.

Suggested fix

Enforce the cap before the min is committed: either build the store's ResourceLimiter
(memory_growing, which wasmtime consults for the initial memory allocation too) from
config.max_memory so it denies an over-cap min at Memory::new time, or check
module.memories().map(|m| m.minimum) against the cap on the parsed module before instantiate.
Either rejects pre-allocation and closes the transient-DoS window.

Reported by the pulseengine-challenge harness (executed RSS measurement + source; the documented
post-instantiate follow-up from #420, now tracked).