Unbounded memory growth in poll_transmit on a lossy uplink — single realloc-doubling Vec (multipath path churn)

[damianmiles]

A long-lived noq connection (via iroh 1.0.2) on a macOS host with a poor uplink grows its physical footprint from ~200 MB to ~19 GB in ~60–80 s, then (in our case) an external memory guard kills the process. It repeats every ~30-90 min. This is genuinely live memory (see the heap breakdown below — DIRTY, not swapped/compressed), dominated by a single realloc-growing Vec on the connection transmit path. A sample during the spike is overwhelmingly in noq_proto::connection::poll_transmit and path management.

We do not use multipath transfer — this is the base endpoint (gossip + relay/direct connectivity). Multipath is negotiated by default and cannot be disabled (see below).

Versions

• noq = 1.0.1, noq-proto = 1.0.1, noq-udp = 1.0.1
• via iroh = 1.0.2 (iroh-blobs 0.103, iroh-gossip 0.101)
• macOS 26.5.2 (Apple Silicon), 8–24 GB RAM hosts

Footprint over time (sampled every 5 s with vmmap)

12:28–12:47   ~272 MB     (flat ~19 min)
12:47:50      10.7 GB
12:48:03      12.0 GB
12:48:29      14.6 GB
12:48:56      16.8 GB
12:49:10      19.0 GB      -> killed & restarted (external ~80% RAM guard)
12:49:16      31.7 MB      (fresh process)

The stair-step (10.7 → 12 → 14.6 → 16.8 → 19) is consistent with successive realloc copies of a doubling buffer.

The smoking gun — heap + vmmap --summary at the spike

heap allocation-size histogram (note the two lone giant blocks, 2.56 GB = 2 × 1.28 GB — a Vec mid-doubling):

All zones: 504130 nodes malloced - Sizes:
  2.62144e6 KB [1]   1.31072e6 KB [1]   20752 KB [1] ...
  ^ 2.56 GB single    ^ 1.28 GB single
465706 "non-object" allocs = 4,535,865,520 bytes (avg ~9.7 KB)   # raw (Rust) heap

vmmap --summary (the growth is DIRTY/live, essentially nothing swapped):

Physical footprint:   3.2G          # (a smaller sample; same shape at 19 GB)
MALLOC_REALLOC        3.6G VIRTUAL  2.3G RESIDENT  2.3G DIRTY   0K SWAPPED
MALLOC_REALLOC(empty) 680M VIRTUAL  433M RESIDENT  433M DIRTY
DefaultMallocZone     4.4G          503777 allocations, 4.2G allocated

So this is not allocator retention of freed pages or compressed cold memory, it is a single live Vec in MALLOC_REALLOC that keeps doubling (1.28 GB → 2.56 GB → 5.12 GB → …), and each doubling's copy transiently needs old+new, which is what drives the footprint toward ~19 GB before the guard fires.

sample <pid> during the spike

Top allocation frame:

core::iter::…Copied<…BTreeMap<PathId, PathState>::Keys> as Iterator>::fold
  … Vec::extend_trusted
  (noq_proto::connection — path key collection)

Busiest non-idle frames (all noq_proto::connection / relay):

noq_proto::connection::Connection::poll_transmit          (many)
noq_proto::connection::Connection::path_data
noq_proto::connection::Connection::poll_transmit_path_space
noq_proto::connection::spaces::PacketSpace::can_send
noq_proto::connection::timer::TimerTable::peek

libsystem_malloc is by far the top symbol (allocation-bound, not compute-bound).

Relevant source (noq-proto 1.0.1, src/connection/mod.rs): self.paths.keys().collect::<Vec<_>>() (≈:770), self.paths.keys().copied().collect() (≈:799), and the poll_transmit path space loop. discard_path → self.paths.remove does run, so the paths BTreeMap itself isn't accumulating stale entries — the unbounded growth is a Vec allocated/extended on the hot transmit path (a pending-frame / send buffer, or a per-path-space collection that isn't drained under a stalled uplink).

Trigger / environment (the amplifier)

Host interfaces at the time:

en0      192.168.7.248     Wi-Fi (real)
utun4    100.64.0.209      Tailscale (real)
feth2328 192.168.194.130   Parallels VM bridge (dead-ends)
feth2328 192.168.194.131   Parallels VM bridge (dead-ends)
en8      169.254.66.213    self-assigned link-local (nothing connected)

iroh-doctor report: udp_v4: true, mapping_varies_by_dest_ipv4: Some(false) (easy/cone NAT — holepunch feasible), but far/slow relays (preferred aps1 Singapore 212 ms; https relay latency 0.7-1.5 s) and a lossy uplink. So established paths flap and get re-probed.

We tried removing the junk interfaces (ifconfig en8 down, destroyed the Parallels bridge) - the balloon still recurs, so it isn't only the dead candidates; it also happens on the real Wi-Fi/Tailscale paths under a lossy uplink.

Can't mitigate from the iroh side

• QuicTransportConfigBuilder::max_concurrent_multipath_paths(n) rejects any n below MAX_MULTIPATH_PATHS ("must be at minimum …, ignoring user supplied value"), and iroh hardcodes multipath on (max_concurrent_multipath_paths(MAX_MULTIPATH_PATHS)), so we can't disable or reduce it.
• iroh 1.0.2 is current and pulls this noq 1.0.1; nothing newer to move to.
• No supported way found to exclude non-routable / link-local / virtual local addresses from path candidate discovery.

Questions

1. Is unbounded growth of a Vec on the poll_transmit / path-space path a known issue on lossy/high-latency links? What's the intended bound?
2. Should path candidate discovery skip clearly-unusable local addresses (169.254/16 link-local, dead VM bridges) rather than probe/abandon them?
3. Is there any supported way to cap or damp multipath path re-probing (given max_concurrent_multipath_paths only allows increasing the count)?

Happy to provide

• Full sample output and a longer footprint trace.
• A MallocStackLogging=1 capture to pinpoint the exact allocation backtrace of the 2.56 GB block (the current heap can't name it without stack logging), say the word and we'll attach it.

Reproduces reliably on a typical dev laptop (VPN + Tailscale + a VM bridge + a lossy uplink), so it likely affects real end-user machines.

[damianmiles]

Update — exact allocation backtrace (MallocStackLogging + malloc_history)

Caught a spike with MallocStackLogging=1 and ran malloc_history <pid> -allBySize. The runaway Vec is the per-path sent-packet tracking buffer — PathData::sent, a Vec<Option<SentPacket>> growing unboundedly via grow_one in finish_and_track on the transmit path:

realloc  (libsystem_malloc)
 └ alloc::raw_vec::RawVecInner::finish_grow
   └ alloc::raw_vec::RawVec<Option<noq_proto::connection::spaces::SentPacket>>::grow_one
     └ noq_proto::connection::paths::PathData::sent
       └ noq_proto::connection::packet_builder::PacketBuilder::finish_and_track
         └ noq_proto::connection::Connection::poll_transmit_path_space
           └ noq_proto::connection::Connection::poll_transmit
             └ noq::connection::ConnectionDriver  (Future::poll)
               └ iroh / noq runtime spawn

What this clarifies

• The sample output's hot frame BTreeMap<PathId, PathState>::keys().collect::<Vec<_>>() is CPU-hot but transient (allocated and freed each poll_transmit). It's a red herring for the memory.
• The live, growing memory is PathData::sent the Vec<Option<SentPacket>> that records in-flight/sent packets per path. This is the block that realloc-doubles (1.28 GB → 2.56 GB → …) to ~19 GB, matching the MALLOC_REALLOC DIRTY footprint from the earlier vmmap.

Hypothesis

On a lossy / high-latency path, packets are sent-and-tracked into PathData::sent but the buffer never drains, sent-packet records for a stalled or abandoned/flapping path don't appear to be reclaimed (acks not arriving, or the per-path sent vec isn't pruned/dropped when the path is discarded). So the vec grows without bound.

Worth checking: when a path is retired (discard_path / paths.remove), is its PathData::sent actually dropped? And under sustained loss, is there any cap / pruning on sent for a path that's making no ack progress?

Happy to attach the full malloc_history -allBySize dump (~137 MB) or capture a MallocStackLoggingNoCompact run for freed-block history if useful.