Release a consumed tuple or Option box on the VM

CHANGELOG.md

@@ -59,6 +59,8 @@ The generated loop is now written from the program's source alone, and the manif
 - **`check` no longer asks a verify block of a function no verify case can call.** A parameter of a capability resource type (`Tcp.Connection`, `Work.Job`, a job kind's handle), or of a tuple, record or sum of the module that always carries one, has no value a case can write, so such a pure branching helper failed `error[missing-verify]` with no way to satisfy it. It is now exempt, the way effectful functions are. A parameter with an empty value (`List`, `Option`, a sum with a resource-free variant) still needs its verify block.
 - **A `main` that answers `Err` exits non-zero on wasm-gc and wasip2**, with the error on stderr on wasm-gc, as it already did on the VM and in generated Rust. Both wasm targets used to exit zero.
 - **`aver replay` of a run whose `main` answered `Err` matches.** The VM replay compared a runtime error against the recorded `Err` value and always reported a mismatch.
+- **The VM updates a Map in a field of a record in place when the record is consumed.** `State.update(state, counts = Map.set(state.counts, k, v), served = state.served + 1)` used to copy the whole Map on every call, even with `state` held by nothing else: the base of the update and the local, still needed for `served`, both held the record while the Map was set. A field that a record update or a record literal reads once, while every other read of the local in it is of another field and the local is not read afterwards, is now taken out of the record first, and the runtime does it only when exactly the holders the compiler accounted for hold the record. An update whose base nothing else holds moves the fields it keeps into the new record, so the dead base no longer keeps the next update of another field copying. In a release build, two thousand requests against a hundred-thousand-key Map now run in 0.06 s instead of 2.3 s, and ten thousand in 0.06 s instead of 9.6 s. A field two levels down (`setting.window.created`) is still copied; take the inner record out first.
+- **A generated loop updates an answer module's state in place on the VM too.** The loop hands the state out of the run in an `Option` and takes the answer back in a tuple, and on the VM the `Option` box and the tuple went on counting as holders of the state after the match had taken them apart, so every answer copied the module's Maps. A tuple, or the box of an `Option.Some`, `Result.Ok` or `Result.Err`, now counts its holders as maps, vectors and records do, and a `match` or `?` whose subject nothing reads afterwards releases what it holds when nothing else holds the tuple or box. The same goes for any program that carries a record through `Option`, `Result` or a tuple from one step to the next. `aver run --profile` also reports how many map entries the writes that were not in place copied. In a release build, 2000 requests against a 100,000-entry Map in the answer module's state run in 0.13 s instead of 1.8 s.
 - **The VM updates a Map in a field of a record in place when the record is consumed.** `State.update(state, counts = Map.set(state.counts, k, v), served = state.served + 1)` used to copy the whole Map on every call, even with `state` held by nothing else: the base of the update and the local, still needed for `served`, both held the record while the Map was set. A field that a record update or a record literal reads once, while every other read of the local in it is of another field and the local is not read afterwards, is now taken out of the record first, and the runtime does it only when exactly the holders the compiler accounted for hold the record. An update whose base nothing else holds moves the fields it keeps into the new record, so the dead base no longer keeps the next update of another field copying. In a release build, two thousand requests against a hundred-thousand-key Map now run in 0.06 s instead of 2.3 s, and ten thousand in 0.06 s instead of 9.6 s.
 - **The VM updates a Map further down a consumed record in place too.** `Setting.update(setting, window = Window.update(setting.window, created = Map.set(setting.window.created, k, v)), rounds = setting.rounds + 1)` and `Setting(window = absorbed(setting.window.created, setting.window.spent), rounds = setting.rounds + 1)` used to copy each Map on every call: `window` held it and `setting` held `window`. A path of fields read once in a record update or literal, under the same conditions as one field, is now taken out at the end of the path, and the runtime does it only when every record on the way down is held by exactly the holders the compiler accounted for: the record above it, and the bases of the updates the read sits in. A record on the way down is taken out of its parent when nothing else reads through it, so the update of that record moves the fields it keeps. A path read at the record's last use is taken the same way. `aver check` no longer reports such a path as `perf-shared-update`. In a release build, two thousand requests that each set a key in one hundred-thousand-entry Map and move a key to another, both inside a record inside the state, run in 0.04 s instead of 5 s.
 - **A wait over sockets and jobs keeps watching its sockets after a job outside its set settles.** The wake from that job used to end the socket poll, and the wait then slept out the rest of its timeout on the job engine alone, missing sockets that became ready meanwhile and never reporting them. The VM, generated Rust and the wasm-gc native host now share one wait loop that polls the whole set again.

aver-memory/src/arena.rs

@@ -195,7 +195,7 @@ impl<T: ArenaTypes> Arena<T> {
                 let idx = self.push(ArenaEntry::String(s));
                 NanValue::new_string(idx)
             }
-            ArenaEntry::Tuple(items) => {
+            ArenaEntry::Tuple { items, .. } => {
                 let imported: Vec<NanValue> =
                     items.iter().map(|v| self.deep_import(*v, source)).collect();
                 let idx = self.push_tuple(imported);
@@ -286,9 +286,9 @@ impl<T: ArenaTypes> Arena<T> {
                 });
                 NanValue::new_variant(idx)
             }
-            ArenaEntry::Boxed(inner) => {
+            ArenaEntry::Boxed { value: inner, .. } => {
                 let imported = self.deep_import(inner, source);
-                let idx = self.push(ArenaEntry::Boxed(imported));
+                let idx = self.push_boxed(imported);
                 NanValue::encode(value.tag(), ARENA_REF_BIT | (idx as u64))
             }
             // Fn/Builtin/Namespace — should not appear in independent product results
@@ -307,10 +307,7 @@ impl<T: ArenaTypes> Arena<T> {
     /// of marking sites for maps, vectors, and records.
     #[inline(always)]
     pub fn note_held_elsewhere(&mut self, value: NanValue) {
-        if value.is_heap_map()
-            || (value.is_vector() && !value.is_empty_vector_immediate())
-            || value.is_record()
-        {
+        if value.counts_holders() {
             self.mark_held_elsewhere(value.arena_index());
         }
     }
@@ -326,7 +323,9 @@ impl<T: ArenaTypes> Arena<T> {
         match self.get_mut(index) {
             ArenaEntry::Map { holder_count, .. }
             | ArenaEntry::Vector { holder_count, .. }
-            | ArenaEntry::Record { holder_count, .. } => {
+            | ArenaEntry::Record { holder_count, .. }
+            | ArenaEntry::Tuple { holder_count, .. }
+            | ArenaEntry::Boxed { holder_count, .. } => {
                 *holder_count = holder_count.saturating_add(1);
             }
             _ => {}
@@ -337,10 +336,7 @@ impl<T: ArenaTypes> Arena<T> {
     /// physically stopped holding `value`.
     #[inline(always)]
     fn release_held_elsewhere(&mut self, value: NanValue) {
-        if value.is_heap_map()
-            || (value.is_vector() && !value.is_empty_vector_immediate())
-            || value.is_record()
-        {
+        if value.counts_holders() {
             self.release_holder(value.arena_index());
         }
     }
@@ -350,7 +346,9 @@ impl<T: ArenaTypes> Arena<T> {
         match self.get_mut(index) {
             ArenaEntry::Map { holder_count, .. }
             | ArenaEntry::Vector { holder_count, .. }
-            | ArenaEntry::Record { holder_count, .. } => {
+            | ArenaEntry::Record { holder_count, .. }
+            | ArenaEntry::Tuple { holder_count, .. }
+            | ArenaEntry::Boxed { holder_count, .. } => {
                 // Saturation is sticky. Once exact cardinality is lost, the
                 // safe answer is "held" forever rather than a future false 0.
                 if *holder_count == u32::MAX {
@@ -384,8 +382,8 @@ impl<T: ArenaTypes> Arena<T> {
     #[inline(never)]
     fn note_entry_holds_takeable(&mut self, entry: &ArenaEntry<T>) {
         match entry {
-            ArenaEntry::Boxed(value) => self.note_held_elsewhere(*value),
-            ArenaEntry::Tuple(items) => {
+            ArenaEntry::Boxed { value, .. } => self.note_held_elsewhere(*value),
+            ArenaEntry::Tuple { items, .. } => {
                 for value in items {
                     self.note_held_elsewhere(*value);
                 }
@@ -492,6 +490,9 @@ impl<T: ArenaTypes> Arena<T> {
                 if self.holds_any_map || self.holds_any_vector || self.holds_any_record {
                     self.note_entry_holds_takeable(&entry);
                 }
+                if matches!(entry, ArenaEntry::Tuple { .. } | ArenaEntry::Boxed { .. }) {
+                    self.holds_any_record = true;
+                }
                 return self.push_heap(entry);
             }
         }
@@ -977,7 +978,10 @@ impl<T: ArenaTypes> Arena<T> {
         self.push(ArenaEntry::String(Rc::from(s)))
     }
     pub fn push_boxed(&mut self, val: NanValue) -> u32 {
-        self.push(ArenaEntry::Boxed(val))
+        self.push(ArenaEntry::Boxed {
+            value: val,
+            holder_count: 0,
+        })
     }
     pub fn push_record(&mut self, type_id: u32, fields: Vec<NanValue>) -> u32 {
         self.push(ArenaEntry::Record {
@@ -1017,10 +1021,7 @@ impl<T: ArenaTypes> Arena<T> {
         for (key, value) in map.values() {
             all_immediate &= key.is_immediate() && value.is_immediate();
             for child in [*key, *value] {
-                if child.is_heap_map()
-                    || (child.is_vector() && !child.is_empty_vector_immediate())
-                    || child.is_record()
-                {
+                if child.counts_holders() {
                     held.get_or_insert_default().push(child);
                 }
             }
@@ -1038,7 +1039,10 @@ impl<T: ArenaTypes> Arena<T> {
         })
     }
     pub fn push_tuple(&mut self, items: Vec<NanValue>) -> u32 {
-        self.push(ArenaEntry::Tuple(items))
+        self.push(ArenaEntry::Tuple {
+            items,
+            holder_count: 0,
+        })
     }
     /// Store a vector, marking every map or vector it holds as held by this
     /// entry — the vector spelling of [`Arena::push_map`]'s marking pass, made
@@ -1059,11 +1063,7 @@ impl<T: ArenaTypes> Arena<T> {
             if child.heap_index().is_some() {
                 all_immediate = false;
             }
-            if marking
-                && (child.is_heap_map()
-                    || (child.is_vector() && !child.is_empty_vector_immediate())
-                    || child.is_record())
-            {
+            if marking && child.counts_holders() {
                 held.get_or_insert_default().push(*child);
             }
         }
@@ -1131,7 +1131,7 @@ impl<T: ArenaTypes> Arena<T> {
     }
     pub fn get_boxed(&self, index: u32) -> NanValue {
         match self.get(index) {
-            ArenaEntry::Boxed(v) => *v,
+            ArenaEntry::Boxed { value, .. } => *value,
             _ => panic!("Arena: expected Boxed at {}", index),
         }
     }
@@ -1144,6 +1144,44 @@ impl<T: ArenaTypes> Arena<T> {
         }
     }
 
+    /// Whether a root or another arena entry has registered a reference to
+    /// this tuple or boxed wrapper. `false` for anything else.
+    pub fn wrapper_or_tuple_is_held_elsewhere(&self, value: NanValue) -> bool {
+        match self.get(value.arena_index()) {
+            ArenaEntry::Tuple { holder_count, .. } | ArenaEntry::Boxed { holder_count, .. } => {
+                *holder_count != 0
+            }
+            _ => true,
+        }
+    }
+
+    /// Empty a boxed wrapper nothing else holds and hand back its value. The
+    /// box stops holding the value, so the value loses that registered holder;
+    /// the caller has established that no root, entry or stack cell can reach
+    /// the box again.
+    pub fn take_boxed_value(&mut self, wrapper: NanValue) -> NanValue {
+        let value = match self.get_mut(wrapper.arena_index()) {
+            ArenaEntry::Boxed { value, .. } => std::mem::replace(value, NanValue::UNIT),
+            _ => panic!("Arena: expected Boxed at {}", wrapper.arena_index()),
+        };
+        self.release_held_elsewhere(value);
+        value
+    }
+
+    /// Empty a tuple nothing else holds, releasing the registered holder it
+    /// was of each item. The caller has established that no root, entry or
+    /// stack cell can reach the tuple again, and has already copied out the
+    /// items it needs.
+    pub fn release_tuple_items(&mut self, tuple: NanValue) {
+        let items = match self.get_mut(tuple.arena_index()) {
+            ArenaEntry::Tuple { items, .. } => std::mem::take(items),
+            _ => panic!("Arena: expected Tuple at {}", tuple.arena_index()),
+        };
+        for item in items {
+            self.release_held_elsewhere(item);
+        }
+    }
+
     /// Whether a root or another arena entry has registered a reference to
     /// this record. Operand-stack aliases are deliberately not represented
     /// here; the VM can inspect those directly after popping its operand.
@@ -1213,7 +1251,7 @@ impl<T: ArenaTypes> Arena<T> {
     }
     pub fn get_tuple(&self, index: u32) -> &[NanValue] {
         match self.get(index) {
-            ArenaEntry::Tuple(items) => items,
+            ArenaEntry::Tuple { items, .. } => items,
             _ => panic!("Arena: expected Tuple at {}", index),
         }
     }

aver-memory/src/lib.rs

@@ -179,8 +179,10 @@ pub struct VectorSlot {
 pub fn entry_holds_slot<T: ArenaTypes>(entry: &ArenaEntry<T>, index: u32) -> bool {
     let holds = |value: &NanValue| value.heap_index() == Some(index);
     match entry {
-        ArenaEntry::Boxed(value) => holds(value),
-        ArenaEntry::Tuple(items) | ArenaEntry::Vector { items, .. } => items.iter().any(holds),
+        ArenaEntry::Boxed { value, .. } => holds(value),
+        ArenaEntry::Tuple { items, .. } | ArenaEntry::Vector { items, .. } => {
+            items.iter().any(holds)
+        }
         ArenaEntry::Record { fields, .. } | ArenaEntry::Variant { fields, .. } => {
             fields.iter().any(holds)
         }
@@ -1088,6 +1090,29 @@ impl NanValue {
         self.is_nan_boxed() && self.tag() == TAG_TUPLE
     }
 
+    /// An `Option.Some`, `Result.Ok` or `Result.Err` whose value lives in an
+    /// arena box rather than inline in the wrapper.
+    #[inline]
+    pub fn is_boxed_wrapper(self) -> bool {
+        self.is_nan_boxed()
+            && matches!(self.tag(), TAG_SOME | TAG_OK | TAG_ERR)
+            && self.payload() & ARENA_REF_BIT != 0
+    }
+
+    /// Whether this value is an arena entry that counts its off-stack holders:
+    /// a heap map, a heap vector, a record, a tuple, or a boxed wrapper. These
+    /// are the entries the runtime may empty in place once nothing else holds
+    /// them, so every entry, root and table that stores one has to register
+    /// itself as a holder.
+    #[inline]
+    pub fn counts_holders(self) -> bool {
+        self.is_heap_map()
+            || (self.is_vector() && !self.is_empty_vector_immediate())
+            || self.is_record()
+            || self.is_tuple()
+            || self.is_boxed_wrapper()
+    }
+
     #[inline]
     pub fn is_builtin(self) -> bool {
         self.is_nan_boxed() && self.tag() == TAG_SYMBOL && self.symbol_kind() == SYMBOL_BUILTIN
@@ -1525,9 +1550,11 @@ pub struct Arena<T: ArenaTypes> {
     /// job: no vector entry means no value here can carry a vector's index,
     /// so the per-push marking pass has nothing to find.
     holds_any_vector: bool,
-    /// Whether this arena has ever stored a record. Records can be consumed by
-    /// the VM's last-use field projection, so an aggregate that stores one has
-    /// to register itself as an off-stack holder just like it does for maps and
+    /// Whether this arena has ever stored a record, a tuple or a boxed
+    /// wrapper. Records can be consumed by the VM's last-use field projection,
+    /// and tuples and boxes give up what they hold when they are destructured
+    /// with nothing else holding them, so an aggregate that stores one has to
+    /// register itself as an off-stack holder just like it does for maps and
     /// vectors.
     holds_any_record: bool,
     /// Which out-of-region slots the descent above has already rewritten, one
@@ -1577,7 +1604,13 @@ pub enum ArenaEntry<T: ArenaTypes> {
     BigInt(Box<num_bigint::BigInt>),
     String(Rc<str>),
     List(ArenaList),
-    Tuple(Vec<NanValue>),
+    Tuple {
+        items: Vec<NanValue>,
+        /// Registered off-stack holders of this tuple. A tuple destructured
+        /// where nothing else holds it gives its items up; see
+        /// [`Arena::release_tuple_items`].
+        holder_count: u32,
+    },
     /// A map, plus the same claim [`ListBody::all_immediate`] makes about a
     /// list body: `all_immediate` is `true` only when every key and every value
     /// in `map` is [`NanValue::is_immediate`], which makes relocating the table
@@ -1716,7 +1749,15 @@ pub enum ArenaEntry<T: ArenaTypes> {
         name: Rc<str>,
         members: Vec<(Rc<str>, NanValue)>,
     },
-    Boxed(NanValue),
+    /// The value inside an `Option.Some`, `Result.Ok` or `Result.Err` that
+    /// is not stored inline in the wrapper itself.
+    Boxed {
+        value: NanValue,
+        /// Registered off-stack holders of this box. A box unwrapped where
+        /// nothing else holds it gives its value up; see
+        /// [`Arena::take_boxed_value`].
+        holder_count: u32,
+    },
 }
 
 /// A borrowed view of an arena-stored integer, discriminating the
@@ -1767,8 +1808,7 @@ impl ListBody {
         let mut holds_takeable = false;
         for value in &items {
             all_immediate &= value.is_immediate();
-            holds_takeable |=
-                (value.is_map() || value.is_vector() || value.is_record()) && !value.is_immediate();
+            holds_takeable |= value.counts_holders();
         }
         Self {
             items,

aver-memory/src/memory.rs

@@ -371,13 +371,25 @@ impl<T: ArenaTypes> Arena<T> {
             ArenaEntry::String(s) => ArenaEntry::String(s),
             ArenaEntry::Builtin(name) => ArenaEntry::Builtin(name),
             ArenaEntry::Fn(f) => ArenaEntry::Fn(f),
-            ArenaEntry::Boxed(inner) => ArenaEntry::Boxed(rewrite(self, inner)),
+            ArenaEntry::Boxed {
+                value,
+                holder_count,
+            } => ArenaEntry::Boxed {
+                value: rewrite(self, value),
+                holder_count,
+            },
             ArenaEntry::List(list) => ArenaEntry::List(self.rewrite_list_with(list, rewrite)),
-            ArenaEntry::Tuple(mut items) => {
+            ArenaEntry::Tuple {
+                mut items,
+                holder_count,
+            } => {
                 for value in &mut items {
                     *value = rewrite(self, *value);
                 }
-                ArenaEntry::Tuple(items)
+                ArenaEntry::Tuple {
+                    items,
+                    holder_count,
+                }
             }
             ArenaEntry::Vector {
                 mut items,
@@ -2281,7 +2293,7 @@ impl<T: ArenaTypes> Arena<T> {
             } => {
                 return entry;
             }
-            ArenaEntry::Vector { items, .. } | ArenaEntry::Tuple(items)
+            ArenaEntry::Vector { items, .. } | ArenaEntry::Tuple { items, .. }
                 if !items.is_empty()
                     && !items
                         .iter()