perf(frag-reuse): hash the remap's fragment-id map with an integer hasher

[amunra]

Problem

FragReuseIndex::remap_row_id translates a stored row address through every fragment-reuse version in the chain. Each CompactRemapStep probes frags: HashMap<u32, OldFragmentRemap> once per address. The map is keyed by a u32 fragment id and was on Rust's default SipHash, so consolidating indexes over hundreds of millions of rows and tens of reuse versions performs billions of cryptographic hashes of a 4-byte, internally generated key, and the hash sits directly on the critical path to the bucket load.

Profiling on a two-probe variant of this module, where every lookup hashed twice, put ~35% of btree consolidation CPU in hash_one plus sip::Hasher::write. The module here makes one probe per step, so the saving is expected to be smaller, but the hash is still the first thing every lookup does.

Change

A module-private IntHasher: multiply by the golden-ratio constant, then fold the high half into the low half in finish. It is exposed as type IntMap<K, V> = HashMap<K, V, BuildHasherDefault<IntHasher>> and used for CompactRemapStep::frags and the build-time per_frag map in GroupRemap::new_with_old_frags. The duplicate-fragment HashSet and the RemapStep::Direct(HashMap<u64, Option<u64>>) map stay on the default hasher. No new runtime dependency.

The fold matters. Multiplication only carries upward, so without it the low bits of the hash, which hashbrown uses to pick the bucket, would ignore the high bits of the key, and fragment ids strided by a power of two would all land in one bucket. On a 2048-bucket table with 1024 keys strided by 2^20 that is 1 distinct bucket without the fold and about 800 with it.

Dropping SipHash is safe here because the keys are fragment ids the system generates itself. They are never attacker-supplied, so the hash-flooding resistance SipHash buys is worth nothing on this path.

Known limitation: write_u32/write_u64/write_usize replace the hasher state rather than mixing into it, so a composite integer key would collide on its last field alone. That is correct but slow (Eq still decides), so the type is documented as single-integer-key only, kept module-private, and the behaviour is pinned by a test so a future key-type change shows up as a deliberate decision.

DeepSizeOf change

impl<K, V> DeepSizeOf for HashMap<K, V> is widened to impl<K, V, S>, because CompactRemapStep::deep_size_of_children calls self.frags.deep_size_of_children and would otherwise stop compiling. The impl body never touched S, so the widened form is the more accurate one.

DeepSizeOf is lance-core's own trait and deepsize.rs holds the only DeepSizeOf for HashMap impl, so the widening is internal to the trait and has no downstream impact beyond type inference in the rare case that S is not already fixed.

One side effect: BuildHasherDefault is a ZST where RandomState is 16 bytes, so size_of::<HashMap<..>>() drops from 48 to 32 and the reported deep size of a remap shrinks by 16 bytes per map. That moves cache accounting toward accuracy, not away from it.

Public API

One existing impl is widened, rust/lance-core/src/deepsize.rs:

before	after
impl<K: DeepSizeOf, V: DeepSizeOf> DeepSizeOf for HashMap<K, V>	impl<K: DeepSizeOf, V: DeepSizeOf, S> DeepSizeOf for HashMap<K, V, S>

Needed because the map this PR re-hashes no longer uses the default hasher. DeepSizeOf is this crate's own trait and this is the only impl of it for HashMap, so the orphan rule leaves no downstream impl that could conflict; see the section below. The new IntHasher and IntMap are module-private.

Numbers

Synthetic dataset, 500M rows, 26 index segments, a fragment-reuse chain of 26 versions covering all rows, cold cache, measured on the two-probe variant of this module with both sides otherwise identical:

	before	after
btree consolidation	154.0s	55.8s	2.8x
bitmap consolidation	152.6s	50.4s	3.0x

Identical end state in both runs (26 segments -> 1, reuse chain drained), RSS unchanged. A control run with no reuse chain puts the floor at 32.6s, so this recovered roughly 80% of the remap overhead on that variant.

Tests

A wrong row-address translation does not crash; it silently points an index at the wrong physical row. So the second commit adds a proptest that compares the compact remap against a materialized old-to-new map address by address over randomized rewrites, with fragment ids drawn from dense, power-of-two-strided and sparse shapes. It is mutation-checked: perturbing the rank arithmetic in CompactRemapStep::get fails it.

Alongside it: the fold in finish (power-of-two-strided fragment ids must spread over buckets), seedless per-key determinism (a key must stay reachable across a resize), the composite-key collision above, and insertion-order independence of every set-shaped output. proptest is added back to lance-core's dev-dependencies for the randomized check; it was there at v10.0.0.

Validation: cargo fmt --all -- --check, cargo clippy --all --tests --benches -- -D warnings, cargo test -p lance-core, cargo test -p lance-table, all clean.

Compatibility

No file-format or on-disk change. The hasher only affects in-memory lookup. Iteration order of the affected maps changes; the set-shaped outputs are order-independent (covered by the insertion-order test), and the only place the order shows is the fragment-id list inside one invalid_input error message.

---

Tracking: Ported from rerun-io#60.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The change removes a measured hashing bottleneck from the per-row/per-version remap path without changing persisted data or row-address semantics. Keeping the hasher module-private and restricted to single-u32 keys is a proportionate solution: a dense table would waste memory for sparse fragment IDs, while a general-purpose dependency would add cost without a stronger contract. The randomized equivalence, distribution, ordering, deep-size, and fragment-reuse integration coverage support the change.