fix: avoid index cache capacity fragmentation

[kamronis]

fix(cache): avoid index cache capacity fragmentation

Problem

Increasing the session index cache from 7.5 to 8 GiB can introduce repeated
partition loading even when the index working set fits within both capacities.
With sufficient CPUs, the existing policy switches from one shard to two at
8 GiB. Quick cache shards have independent weight limits and cannot borrow
unused capacity from each other. A few large partition entries can overload
one shard while leaving space in the other.

This was observed with a synthetic 10M-row, 128-dimensional IVF_HNSW_FLAT index
containing nine partitions and an approximately 6.808 GiB cached working set.
Large partitions are relevant beyond this dataset: the current HNSW defaults
target approximately one million vectors per partition.

Change

Use a single Quick shard for automatically constructed session index caches,
giving all entries access to one shared weight budget. Apply this consistently
through Session::new, Session::default, and the default/size variants of
Session::with_cache_backends.

Metadata caches and QuickCacheBackend::with_capacity retain recommended
sharding. Caller-provided backends remain unchanged. The explicit
QuickCacheShardPolicy option is additive; there are no persistent-format or
existing public API signature changes. The configured capacity remains a
weighted-entry budget, not a process RSS limit, and Quick's existing admission
limits still apply.

Validation

The local workload reused the same saved synthetic index and queries with
L2, k=10, nprobes=9, and ef=320. Each residency run used a fresh process,
100 warmup calls, and 100 measured queries. A run without reloads means zero
partition loads during its measured phase. The p95 range below is across
individual processes.

Configuration	Runs without reloads	Per-process p95 range
Previous Quick, 7.5 GiB	5/5	7.75–11.05 ms
Previous Quick, 8 GiB	2/5	11.02–1072.62 ms
Single-shard Quick, 7.5 GiB	20/20	7.16–11.33 ms
Single-shard Quick, 8 GiB	20/20	9.64–11.11 ms

The previous-policy results are historical measurements, not a new interleaved
comparison against a rebuilt baseline. Moka retained the working set in all
20 runs at each capacity. The insufficient-capacity 6 GiB Quick control
continued reloading partitions in all five runs and stayed within its budget.

Tests

Regression tests cover deterministic shard imbalance with unequal declared
weights, direct insertion, capacity/admission limits, shard-count boundaries,
loader failure/retry and cancellation, and automatic session construction.

Local checks passed on the implementation used for the experiments:

• cargo test -p lance-core cache::quick::tests --lib: 9 passed
• cargo test -p lance session::tests --lib: 11 passed
• cargo fmt --all -- --check
• cargo clippy --all --tests --benches -- -D warnings

[lance-gatekeeper]

✅ Gate recommendation: approve.

Another merge from main leaves the reviewed cache patch unchanged. Automatic index caches continue to use one shared weight budget while metadata caches and caller-provided backends preserve their existing sharding behavior; the focused Quick cache and Session suites pass on the current beta.4 head.

[wjones127]

It seems unfortunate we would have to make the user think about configuring this. Would this be better solved by breaking up the HNSW cache entries into smaller pieces?

[kamronis]

@wjones127

It seems unfortunate we would have to make the user think about configuring this. Would this be better solved by breaking up the HNSW cache entries into smaller pieces?

Agreed, users shouldn’t need to configure this. This PR already applies the policy automatically to session index caches; the explicit policy option is only for custom backend construction.
Smaller HNSW entries would help too, though we currently cache the whole graph and vector storage per partition, so that’s a larger change.
I suggest keeping this fix for the capacity regression and do smaller entries in another PR.