Release 3.3.0: scratch geometry ceilings, archive peek, and one float32 kernel-selection rule

.gitignore

@@ -6,3 +6,5 @@ out/
 *.ilk
 .vs/
 .idea/
+__pycache__/
+*.pyc

CHANGELOG.md

@@ -9,6 +9,81 @@ per entry. Reconstructed from git history 2026-07-12.
 The format follows [Keep a Changelog](https://keepachangelog.com); this project versions
 by feature tier (minor = new capability, patch = fix), not strict SemVer of a public ABI.
 
+## [3.3.0] — 2026-07-21
+
+### Added
+- **`PeekScratchArchive`** — reads a serialized scratch archive's header and channel
+  table out of a byte span and reports its geometry plus `archiveBytes`, exactly the
+  number of bytes a `Load` consumes, without allocating or reading the payload. It runs
+  the same header validation `Load` does (the two now share one validator, so they cannot
+  drift). A host that appends its own trailer after the archive can therefore locate and
+  validate that trailer *before* committing the load, rather than discovering a broken
+  trailer with the rows already replaced.
+
+### Changed
+- **`ComputePrincipalComponents`'s `scratch` parameter is now `double*`** (was `float*`).
+  The covariance-apply accumulator now matches the mean's accumulation precision — a
+  float accumulator rounded once per row and degraded the power iteration with bank
+  size — and the caller-owned scratch buffer changed type to match. This is
+  source-breaking: every existing caller must change the parameter's declared type from
+  `float*` to `double*` **and** double the size of the buffer it allocates (`dims`
+  doubles, not `dims` floats).
+- **`MeanNNCrossDeviceChannel`/`MaxNNCrossDeviceChannel` can now return `OutOfMemory`.**
+  Both now stage a pre-lift of the target's non-excluded sub-rows into the `Workspace`
+  argument (so each target's self-dot is computed once rather than once per source row,
+  see the self-dot hoist below); if that reservation fails, the call now returns
+  `OutOfMemory`, a status neither function could previously return.
+
+### Fixed
+- **`ScratchBank::Load` now validates the retention region.** A retained row is by
+  construction the post-normalization row the quantizer consumed, so `Load` replays the
+  bake on each retained row and requires it to reproduce the stored row exactly —
+  allocation-free, mirroring the quantizer's own arithmetic. Previously a
+  fabricated-but-finite retained array loaded clean and handed `MeasureScratchRecall` an
+  invented reference to audit against, through the one API whose whole job is an honest
+  number. Non-finite retained values are rejected the same way (`BadFormat`).
+- **Two kernel-selection rules made the segmented scan disagree with the whole-row
+  scan.** The float32 dispatchers `DotF32`/`L2F32` use AVX2+FMA only when the length they
+  are handed is a multiple of 8; `ResolveRowKernels`, which feeds the segmented scan and
+  `DecomposeRowScore`, wired the AVX2 kernels in directly with no such rule. Float32
+  strides are multiples of 4, so at `paddedDims` 20, 36, 52, 100, 132 and similar the two
+  paths computed the same row with different accumulation widths and disagreed in the
+  last ulp. That falsified two published guarantees — that a degenerate one-segment query
+  equals the whole-row scan, and that decomposition contributions sum bit-exactly to the
+  scan's own score, "no second code path exists to drift". `ResolveRowKernels` now holds
+  the dispatchers, so the rule is applied once, to the length actually in hand — which
+  matters because the segmented scan calls the kernel per scan RANGE, and a bank whose
+  stride is a multiple of 8 can still present a range that is not.
+  **`Exactness::CrossDevice` was never affected** (int8 lengths are always multiples of
+  16), so the cross-machine contract is unchanged, and no pinned golden moves: every one
+  of them sits at a width or quantization where the two paths already agreed.
+- **`ScratchBank::Create` and `Grow` performed signed size arithmetic before bounding
+  their geometry.** `ArenaBytes` multiplies capacity by dims in signed `int64`; both
+  entry points accepted any positive `int32` pair, so a large valid-typed request
+  overflowed the arena computation — undefined behavior rather than a refused
+  allocation. Both now apply the format's own ceilings (`kMaxBankRows` rows,
+  `kMaxCrossDeviceDims` dims) before any size is computed, returning `InvalidArgument`.
+  The archive loader already applied these caps; direct construction did not, and
+  `Grow` bounded its request only against the current capacity.
+- **Bank-inspection review follow-ups (lower-severity, from a whole-project pass).**
+  `graph.cpp`: `MutualFilter`/`ConnectedComponents` bound caller-supplied neighbour and
+  duplicate-group values to `[0, count)` before indexing the neighbour list and the
+  union-find scratch, so a malformed hand-built input degrades to a dropped edge rather
+  than an out-of-bounds access. `analytics.cpp`: the int8 off-grid channel guard from
+  `novelty.cpp` now also protects the channel-analytics legs (added once in the shared
+  `ChannelSubRange`), and the channel NN divergence pre-lifts each target's self-dot once
+  instead of once per source row. `pca.cpp`: `ProjectRowsOntoComponents` bounds
+  `componentCount` by `dims` like its sibling. Scale decode goes through
+  `detail::FloatBitsToDouble` on the remaining `compose.cpp`/`pca.cpp`/`analytics.cpp`
+  paths, so a subnormal scale decodes identically under any FTZ/DAZ mode. A comment
+  documents that `novelty.cpp`'s `sampleLimit` is a ceiling, not a down-sampler.
+- **A Cosine channel bank with zero rows was unrepresentable.** `ValidateBank` required
+  the per-channel inverse sub-norm array whenever a Cosine bank carried channels, but
+  those norms are one per row — a bank with no rows requires none, and no scan reads the
+  array at zero rows. The rule now applies only when `count > 0`. The shape is not
+  degenerate: it is what a channel-carrying scratch bank graduates into once every row
+  has been removed, and rejecting it forced that graduation to drop its channels.
+
 ## [3.2.1] — 2026-07-21
 
 ### Fixed
@@ -66,6 +141,26 @@ by feature tier (minor = new capability, patch = fix), not strict SemVer of a pu
   test suite) still carried 3.0.1 through both 3.1 releases; both now read 3.1.2. Found while
   closing an external review's release-identity finding.
 
+## [3.1.1] — 2026-07-18
+
+### Fixed
+- **AVX2 float32 dot/L2 dropped a sub-8 segment remainder.** `DotF32Avx2`/`L2F32Avx2` (and
+  their scalar mirrors) accumulated only in whole 8-lane groups with no remainder; a
+  float32 segment/channel stride lies on the 4-float (16-byte) grid, so a range whose
+  length is congruent to 4 mod 8 had its trailing 4 elements dropped entirely — a
+  length-4 channel scored exactly 0 on AVX2 while SSE/portable/NEON scored it correctly.
+  **This changed numeric results for affected widths**: `ResolveRowKernels` wires the
+  AVX2 kernels in directly for the segmented and per-channel-cosine scan, so any
+  named-channel or segmented query over a sub-8-remainder length (e.g. a length-4
+  channel) on AVX2 hardware returned a different, wrong score before this fix — a
+  consumer bisecting a score change across 3.1.0 to 3.1.2 should attribute the delta at
+  those widths to this fix, not to 3.1.0's `Relabel`. Whole-row scans were unaffected
+  (`DotF32`/`L2F32` route non-multiple-of-8 `paddedDims` to the SSE path). The 4-element
+  remainder is now added to both the intrinsics and their scalar mirrors, bit-identically.
+- CI now requires the AVX2 path on the Linux job (previously accepted SSE or AVX2), so a
+  runner without AVX2 hardware cannot silently skip the new sub-8 remainder coverage;
+  Windows keeps the looser SSE-or-AVX2 check as the general x86 path.
+
 ## [3.1.0] — 2026-07-17
 
 ### Added
@@ -89,6 +184,23 @@ by feature tier (minor = new capability, patch = fix), not strict SemVer of a pu
   (~5.5× cheaper), ~2.1× on float32; promote/demote adds/frees a `capacity × channelCount × 4`
   sub-norm arena (~3.0 MB at 100k rows / 8 channels).
 
+## [3.0.1] — 2026-07-13
+
+### Fixed
+- **`version.h` still read 2.5.0 at the v3.0 tag.** Corrected to 3.0.0, with a coherence
+  test added so a future release cannot ship the same drift.
+- **`MeanNNCrossDeviceChannel`/`MaxNNCrossDeviceChannel` rejected a zero-energy Cosine
+  source and skipped zero-energy targets**, contradicting the documented "a degenerate
+  member floors to a defined 0 in a reduction" contract already honored by
+  `CentroidDistanceCrossDeviceChannel`/`SpreadCrossDeviceChannel`. Both guards are
+  removed so a degenerate channel member floors to 0 through the shared pair score,
+  consistent across all four channel operators.
+- **`MeasureRecallLockedChannel` aborted the whole per-channel recall audit** when a
+  sampled row's channel sub-vector was zero-energy — a valid row whose per-channel
+  self-query is `ZeroNormQuery`. It now excludes that sample, refills the reservoir from
+  the remaining rows, and reports the sample count actually scored rather than the
+  pre-set target.
+
 ## [3.0.0] — 2026-07-12
 
 ### Added

docs/API.md

@@ -413,6 +413,9 @@ payloads carry no per-row sub-norm). Scratch sizing matches the whole-vector
 operators. A channel outside `[0, channelCount)`, or a bank with no channel table, is
 `InvalidArgument`; a degenerate zero-sub-norm channel member floors to a defined `0` in a
 reduction (a single per-channel query on a zero-norm sub-vector is still `ZeroNormQuery`).
+`MeanNNCrossDeviceChannel`/`MaxNNCrossDeviceChannel` additionally pre-lift the target's
+non-excluded sub-rows into `ws` before scoring, so both can also return `OutOfMemory` if
+that reservation fails (v3.3).
 "This mind's identity is drifting but its appearance is stable" is a `CentroidDistanceCross
 DeviceChannel` over the identity channel versus the appearance channel across checkpoints.
 Determinism per channel: [DETERMINISM.md §2e](DETERMINISM.md).
@@ -422,7 +425,7 @@ Determinism per channel: [DETERMINISM.md §2e](DETERMINISM.md).
 ```cpp
 Status ComputePrincipalComponents(const BankView&, int32_t componentCount,
                                   int32_t iterationsPerComponent, float* outMean,
-                                  float* outComponents, float* scratch);
+                                  float* outComponents, double* scratch); // dims doubles
 Status ProjectRowsOntoComponents (const BankView&, const float* mean,
                                   const float* components, int32_t componentCount,
                                   float* outCoords);
@@ -605,6 +608,13 @@ struct ScratchRecallReport {     // v2.3
                                  //   mathematically valid, statistically uninformative
 };
 struct ScratchArchive { bool(*write)(void*, const void*, size_t); bool(*read)(void*, void*, size_t); void* user; };
+struct ScratchArchiveInfo {      // v3.3.0
+    int32_t capacity, count, dims, paddedDims;
+    Metric metric; Quantization quant; bool retainFloats;
+    int32_t channelCount; ChannelInfo channels[kMaxChannels];  // 0 / unused for single-space
+    int64_t archiveBytes;        // exactly what a Load consumes
+};
+Status PeekScratchArchive(const void* data, int64_t bytes, ScratchArchiveInfo* outInfo); // v3.3.0
 ```
 
 A snapshot IS a `BankView` — every query entry point works on it unchanged.
@@ -620,6 +630,26 @@ allocation: everything lives in the arena from `Create` (retention included —
 `ArenaBytes` simply grows). Archive format: [FORMAT.md](FORMAT.md) section 3;
 concurrency guarantees: [DETERMINISM.md](DETERMINISM.md) section 2b.
 
+**Geometry ceilings (v3.3.0):** every `Create` overload and `Grow` bound
+`capacity` at `kMaxBankRows` and `dims` at `kMaxCrossDeviceDims` *before* the
+arena size is computed, returning `InvalidArgument` above either. The arena math
+multiplies capacity by dims in signed `int64`, so an unbounded `int32` pair is
+overflow rather than a refused allocation. These are the immutable format's own
+caps, already applied by the archive loader — a geometry `Create` refuses is one
+no archive could have carried.
+
+**Peeking an archive (v3.3.0):** `PeekScratchArchive` reads a serialized
+archive's header (and channel table) out of a contiguous byte span and reports
+its geometry plus `archiveBytes` — exactly the number of bytes a `Load` will
+consume — without allocating, reading the payload, or touching a bank. It
+applies the same header rules `Load` does, so whatever a peek rejects a load
+rejects, and a span shorter than the archive it declares is `BadFormat`; a
+*longer* span is not, because whatever follows belongs to the caller. That is
+the point: a host appending its own trailer after the archive (channel names,
+say) uses `archiveBytes` to find and validate that trailer *before* committing
+the load, instead of discovering a broken trailer with the rows already
+replaced. Reading the header twice is free; unwinding a load is not.
+
 **Recall audit (v2.3):** the import-time recall-honesty pattern, extended to the
 mutable half. `Create(..., retainFloats = true)` — opt-in, never the default —
 retains the post-normalization float row the quantizer consumed (on Cosine

include/superfaiss/analytics.h

@@ -109,8 +109,13 @@ Status SpreadCrossDevice(
 // inherits it once Tier 1 put the channel table on the snapshot. A bank with no channel
 // table, or `channel` outside [0, channelCount), is InvalidArgument; a zero-sub-norm
 // channel member floors to a defined 0 in a reduction, while a single
-// per-channel query on a zero-norm sub-vector still rejects (ZeroNormQuery). The scratch
+// per-channel query on a zero-norm sub-vector still rejects (ZeroNormQuery). The
 // scratch-buffer sizing matches the whole-vector operators.
+//
+// MeanNNCrossDeviceChannel/MaxNNCrossDeviceChannel additionally stage a pre-lift of the
+// target's non-excluded sub-rows in `ws` (each target's self-dot is computed once, not
+// once per source row), so both can return OutOfMemory if that reservation fails, in
+// addition to the InvalidArgument cases above.
 Status CentroidDistanceCrossDeviceChannel(
 	const BankView& bankA, const int32_t* rowIndicesA, int32_t rowCountA,
 	const int32_t* weightsA, const uint32_t* excludeBitsA,

include/superfaiss/pca.h

@@ -15,20 +15,24 @@ namespace superfaiss
 // Buffers (caller-provided):
 //   outMean        dims floats — the bank mean (int8 rows dequantized).
 //   outComponents  componentCount x dims floats — unit-norm, mutually orthogonal.
-//   scratch        dims floats.
-// A degenerate direction (zero variance after deflation) yields a zero component
-// vector rather than an error; callers render fewer axes.
+//   scratch        dims doubles — the covariance-apply accumulator. Double so it
+//                  matches the mean's accumulation precision; a float accumulator
+//                  rounds once per row and degrades the power iteration with bank size.
+// componentCount must be in [1, dims]. A degenerate direction (zero variance after
+// deflation) yields a zero component vector rather than an error; callers render fewer axes.
 Status ComputePrincipalComponents(
 	const BankView& bank,
 	int32_t componentCount,
 	int32_t iterationsPerComponent,
 	float* outMean,
 	float* outComponents,
-	float* scratch);
+	double* scratch);
 
 // Projects every row onto the components: outCoords is count x componentCount,
 // row-major — coordinate j of row r is (x_r - mean) . component_j. Rows are
-// dequantized exactly as the kernels see them.
+// dequantized exactly as the kernels see them. componentCount must be in [1, dims]:
+// each of the componentCount vectors in `components` is a dims-length row, so an
+// over-large count would read past the buffer (matches ComputePrincipalComponents).
 Status ProjectRowsOntoComponents(
 	const BankView& bank,
 	const float* mean,

include/superfaiss/scratch.h

@@ -59,6 +59,37 @@ struct ScratchRecallReport
 	bool informative = false;  // false when liveRows < kRecallInformativeRows
 };
 
+// What a serialized scratch archive declares about itself, read from its header
+// without loading it (PeekScratchArchive below).
+struct ScratchArchiveInfo
+{
+	int32_t capacity = 0;
+	int32_t count = 0;
+	int32_t dims = 0;
+	int32_t paddedDims = 0;
+	Metric metric = Metric::Dot;
+	Quantization quant = Quantization::Float32;
+	bool retainFloats = false;
+	// The archive's channel table, or count 0 for a single-space archive.
+	int32_t channelCount = 0;
+	ChannelInfo channels[kMaxChannels] = {};
+	// Exactly the number of bytes ScratchBank::Load consumes for this archive.
+	int64_t archiveBytes = 0;
+};
+
+// Reads a scratch archive's header (and channel table) from a contiguous byte span and
+// reports its geometry plus the exact byte length a Load will consume — without
+// allocating an arena, reading the payload, or touching any bank. Applies the same
+// header rules Load does, so a header Load would reject is rejected here with the same
+// status; a span shorter than the archive it declares is BadFormat.
+//
+// This exists for hosts that append their own trailer after the archive (the plugin's
+// channel-name frame is one): archiveBytes locates that trailer so it can be validated
+// BEFORE anything is committed. Without it a host must load first and discover a broken
+// trailer with the rows already replaced, which is degrade-then-notice rather than
+// reject-over-degrade. Reading the header twice is free; unwinding a load is not.
+Status PeekScratchArchive(const void* data, int64_t bytes, ScratchArchiveInfo* outInfo);
+
 class ScratchBank
 {
 public:
@@ -78,6 +109,13 @@ class ScratchBank
 	// Allocates the arena (rows + scales + tombstones + staging) in ONE allocation
 	// through the seam — zero steady-state allocation after Create. Capacity is the
 	// caller's memory budget made explicit. Not valid to call on a created bank.
+	//
+	// Geometry ceilings (every Create overload, and Grow): capacity <= kMaxBankRows and
+	// dims <= kMaxCrossDeviceDims, enforced BEFORE the arena size is computed. The arena
+	// math multiplies capacity by dims in signed int64, so an unbounded pair is overflow
+	// — undefined behavior rather than a refused allocation. Over-cap geometry is
+	// InvalidArgument. These are the immutable format's own caps, so a bank the
+	// constructor refuses is one no archive could carry either.
 	Status Create(
 		int32_t capacity,
 		int32_t dims,

include/superfaiss/version.h

@@ -3,8 +3,8 @@
 #include <cstdint>
 
 #define SUPERFAISS_VERSION_MAJOR 3
-#define SUPERFAISS_VERSION_MINOR 2
-#define SUPERFAISS_VERSION_PATCH 1
+#define SUPERFAISS_VERSION_MINOR 3
+#define SUPERFAISS_VERSION_PATCH 0
 
 namespace superfaiss
 {