Add a packed inference representation for trained forests

CLAUDE.md

@@ -59,8 +59,9 @@ Tests are split into feature systems, each of which can be run on its own:
 | `cl-random-forest-test/parallel` | parallelized training accuracy (4, SBCL only) |
 | `cl-random-forest-test/regression` | univariate regression behaviour (5) |
 | `cl-random-forest-test/pruning` | global pruning behaviour (5) |
+| `cl-random-forest-test/packed` | packed inference representation (11) |
 
-`cl-random-forest-test` is the aggregate that runs all seven.
+`cl-random-forest-test` is the aggregate that runs all eight.
 `cl-random-forest-test/fixture` holds the shared dataset loaders and helpers and has no tests.
 
 Load the feature system first (`ql:quickload` or `asdf:load-system`), then:
@@ -71,14 +72,21 @@ Load the feature system first (`ql:quickload` or `asdf:load-system`), then:
 ```
 
 There is no lint step. CI (`.github/workflows/ci.yml`) runs the matrix
-{sbcl-bin, ccl-bin} × {ubuntu-latest, macOS-latest}.
+{sbcl-bin, ccl-bin} × {ubuntu-latest, macOS-latest}, less ccl-bin on macOS, which the
+workflow excludes because macOS CCL binaries are no longer distributed — three jobs, not
+four. Roswell has no ccl-bin for **ARM64 Linux** either, so on an aarch64 development
+machine the CCL half of the matrix cannot be reproduced locally at all and CI is the only
+place it runs. Treat a green local suite as evidence about SBCL only: CCL does not check
+type declarations, and its `integer-decode-float` normalises a denormal's significand where
+SBCL leaves it alone — both have produced CI-only failures in this repository.
 
 Test/example caveats:
 - `cl-random-forest-test/regression`, `.../pruning`, and the seven synthetic tests in
   `.../refinement` (`refine-learner-default-path-unchanged`, three `refine-learner-of-type-*`
   and three `refine-learner-process-*`) need no network: they use
   `cl-random-forest-test/fixture`'s deterministic synthetic data, or build their own.
   Everything else downloads datasets.
+- `cl-random-forest-test/packed` uses the fixture's synthetic data and needs no network.
 - Of the regression and pruning suites' ten tests, seven are **property assertions**, not
   pinned accuracy numbers, and three deliberately pin bugs that are still open: `regression-refine-learner-default-gamma-diverges`
   (issue #16), `pruning-strands-leaves-without-sample-indices` (issue #14) and
@@ -185,6 +193,49 @@ refine learner before training again.
 the kernel to `nil` restores serial execution with no recompilation. Parallelized:
 `make-forest`, `make-regression-forest`, `make-refine-dataset`, `train-refine-learner`.
 
+### Packed inference
+
+`src/packed/` is a separate system, `cl-random-forest/src/packed`, that the core does not
+depend on and the facade does not re-export. Load it explicitly:
+
+```lisp
+(ql:quickload :cl-random-forest/src/packed)
+```
+
+It flattens a trained forest into arrays for inference: 5-8x faster than `predict-forest`,
+reentrant where `predict-forest` is not, and serialisable. It is a derived read-only view --
+training, pruning, feature importance and reconstruction all keep using the `node` structs.
+
+Two things to know. A packed model is a snapshot: prune the forest and the packed copy goes
+on predicting with the old structure, silently. And `build-packed-topology` refuses a forest
+that has been *pruned*: `pruning!`'s `delete-children!` turns an already-split node -- whose
+sample indices were nil'd when it was split -- back into a leaf without restoring them, so
+that leaf's class distribution would come out uniform rather than signalling (issue #14). A
+default-built, unpruned forest packs fine regardless of what `:remove-sample-indices?` was
+given at construction time; `t/packed.lisp` asserts this directly.
+
+The whole walk runs at `(safety 0)`, so **every index it will follow is checked before it
+starts** -- in `check-packable` when building, and again in `packed-load` for a file the
+process did not write. Three of those checks are not obvious:
+
+- `left`/`right` must not merely be in range but must *exceed their own node index*.
+  `build-packed-topology` numbers a subtree after its root, so this holds by construction;
+  it is what makes the walk provably terminate, and without it `left[i] = i` is an in-range
+  cycle that hangs prediction with no way to recover.
+- `feature` is bounded by `datum-dim`, the training width, which is a slot on the topology
+  and a field in the file for exactly this reason. `datum-dim` is itself untrusted in a
+  file, so it is paired with `check-datamatrix-width`, which bounds it by the matrix
+  actually passed in. Composed, `feature < datum-dim <= the real column count`.
+- Each array's declared length is checked against the bytes left in the file *before* it is
+  allocated, so an inflated count is an error rather than a multi-gigabyte allocation.
+
+`check-datamatrix-width` also catches the ordinary version of that mistake -- predicting
+with a narrower dataset than the forest was trained on -- which is otherwise a silent read
+past the end of a row. It costs nothing measurable: 39-41k predictions/s on a 500-tree
+depth-10 forest either way, a spread smaller than the run-to-run noise.
+
+The design and the measurements behind it are in `docs/packed-forest-layout.md`.
+
 ## Known broken code
 
 `src/feature-importance.lisp` and `src/reconstruction.lisp` `:use` only the *exported* symbols of

README.org

@@ -182,6 +182,32 @@ The following figure shows the accuracy for test dataset and the number of leaf
 
 [[./docs/img/clrf-mnist-pruning.png]]
 
+**** Packed inference
+
+For serving a trained model, =cl-random-forest/src/packed= flattens a forest into arrays:
+5-8x faster than =predict-forest=, safe to call from several threads, and serialisable.
+
+A forest you intend to prune /and/ pack must be built with =:remove-sample-indices? nil=:
+pruning turns internal nodes back into leaves without restoring their sample indices (issue
+#14), and the packed builder refuses those. =*forest*= above was built with the default
+=:remove-sample-indices? t= and was then pruned in the Global Pruning section, so packing it
+as shown below will signal a =packed-build-error= unless it is rebuilt with
+=:remove-sample-indices? nil= first.
+
+#+BEGIN_SRC lisp
+(ql:quickload :cl-random-forest/src/packed)
+
+(defparameter *packed* (clrf.packed:build-packed-classifier *forest*))
+(defparameter *acc* (clrf.packed:make-packed-accumulator *packed*))
+
+(clrf.packed:packed-predict *packed* *datamatrix* 0 *acc*)
+
+(clrf.packed:packed-save *packed* #p"model.packed")
+(defparameter *reloaded* (clrf.packed:packed-load #p"model.packed"))
+#+END_SRC
+
+One accumulator per thread. The packed model is a snapshot of the forest at build time.
+
 **** Parallelization
 The following several functions can be parallelized with [[https://lparallel.org/][lparallel]].
 

cl-random-forest-test.asd

@@ -65,6 +65,14 @@
              (unless (symbol-call :rove :run c)
                (error "Tests failed."))))
 
+(defsystem "cl-random-forest-test/packed"
+  :description "Tests for the packed inference representation"
+  :depends-on ("rove" "cl-random-forest-test/fixture" "cl-random-forest/src/packed")
+  :components ((:module "t" :components ((:file "packed"))))
+  :perform (test-op (o c) (declare (ignore o))
+             (unless (symbol-call :rove :run c)
+               (error "Tests failed."))))
+
 (defsystem "cl-random-forest-test"
   :author "Satoshi Imai"
   :license "MIT Licence"
@@ -76,7 +84,8 @@
                "cl-random-forest-test/refinement"
                "cl-random-forest-test/parallel"
                "cl-random-forest-test/regression"
-               "cl-random-forest-test/pruning")
+               "cl-random-forest-test/pruning"
+               "cl-random-forest-test/packed")
   ;; NOTE: This system has no components of its own, so rove's SYSTEM-SUITES would
   ;; return an empty list and (rove:run c) would silently report zero tests.
   ;; The feature systems must be listed explicitly.
@@ -89,5 +98,6 @@
                                     "cl-random-forest-test/refinement"
                                     "cl-random-forest-test/parallel"
                                     "cl-random-forest-test/regression"
-                                    "cl-random-forest-test/pruning"))
+                                    "cl-random-forest-test/pruning"
+                                    "cl-random-forest-test/packed"))
                (error "Tests failed."))))

cl-random-forest.asd

@@ -2,6 +2,18 @@
   (:use :cl :asdf))
 (in-package :cl-random-forest-asd)
 
+;; SVMFORMAT is a nickname of the CL-LIBSVM-FORMAT package, and package-inferred-system
+;; derives a system name by downcasing the package name -- which would look for a
+;; nonexistent "svmformat" system. Registering the mapping is what lets src/utils.lisp
+;; declare its dependency on the package it actually uses.
+(asdf:register-system-packages "cl-libsvm-format" '(#:svmformat))
+
+;; CLOL and CLOL.VECTOR are nicknames of CL-ONLINE-LEARNING and CL-ONLINE-LEARNING.VECTOR
+;; respectively, both provided by the single CL-ONLINE-LEARNING system. Same reasoning as
+;; the CL-LIBSVM-FORMAT mapping above: without this, src/random-forest.lisp's :import-from
+;; of these nicknames would send ASDF looking for nonexistent "clol"/"clol.vector" systems.
+(asdf:register-system-packages "cl-online-learning" '(#:clol #:clol.vector))
+
 (defsystem cl-random-forest
   :version "0.2"
   :author "Satoshi Imai"

src/packed.lisp

@@ -0,0 +1,8 @@
+(uiop:define-package :cl-random-forest/src/packed
+    (:use :cl)
+  (:nicknames :clrf.packed)
+  (:use-reexport :cl-random-forest/src/packed/topology)
+  (:use-reexport :cl-random-forest/src/packed/classifier)
+  (:use-reexport :cl-random-forest/src/packed/io))
+
+(in-package :cl-random-forest/src/packed)