Skip to content

Cover the exported API the golden-value tests never reach - #4

Merged
masatoi merged 5 commits into
masterfrom
test-coverage-for-untested-api
Aug 1, 2026
Merged

masatoi merged 5 commits into
masterfrom
test-coverage-for-untested-api

Conversation

@masatoi

@masatoi masatoi commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Test-only. No production code changes — git diff master..HEAD --name-only is
t/cl-online-learning.lisp alone.

What was uncovered

The suite is a golden-value regression test: it asserts exact learned weight vectors, biases
and accuracies. That is thorough about the learning, and blind to everything around it. An
sb-profile run over the whole exported API during a suite run found 16 exported functions
the suite never executed
.

This covers five groups of them. 36 deftests / 104 assertions → 54 / 178; runtime stays
under a second on a warm cache.

1. clol:save / clol:restore

Serialization had no test at all, and it is the most fragile thing here.
one-vs-rest and one-vs-one cache function objects in struct slots that cl-store cannot
serialize, so save nulls them, stores, then re-resolves, and restore re-resolves after
loading. Add a slot holding a function and forget the matching pair of helpers, and the model
saves and loads fine — it just cannot train any more.

So every round-trip test trains the restored learner, not merely tests it. Six tests: dense
binary, sparse binary, one-vs-rest, one-vs-one, sparse multiclass, and RLS.

Verified the guard bites: stubbing out restore's re-resolution turns the three multiclass
tests red and leaves the binary and regression ones green.

2. dim-of / n-class-of / sparse-learner?

clol-predict calls these three on a restored model to decide how to read the test file —
n-class-of > 2 selects multiclass label handling, sparse-learner? selects the sparse
reader, dim-of gives the width. Get one wrong and the tool silently reads the dataset the
wrong way. None was executed.

Hardcoding n-class-of to 2 turns two assertions red.

3. The classifier :stream path

The previous PR covered :stream for regression. The classifier path — what clol-predict
actually emits — was still untested, and it is the far more common use. The tests pin that a
binary model emits the rounded sign and a multiclass one emits class indices 0..K-1, not
the original LIBSVM labels (iris.scale is labelled 1..3 and read-data subtracts one). That
off-by-one is deliberate and now guarded.

Printing a constant instead of the rounded prediction turns three assertions red.

4. clol.utils

to-int, to-float, class-min/max, shuffle-vector — nothing in the library calls them;
they exist for the CLI, which parses every option as a string. to-float's tests assert the
result is a single-float, since a double would break the type declarations the update bodies
compile under. shuffle-vector is randomized, so it is asserted on by length, multiset
equality and in-place identity, never on a specific permutation.

5. The roswell scripts

clol-train and clol-predict are the library's only user-facing programs and the only place
defmain's option parsing runs. Driven as subprocesses; a pair of runs costs about a second.
They skip when ros is not on PATH rather than failing for an environmental reason —
verified by forcing that path, which leaves the suite green with two skips.

Getting the assertions right here took two rounds of review:

  • First attempt asserted (probe-file model) and (zerop exit-code). Both are vacuous.
    uiop:with-temporary-file creates its file before the subprocess runs, and defmain wraps
    every script body in a handler-case that prints the condition and returns normally — so a
    failing script still exits 0. Dropping clol-train's save call was caught only indirectly,
    by a downstream line-count mismatch surfacing as an opaque error.
  • Second attempt asserted stderr was empty. That works, but couples the tests to every
    compile-time style-warning the subprocess emits — and CI runs them under CCL as well as SBCL.
  • What landed: both of defmain's handlers print the usage text to standard output, and a
    successful run never does, so that is the marker. stderr is still checked for defmain's own
    "Error:" prefix, which surfaces the swallowed condition in the failure output. Model file
    size replaces probe-file.

Dropping clol-train's save call now turns eight assertions red, each naming its cause.
Injecting a deliberate style-warning into the script leaves the suite green, where the previous
version went red.

How each test was validated

Every group was checked by breaking the production code it covers and confirming the suite goes
red, then restoring. A test that cannot fail is worth nothing, and for coverage added to code
that already works, deliberately breaking it is the only way to know:

break red
restore stops re-resolving function slots 3 multiclass round-trip tests
n-class-of hardcoded to 2 2 assertions
classifier :stream prints a constant 3 assertions
clol-train stops calling save 8 assertions

Still uncovered, deliberately

clol.vector:ds-v/, ds2s-v*, s-v*n, make-sparse-vector and sparse-vector-length are
exported but reached from nowhere — not src/, not roswell/, not the tests. ds-v/ and
ds2s-v* appear exactly twice in the tree: their defun and the export list. They compute
correctly when called by hand, so this is a question of whether they are public API worth
testing or leftovers worth unexporting — a design call, not a coverage gap, so it is left for
you.

clol-train also cannot produce a regression or logistic-regression model: its -type switch
only offers Perceptron, AROW and SCW-I. Worth knowing given the previous PR fixed
clol-predict for RLS models — those can currently only be produced from a REPL.

🤖 Generated with Claude Code

masatoi and others added 5 commits August 1, 2026 09:10
…alize

ONE-VS-REST and ONE-VS-ONE cache function objects in struct slots that
cl-store cannot serialize, so SAVE nulls them and RESTORE re-resolves
them. Nothing tested that. Each round-trip test trains the restored
learner rather than only testing it, since a learner that restores but
cannot train is exactly what a missed slot produces.

Verified the guard bites: stubbing out RESTORE's re-resolution turns the
three multiclass tests red and leaves the binary and regression ones
green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DIM-OF, N-CLASS-OF and SPARSE-LEARNER? are how CLOL-PREDICT decides to
read a test file; none was executed by the suite. Nor was :STREAM for
classifiers -- REGRESSION-RLS covers it only for regression, while the
classifier path is what CLOL-PREDICT actually emits.

Verified both guards bite: hardcoding N-CLASS-OF to 2 turns two
assertions red, and printing a constant instead of the rounded
prediction turns three red.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing in the library calls to-int, to-float, class-min/max or
shuffle-vector -- they exist for the CLI, which parses every option as a
string. clol-train and clol-predict themselves were untested, and are
the only place defmain's option parsing runs; each pair of subprocess
runs costs about a second.

The CLI tests skip when ros is not on PATH rather than failing for an
environmental reason. Verified: forcing roswell-available-p to nil
leaves the suite green with two skips, and dropping clol-train's save
call turns two assertions red.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review caught two vacuous assertions in the roswell script tests.
UIOP:WITH-TEMPORARY-FILE creates its file before the subprocess runs, so
(probe-file model) was always true. And DEFMAIN wraps every script body
in a HANDLER-CASE that prints the condition and returns normally, so a
failing script still exits 0 and (zerop code) was always true too.

The scripts' only real failure signal is what they print, so assert
stderr is empty and the model file is non-empty. Dropping clol-train's
save call now turns eight assertions red, each naming its cause, where
before it turned two red only via a downstream line-count mismatch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Re-review noted that asserting stderr is empty couples these tests to
every compile-time style-warning the subprocess emits -- a warning
anywhere in reachable source would turn them red for an unrelated
reason, and CI runs them under CCL as well as SBCL.

Both of DEFMAIN's handlers print the usage text to standard output and a
successful run never does, so that is the marker. stderr is still
checked for DEFMAIN's own Error: prefix, which surfaces the swallowed
condition in the failure output.

Verified both directions: dropping clol-train's save call still turns
eight assertions red, and injecting a deliberate style-warning into the
script leaves the suite green where the old assertion went red.

Also made file-size return 0 for a missing file rather than signalling.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@masatoi
masatoi merged commit c4dfad0 into master Aug 1, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant