Skip to content

Fix clol:test for regression learners, and migrate the test suite to rove - #3

Merged
masatoi merged 13 commits into
masterfrom
rls-stream-fix-and-rove-migration
Aug 1, 2026
Merged

masatoi merged 13 commits into
masterfrom
rls-stream-fix-and-rove-migration

Conversation

@masatoi

@masatoi masatoi commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Two pieces of work: a bug fix in src/rls.lisp, and a migration of the test suite from prove to rove.

1. clol:test was unusable for regression learners

clol:test always calls <TYPE>-TEST with both :quiet-p and :stream, but the -TEST
generated by define-regression-learner accepted only :quiet-p:

[UNKNOWN-KEYWORD-ARGUMENT] Unknown &KEY argument: :STREAM
  0: CL-ONLINE-LEARNING:RLS-TEST (src/rls.lisp:85)

So every (clol:test rls-learner data) signalled, and clol-predict could not be used with
RLS or SPARSE-RLS models at all. The classifier -TEST generated by define-learner has
accepted :stream all along — only the regression macro was left behind, which is why this
survived: regression had no test coverage.

The fix accepts :stream and writes one predicted value per test datum, mirroring the
classifier. The raw value is printed rather than its rounded sign, since that is the prediction
for a regression model. (when stream …) guards the write, avoiding the discarded-string
consing the classifier does when stream is nil.

End to end, after the fix:

$ ros roswell/clol-predict.ros t/dataset/a1a rls-a1a.model rls-pred.txt
Loading dataset
RMSE: 0.6596754
$ wc -l < rls-pred.txt
1605

2. prove → rove

t/cl-online-learning.lisp was one flat file of top-level prove is forms. It is now 36
deftest forms carrying 104 ok assertions — the same golden values, plus the new regression
coverage.

Why migrate. cl-mcp's run-tests tool reported ✓ PASS / Passed: 0, Failed: 0 for this
repository even when assertions failed. With one golden value deliberately corrupted, ros
reported × 1 of 96 tests failed while the tool reported PASS. Three layers compose into
that: the tool's :prove branch is an unimplemented stub that falls back to a runner treating
"no condition signalled" as success, and prove-asdf:run-test-system reports failure by return
value rather than by signalling. Filed upstream as cl-ai-project/cl-mcp#131. rove is the one
framework that tool supports end to end, including running a single test.

What changed.

  • cl-online-learning-test.asd depends on rove instead of prove; :defsystem-depends-on (:prove-asdf) and the :test-file component type are gone. The component must be a
    plain (:file …) — rove resolves a system to its suites through component-source-files,
    which only walks asdf:cl-source-file, so :test-file would silently yield zero suites.
  • test-op's :perform now fails on both ways a rove run can be green without being good:
    a red suite (rove reports failure by return value, so the usual idiom would leave
    asdf:test-system exiting 0) and a suite that registered no tests at all. See the CI section
    below for why the second check exists.
  • Datasets load once at top level; each learner is built inside its own deftest. The prove
    file shared learners between the one-vs-rest and one-vs-one sections through top-level
    defvars rebound with setf; that coupling is gone, which is what makes single-test
    execution meaningful.
  • The package shadowing-imports cl-online-learning:test — it name-conflicts with rove's
    re-exported rove/core/result:test under :use.
  • CLAUDE.md updated. Worth knowing: the ros -Q run --eval … one-liner needs
    (sb-ext:disable-debugger) or SBCL enters the debugger when test-op signals, hits EOF on
    closed stdin, and exits 0 — a green exit for a red suite. ./t/run-test.ros is correct
    without that caveat and is now the documented default.

Golden values are unchanged. Every expected weight vector, bias and accuracy tuple was moved,
not retyped. Verified by extracting all 1771 numeric literals from the pre- and post-migration
files: the added multiset equals the deleted multiset, and the 96 original expected s-expressions
parse identically. Any value that had moved would have been a migration bug, not a value to
regenerate.

3. CI

Three changes, each prompted by something this branch's first CI run exposed.

  • fail-fast: false. One broken leg was cancelling the other three before they reported,
    so two genuinely green SBCL runs and one unfinished CCL run all showed as failures.
  • ./t/run-test.ros instead of rove cl-online-learning-test.asd. On ccl-bin the rove CLI
    printed not one assertion and still exited 0 — the same silently-green failure this
    migration exists to escape. Its run-file-tests resolves the .asd with
    (asdf:load-system (pathname-name file)) and no load-asd, and found no suites there. Going
    through test-op instead, CCL now runs all 104 assertions. The :perform guard described
    above would have caught the false green either way; it is kept as the structural backstop.
  • ccl-bin × macOS-latest excluded. Those runners are arm64 and CCL no longer ships a
    Darwin binary Roswell can install (Not supported platform arm64), so the leg failed at the
    Roswell install step before any of this project's code ran.

Remaining matrix: sbcl-bin on ubuntu and macOS, ccl-bin on ubuntu — all three running the
full 104 assertions.

Testing

$ ./t/run-test.ros ; echo $?
...
✓ 1 test completed
Summary:
  All 1 test passed.
0

36 deftests, 104 assertions. The test-op guard was verified against all three outcomes: a
green suite exits 0, a corrupted golden value exits 1, and a suite with no registered deftests
exits 1 with No rove tests are registered for cl-online-learning-test.

New regression coverage — the library previously had none:

  • regression-rls — RMSE, and that :stream emits one line per datum whose values equal
    rls-predict
    . A line-count-only check would pass even if the fix printed the wrong thing.
  • regression-sparse-rls — the same for the sparse variant, which the same bug affected.

Known, deliberately out of scope

The regression -TEST uses (reduce #'+ (mapcar …)), so it rejects a vector test-data that
-TRAIN accepts via etypecase — the classifier -TEST uses count-if and handles both.
(clol:test rls-learner (coerce data 'vector)) signals a TYPE-ERROR. Empty test-data also
divides by zero. Both predate this branch; leaving them for a separate change kept this diff
focused on the keyword-argument bug.

A clean compile emits two style-warnings from src/cl-online-learning.lisp, where dim-of and
n-class-of reference the one-vs-rest / one-vs-one structure accessors before those
defstructs appear. Pre-existing; this branch does not touch that file.

🤖 Generated with Claude Code

masatoi and others added 13 commits August 1, 2026 08:02
CLOL:TEST always calls <TYPE>-TEST with both :quiet-p and :stream, but
the -TEST generated by DEFINE-REGRESSION-LEARNER accepted only :quiet-p.
Every call of (clol:test rls-learner data) therefore signalled
UNKNOWN-KEYWORD-ARGUMENT, which also made clol-predict unusable for RLS
and SPARSE-RLS models.

Accept :stream and write one predicted value per test datum, mirroring
the classifier -TEST. The raw value is printed rather than its rounded
sign, since that is the prediction for a regression model.

Add the first regression assertions to the test suite (94 -> 96).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The remaining prove forms are parked in a block comment and converted
section by section in following commits.

Also adds :shadowing-import-from :cl-online-learning :test to the
cl-online-learning.test package: CL-ONLINE-LEARNING:TEST and rove's
re-exported ROVE/CORE/RESULT:TEST name-conflict under plain :USE
(ANSI 11.1.1.2.5), and every learner section calls (test learner data)
expecting CL-ONLINE-LEARNING:TEST.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Move the sparse perceptron, AROW, SCW-I, LR+SGD and LR+ADAM prove forms out
of the parked block comment into five deftests, following the shape Task 2
established for the dense binary learners. Every weight/bias/accuracy
literal is copied verbatim from the parked prove form it replaces.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Move the ten dense multiclass learner assertions (five one-vs-rest,
five one-vs-one) from the parked prove block into deftests, moving
the #| opener past both sections. Each deftest builds its own
one-vs-rest/one-vs-one wrapper in a let, removing the setf-reuse
coupling the prove forms had between the two sections. Also drops
the now-redundant "Read libsvm dataset (Dence, Multiclass)" is form,
since iris/iris-dim are already header defparameters with their own
deftest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Move the #| opener past both sparse one-vs-rest and one-vs-one
multiclass sections, converting the ten prove is/:test forms into
deftests. Each deftest now builds its own one-vs-rest/one-vs-one
wrapper in a let, removing the prove file's setf-based sharing
between the two sections. Drops the redundant sparse iris dataset
read (already covered by the read-iris-sparse deftest) and the
section's format/defvar scaffolding. Only the RLS regression section
remains parked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CLAUDE.md still described the prove-era test setup after tasks 1-7
migrated the suite to rove. Corrects: the test framework and
run-tests/auto-detection claim, the test invocation command (now needs
sb-ext:disable-debugger to fail correctly on a red suite), the CI
form, the "no single-test runner" claim, the assertion count in
Testing notes, and the stale "RLS has no test coverage" statement
(regression-rls covers it as of adbcd21).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- regression-rls now asserts that :STREAM emits the learner's actual
  predictions (read back and compared to RLS-PREDICT), not just a
  matching line count, which a constant or the pre-fix value would
  also have satisfied.
- Add regression-sparse-rls, mirroring regression-rls for SPARSE-RLS,
  which shares the same :STREAM bug fix and previously had no coverage.
- Wrap src/rls.lisp:41 to fit the 100-column house style.
- Note in ci.yml why `rove cl-online-learning-test.asd` resolves: it
  depends on install-for-ci.sh's ASDF source-registry tree entry, since
  rove's CLI does not call load-asd the way the old prove runner did.
- Lead CLAUDE.md's test command with the portable `./t/run-test.ros`,
  demoting the SBCL-only one-liner (with its disable-debugger caveat)
  to an alternative. Update deftest/assertion counts (36 / 104) to
  match the two new assertions groups added above.
fail-fast cancelled three matrix jobs the moment ccl-bin on macOS-latest
failed to install, so two green SBCL runs and one unfinished CCL run all
reported as failures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rove reports success when it finds no suites at all, so a run that
executed nothing is indistinguishable from a passing one. Observed on
ccl-bin in CI, where `rove cl-online-learning-test.asd' printed not one
assertion and still exited 0 -- the same silently-green failure this
suite was migrated off prove to escape.

test-op's :perform now checks that the system registered rove tests
before running them, and CI invokes ./t/run-test.ros so it goes through
that check instead of calling the rove CLI, which bypasses test-op.

This turns the ccl-bin legs from a false green into an honest red until
the underlying CCL suite-resolution problem is understood.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
macos-latest runners are arm64 and CCL no longer ships a Darwin binary
Roswell can install, so that leg failed at the Roswell install step
before any of this project's code ran.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@masatoi
masatoi merged commit 534a404 into master Aug 1, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant