Skip to content

Return the epoch train-refine-learner-process says it is returning - #23

Merged
masatoi merged 2 commits into
masterfrom
fix-refine-process-rollback
Aug 6, 2026
Merged

masatoi merged 2 commits into
masterfrom
fix-refine-process-rollback

Conversation

@masatoi

@masatoi masatoi commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Fixes #20.

Requires masatoi/cl-online-learning#10, which adds clol:copy-learner.

Two defects, not one

train-refine-learner-process-inner snapshots the learner and hands the snapshot back
when accuracy stops improving, alongside the best accuracy it saw. Both halves of that
were wrong.

The snapshot was shallow. It used clol::copy-sparse-arow / clol::copy-one-vs-rest,
which were defstruct's generated copiers, so it shared the weight arrays — and even
learners-vector — with the learner that kept training. It tracked its original instead
of preserving anything:

ALIASED-WEIGHT-ARRAY T
ALIASED-LEARNERS-VECTOR T
SNAPSHOT-MUTATED-BY-TRAINING T (0.0 -> -0.018511506)
REPORTED-MAX 90.8333  RETURNED-LEARNER-ACTUAL 90.6667  MATCH NIL

The snapshot was also taken at the wrong time — before each epoch, which is the best
epoch only when the loop breaks early. Running out of max-epoch returned the
second-to-last epoch beside the last one's accuracy. At max-epoch 1 that is an untrained
learner reported as though it had learned: 25.00 (chance) against a reported 97.00.

The fix

Snapshot after an epoch that improved, using clol:copy-learner. That is correct at
both exits and copies less often than before. copy-learner also covers learner types the
old etypecase over sparse-arow and one-vs-rest could not see.

The function now has a docstring stating the contract, including that the argument learner
is left holding the last epoch and the caller wants the returned one.

Tests

Three, in cl-random-forest-test/refinement. They synthesise a refine dataset directly —
make-refine-dataset's output is just a simple-vector of leaf-index vectors and nothing
downstream knows where the indices came from — so they need no network, no forest, and run
in under a second.

They carry enough training-label noise that accuracy peaks on epoch 1 (97.00) and drops on
epoch 2 (96.67). That matters: on a separable problem the best and last epochs coincide
and none of this is observable, so a test written against clean data passes with the bug
still in place.

Each targets a distinct defect. Verified by running them against the intermediate state —
new clol, old cl-random-forest — where the shallow-copy tests pass and the max-epoch one
still fails:

✓ the best epoch 97.0000 really did beat the last 96.6667
✓ returned learner scores 97.0000, process reported 97.0000
× after one epoch, returned 25.0000 against reported 97.0000
✓ five more epochs on the original left the returned learner untouched

Note on upgrading

SBCL open-codes defstruct copiers, so this system must be recompiled after updating
cl-online-learning (asdf:load-system :cl-random-forest :force t). With a stale fasl the
inlined shallow copy survives and the bug looks unfixed. Recorded in CLAUDE.md.

Existing accuracy tests still pass with the corrected rollback: a9a 81.2051 and letter
97.1440, both inside their ±1.0 windows. Full suite green on x86-64 SBCL, 7 systems, 0
failures.

🤖 Generated with Claude Code

masatoi and others added 2 commits August 6, 2026 14:32
Two things made the returned learner disagree with the accuracy returned beside it
(issue #20).

The snapshot was taken with clol's defstruct copiers, which are shallow and share the
weight arrays, so it tracked the learner that kept training instead of preserving
anything. cl-online-learning now gives every learner a real deep copy and takes the
shallow one away; this calls clol:copy-learner, which also covers learner types the old
etypecase over sparse-arow and one-vs-rest could not see.

The snapshot was also taken before each epoch, which is the best epoch only when the loop
breaks early. Running out of max-epoch returned the second-to-last epoch beside the last
one's accuracy -- at max-epoch 1, an untrained learner reported as though it had learned.
Snapshotting after an epoch that improved is right at both exits and copies less often.

The three new tests separate the two: with the clol fix alone, the max-epoch one still
fails, returning 25.00 (chance) against a reported 97.00. They synthesise a refine dataset
rather than building a forest, and carry enough label noise that accuracy peaks on epoch 1
and drops on epoch 2 -- on a separable problem the best and last epoch coincide and none
of this is observable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both sides append to t/refinement.lisp and edit the same CLAUDE.md row, so the conflicts
are purely additive: keep master's four learner-type-plumbing tests and this branch's
three convergence-detection ones, and both :import-from clauses.

While there: master's widened "needs no network" bullet had left the following bullet's
"Seven of those tests" without an antecedent, and called the refinement tests "four" when
this branch's three also match `refine-learner-*`. Both counts now say what they mean.
@masatoi
masatoi merged commit b90f7c2 into master Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

train-refine-learner-process never rolls back to the best epoch

1 participant