Return the epoch train-refine-learner-process says it is returning - #23
Merged
Merged
Conversation
Two things made the returned learner disagree with the accuracy returned beside it (issue #20). The snapshot was taken with clol's defstruct copiers, which are shallow and share the weight arrays, so it tracked the learner that kept training instead of preserving anything. cl-online-learning now gives every learner a real deep copy and takes the shallow one away; this calls clol:copy-learner, which also covers learner types the old etypecase over sparse-arow and one-vs-rest could not see. The snapshot was also taken before each epoch, which is the best epoch only when the loop breaks early. Running out of max-epoch returned the second-to-last epoch beside the last one's accuracy -- at max-epoch 1, an untrained learner reported as though it had learned. Snapshotting after an epoch that improved is right at both exits and copies less often. The three new tests separate the two: with the clol fix alone, the max-epoch one still fails, returning 25.00 (chance) against a reported 97.00. They synthesise a refine dataset rather than building a forest, and carry enough label noise that accuracy peaks on epoch 1 and drops on epoch 2 -- on a separable problem the best and last epoch coincide and none of this is observable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both sides append to t/refinement.lisp and edit the same CLAUDE.md row, so the conflicts are purely additive: keep master's four learner-type-plumbing tests and this branch's three convergence-detection ones, and both :import-from clauses. While there: master's widened "needs no network" bullet had left the following bullet's "Seven of those tests" without an antecedent, and called the refinement tests "four" when this branch's three also match `refine-learner-*`. Both counts now say what they mean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #20.
Requires masatoi/cl-online-learning#10, which adds
clol:copy-learner.Two defects, not one
train-refine-learner-process-innersnapshots the learner and hands the snapshot backwhen accuracy stops improving, alongside the best accuracy it saw. Both halves of that
were wrong.
The snapshot was shallow. It used
clol::copy-sparse-arow/clol::copy-one-vs-rest,which were
defstruct's generated copiers, so it shared the weight arrays — and evenlearners-vector— with the learner that kept training. It tracked its original insteadof preserving anything:
The snapshot was also taken at the wrong time — before each epoch, which is the best
epoch only when the loop breaks early. Running out of
max-epochreturned thesecond-to-last epoch beside the last one's accuracy. At
max-epoch 1that is an untrainedlearner reported as though it had learned: 25.00 (chance) against a reported 97.00.
The fix
Snapshot after an epoch that improved, using
clol:copy-learner. That is correct atboth exits and copies less often than before.
copy-learneralso covers learner types theold
etypecaseoversparse-arowandone-vs-restcould not see.The function now has a docstring stating the contract, including that the argument learner
is left holding the last epoch and the caller wants the returned one.
Tests
Three, in
cl-random-forest-test/refinement. They synthesise a refine dataset directly —make-refine-dataset's output is just a simple-vector of leaf-index vectors and nothingdownstream knows where the indices came from — so they need no network, no forest, and run
in under a second.
They carry enough training-label noise that accuracy peaks on epoch 1 (97.00) and drops on
epoch 2 (96.67). That matters: on a separable problem the best and last epochs coincide
and none of this is observable, so a test written against clean data passes with the bug
still in place.
Each targets a distinct defect. Verified by running them against the intermediate state —
new clol, old cl-random-forest — where the shallow-copy tests pass and the
max-epochonestill fails:
Note on upgrading
SBCL open-codes
defstructcopiers, so this system must be recompiled after updatingcl-online-learning (
asdf:load-system :cl-random-forest :force t). With a stale fasl theinlined shallow copy survives and the bug looks unfixed. Recorded in CLAUDE.md.
Existing accuracy tests still pass with the corrected rollback: a9a 81.2051 and letter
97.1440, both inside their ±1.0 windows. Full suite green on x86-64 SBCL, 7 systems, 0
failures.
🤖 Generated with Claude Code