Add FTRL-Proximal logistic regression (lr+ftrl / sparse-lr+ftrl)

[masatoi]

Adds lr+ftrl and sparse-lr+ftrl — logistic regression trained by FTRL-Proximal. This
is the first learner in the library that drives weights to exactly zero, so a model
can be shrunk by dropping features rather than only accepting sparse input.

Measured on t/dataset/a1a over ten passes: lambda1 0 leaves 113 of 123 weights
non-zero, lambda1 10 leaves 70 — 43% of the model exactly zero — at a cost of 0.56
accuracy points (84.80% → 84.24%).

The algorithm

McMahan, Holt, Sculley et al., Ad Click Prediction: a View from the Trenches, KDD 2013,
Algorithm 1. Per coordinate the state is (z_i, n_i) and the weight is derived, not
stored:

w_i = 0                                                        if |z_i| <= lambda1
w_i = -(z_i - sgn(z_i) lambda1) / ((beta + sqrt(n_i))/alpha + lambda2)  otherwise

API

(clol:make-lr+ftrl        input-dimension alpha beta lambda1 lambda2)
(clol:make-sparse-lr+ftrl input-dimension alpha beta lambda1 lambda2)

All positional, like every other learner — which is what lets make-one-vs-rest and
make-one-vs-one apply their &rest learner-params with no special case.

The one design decision worth reviewing

define-learner's generated -PREDICT reads a weight slot, but FTRL has no weight
vector to give it. The derived weight is materialized into a weight slot as a cache.
That is exact because w_i is a pure function of (z_i, n_i), which change only for the
coordinates an update touches — but only if the cache is refreshed at the END of the
update. Refreshing at the start leaves each coordinate one update stale: measured at
89 of 123 coordinates wrong, with accuracy indistinguishable from correct either way.

lr+ftrl-weight-cache-invariant is the test that catches this, and it is the only thing
that can — no accuracy assertion would.

What this bought

The whole branch is additive: 508 insertions, 0 deletions. Because FTRL presents as an
ordinary binary learner with weight and bias, nothing existing needed to change —
dim-of, n-class-of, sparse-learner?, save, restore, one-vs-rest and
one-vs-one all work untouched. Tests check that rather than assume it:
make-one-vs-rest with sparse-lr+ftrl reaches 87.33% on iris.scale.

The sparse update is genuinely O(nnz). Note for contrast that sparse-lr+adam's update is
sparse only in its gradient — its moment updates sweep the full dimension. That is
pre-existing and out of scope here, but is not something this learner copies.

No scratch vectors: the update is coordinate-local, so storage is weight, z, n and
nothing else.

Verification

Golden values cannot validate the code they came from, so independent grounds came first:

1. Two independent reference transcriptions of Algorithm 1 were written — one during
   implementation, one during review — each recomputing w from (z, n) with no cache
   and using the paper's 0/1-label gradient (p−y)x instead of this repo's ±1 form. Both
   were compared against the implementation over a full a1a pass at three parameter
   settings: max |Δw| ≈ 2.5e-7 to 3.6e-7. These exercise the sigma_i·w_i term and the L1
   branch, which the all-zero first update cannot reach.
2. Dense and sparse are bit-identical on a1a — equalp on weight, z and n after
   both 1 and 10 passes.
3. ftrl-weight-of-known-values pins the weight derivation against four hand-derived
   literals, so the two invariant tests above are no longer circular.
4. An accuracy floor of 80%, against a1a's 75.39% always-answer-negative baseline.

Suite: 67 → 82 deftest forms, 239 → 291 assertions, 0 failures. No existing test,
golden value, or assertion was modified.

Non-goals

No CLI flag (lr+sgd and lr+adam are library-only for the same reason — FTRL's four
hyperparameters map onto none of -gamma / -eta / -c), no example/ entry, no
probability output, and no multiclass-native FTRL.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e1a9b1141f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Avoid overflowing the sigmoid for large negative margins

When a valid example produces a sufficiently negative margin, this expands to exp(-margin) and can signal floating-point overflow instead of returning a gradient near its limiting value. For example, with one feature valued 1000.0, alpha=0.1, and conflicting consecutive labels, the first FTRL update yields a score around 100 and the second evaluates exp(100.0); SBCL configurations that trap overflow will abort training. The sparse path has the same issue at line 1144. Use a sign-dependent sigmoid formulation so the exponential is always evaluated with a non-positive argument.

Useful? React with 👍 / 👎.