Skip to content

Gradient descent belongs in section 08, at scalar, vector and matrix scale #49

Description

@project-delphi

Section 08 defines recursion as apply the same map again and again, then works
three of them — Fibonacci as np.linalg.matrix_power, power iteration, and the AR(12)
airline forecast fed its own output — and closes by naming the RNN hidden-state step in
prose (notebooks/08-recursion-with-matrices.ipynb, exercises 1–3 and the What just
happened
cell). The recursion everyone in the room will actually run is missing:
gradient descent, x_{k+1} = x_k − η ∇f(x_k).

It fits the section's thesis exactly. The same update, written three times at three
ranks, is the clearest way to show that the shape of the thing being updated is a
detail and the recursion is the point.

1. Scalar — w ← w − η f'(w)

Minimise something with a known answer, e.g. f(w) = (w − 3)², so the iterates can be
checked against w* = 3.

Plot: the parabola with the iterates dropped onto it as connected points, plus a
second panel sweeping η — too small (crawls), right (lands), too large (oscillates,
then diverges). The learning-rate panel is the one that earns its space.

2. Vector — w ← w − η Xᵀ(Xw − y)

Reuse the section's own airline design matrix X and target y[p:], so descent
converges to the same w that np.linalg.pinv(X) @ y[p:] already produces two
cells earlier. Two routes, one answer — that is the link back to section 07.

Plot: loss vs. iteration on a log y-axis with the closed-form loss as a horizontal
line; and a contour of the loss over two of the weights with the descent path traced
across it.

3. Matrix — W ← W − η ∇_W L

Low-rank factorization X ≈ A B, fitted by descent on both factors, on
load_digits() — already used elsewhere in the workshop, and the natural on-ramp to
section 10.

Plot: reconstruction error vs. iteration, with a strip of the reconstructed digit at
a few checkpoints so the reader watches the image resolve as the recursion runs.

Close with one line tying it back: the RNN cell already named at the end of the section
is trained by exactly this, which is why gradient descent and recursion belong on the
same page.

Housekeeping

  • Teaching cells are notebook-owned: write them directly into
    notebooks/08-recursion-with-matrices.ipynb, by hand or in Colab with Gemini. Only
    the header, objectives, Colab badge, Setup preamble, Setup code and footer are
    generated, and those live in _variables.yml / scripts/content.py. The normalizer
    preserves the body — see #43.
  • Section 08 already imports Plotly and ipywidgets in its Setup cell
    (scripts/content.py, CONTENT["08"]). Draw with those rather than adding a stack.
    load_digits() for the matrix example needs from sklearn.datasets import load_digits
    in the cell that uses it — Setup does not import it, and Setup is generated, so don't
    hand-edit it there.
  • Follow the section's cell pattern: markdown exercise statement → # TODO stub →
    folded #@title Solution / Solución cell.
  • Keep the bilingual pattern: each new markdown cell carries its 🇪🇸 gloss.
  • No visible cell may use a name bound only inside a solution cell —
    check_links.py checks this.
  • Section 08 is budgeted at 10 minutes (_variables.yml, s08.minutes). Three
    examples will not fit that. Either keep them tight and demo-only, or raise minutes
    and update start/end plus the agenda rows — check_links.py verifies the
    running clock and fails if only one of those moves.
  • Rerun gen_notebooks.py (and gen_tables.py if _variables.yml moved), then
    check_links.py.
  • quarto render (pinned 1.6.40) and commit docs/ — a notebook change committed
    without a re-render leaves docs/notebooks/ serving the old copy and nothing fails.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

contentTeaching content: accuracy, examples, referencesenhancementNew feature or requestnotebooksnotebooks/ and scripts/content.py

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions