Improve all workshop notebooks for clear real-data learning (#44)

[Laverde97]

Summary

Addresses #44.

Reworks notebooks 00–11 so each section teaches a clear concept end-to-end with
real data, or explicitly justified synthetic examples where they are
pedagogically appropriate.

Highlights:

• standardized notebook teaching structure
• bilingual English/Spanish learning objectives and teaching prose
• real datasets across image, housing, video, taxi, airline, cancer, and audio examples
• explicit synthetic-data justification where appropriate
• interactive Colab widgets where useful
• folded solution metadata preserved after Colab round trips
• no committed outputs or execution counts
• source notebooks and docs/notebooks copies kept byte-identical
• derived section/notebook tables regenerated
• site re-rendered with Quarto 1.6.40

Notebook coverage

• 00 — Setup and welcome
• 01 — What a tensor is
• 02 — Thinking in N dimensions
• 03 — Indexing and broadcasting real data
• 04 — Reshape and transpose real images
• 05 — Video pipeline design
• 06 — Contraction with einsum
• 07 — Inverses and the pseudoinverse
• 08 — Recursion with matrices and vectors
• 09 — Convolution and deconvolution
• 10 — Tucker decomposition on real data
• 11 — Wrap-up and take-homes

Validation

• scripts/gen_tables.py — idempotent
• scripts/gen_notebooks.py — idempotent; 12/12 valid
• scripts/check_links.py — all checks pass
• git diff --check — clean
• Quarto 1.6.40 render — successful and reproducible
• all 12 notebooks/*.ipynb byte-identical to docs/notebooks/*.ipynb
• no outputs or execution counts committed

Follow-up evidence

Issue #44 also asks for before/after screenshots of all twelve notebooks and
final learning-outcome sign-off. Those are issue-level review/evidence
follow-ups, are still outstanding, and are not embedded as notebook outputs.
This PR references #44 but does not close it — the issue stays open for the
remaining screenshot and sign-off work.

🤖 Generated with Claude Code

[project-delphi]

Review

This is the PR for issue #44 (there is no pull request numbered 44). The teaching rewrite is real and mostly in the right direction — bilingual objectives now live in _variables.yml, the named synthetic-data spots in §07 / §08 / §11 are either replaced or explicitly justified, the exercise triple is now consistent, and notebooks/ is byte-identical to docs/notebooks/. CI render is green.

I am requesting changes for the evidence #44 asked for, plus a few things that would ship a different workshop than the one the decks still describe.

Requested changes

1. No Colab “Run all” screenshots anywhere

Issue #44 is explicit about this:

Before-and-after screenshots of each notebook posted as a comment on this issue. The notebooks are committed with no outputs, so screenshots are the only record anyone has of what a run actually looks like.

I checked PR #50 (description, conversation, review threads) and issue #44 (comments). There are zero screenshots, before or after, for any of the twelve notebooks. The PR body correctly says this is still outstanding and that the PR should not close #44 — but that evidence is also what a reviewer needs in order to believe Run All actually works in Colab. Static source is not a substitute: widgets, Plotly, video decode, and %pip cells only show their real shape after a live run.

Please post, on issue #44 (and link from this PR):

• one Run all screenshot per notebook (00–11), taken in Colab, showing the executed outputs / widgets / plots, not just the unrun source
• ideally a matching before shot, or a short note per notebook of what changed visually if a true before is gone

Without those, #44 cannot be signed off.

2. Committed widget state is leftover run output

Cell outputs and execution_count are cleared, but notebooks 05–11 still ship metadata.widgets from the Colab round-trip — including rendered Plotly HTML, stdout, and PNG snapshots. That is committed output, just stored one level up:

notebook	file size	metadata.widgets
05	322 KB	~300 KB
06	503 KB	~461 KB
07	92 KB	~50 KB
08	93 KB	~60 KB (includes Fibonacci stdout + a Plotly figure)
09	423 KB	~395 KB
10	205 KB	~172 KB
11	1.36 MB	~1.32 MB

scripts/gen_notebooks.py strips per-cell outputs and Colab cell metadata, then preserves notebook-level metadata as-is, so this state survives a normalize pass. Please drop metadata.widgets in the normalizer (and regenerate). Screenshots on the issue are the record of a run; the .ipynb files should stay empty.

3. §02 notebook and the decks no longer describe the same session

The notebook is now three real-data coding exercises: it %pip installs ffmpeg, downloads and checksums the 5.9 MB Almadrava clip, and builds a padded order-5 batch. Meanwhile:

• _variables.yml still has format_en: group
• the EN deck title slide still says *Part II · group discussion · no code*
• both decks and the handbook still teach the shape ladder with np.zeros((28, 28)) etc.
• the handbook still says “Go to your breakout channel. No code.”

The notebook even says “We will not invent arrays with np.zeros to build a shape ladder.” Facilitators following the deck will run a different lesson than students who open Colab. Either update the slides/handbook/format to match the new notebook, or keep §02 as the discussion block the clock still allocates 20 minutes of group time for.

4. Spanish objectives never appear in the notebooks

objectives_es was added for every section in _variables.yml (good — that was a named #44 gap). header_cell() in scripts/gen_notebooks.py still only renders objectives_en. Students reading the notebook never see the Spanish outcomes; only the title/summary blockquote is bilingual. Please print both lists in the header (or an > 🇪🇸 block under “What you will be able to do”).

Also please fix (smaller)

• §01 has no ## Why this matters. It jumps Setup → ## 1.1 Vocabulary. Every other rewritten notebook uses the standard shape.
• notebooks/00-setup-and-data.ipynb cell “A note on how these notebooks work” has no 🇪🇸 blockquote. Issue #44 asked for one on every new markdown cell.
• Generated ## Done with this section footers are English-only. Pre-existing, but this PR is the natural place to add a Spanish line in footer_cell().

What looks good

• Named #44 synthetic spots were actually addressed: §07 exercises now run on California housing; §08 keeps Fibonacci / a controlled eigen-gap matrix and says so, then forecasts real airline passengers; §11 take-homes B and D justify synthetic QKV / covariance, and E uses a real voice clip.
• Exercise shape is now consistent (markdown → # TODO → folded #@title Solution) with 3/3 on 01–10 and 5 take-homes on 11.
• Kahoot footers land after 04, 07, 10; everyone else gets “Done with this section.”
• Widget notebooks call enable_custom_widget_manager().
• notebooks/*.ipynb and docs/notebooks/*.ipynb are byte-identical. No cell-level outputs or execution counts.

Once the twelve Run All screenshots are on issue #44 and the widget-state / §02 / objectives_es items are cleaned up, this is in good shape to merge as the content PR, with #44 staying open only if learning-outcome sign-off is still pending.

[project-delphi]

@Laverde97 — per-notebook review for #44 / PR #50. Twelve comments follow, one per notebook (00–11).

Cross-cutting (all notebooks):

• No Run All screenshots on issue Every notebook teaches one concept clearly, on real data #44 yet — please add one Colab screenshot per notebook there.
• objectives_es in _variables.yml is not rendered in notebook headers (English only under “What you will be able to do”).
• Notebooks 05–11 ship committed metadata.widgets (Plotly/HTML/PNG snapshots) despite empty cell outputs — strip in gen_notebooks.py.
• Generated footers (“Done with this section”, Kahoot block) are English-only.

Details per notebook below.

[project-delphi]

@Laverde97

§00 · Setup and welcome

Verdict: ✅ Mostly ready — small polish only.

Structure: Header → Setup → Why this matters → 2 exercises (TODO + folded solution) → What just happened → Done. Matches #44 shape. No Kahoot (correct).

Data: All real — breast cancer, digits, skimage photos, housing/taxis/flights CSVs, video reachability check. “Why this matters” scopes the workshop’s real-data claim and names later synthetic exceptions. Good fix for the old “nothing is invented” overstatement.

Run All: Should work in Colab. %pip install imageio[ffmpeg], three CSV downloads, Range request on video. First cell students hit is the heaviest; failure messaging in prose is clear.

Actions:

1. Add 🇪🇸 blockquote to “A note on how these notebooks work” (only new markdown cell missing Spanish).
2. Post a Run All screenshot on Every notebook teaches one concept clearly, on real data #44 showing setup prints + shape previews.

[project-delphi]

@Laverde97

§01 · What a tensor is

Verdict: ✅ Strong content — one structural gap.

Structure: 3 exercises with TODO/solution triples, What just happened, Done footer. Missing ## Why this matters — jumps Setup → ## 1.1 Vocabulary. Every other rewritten notebook has the standard Why block.

Data: Tiny rng.standard_normal((2,3,4)) demo is explicitly justified (“isolate structure… switch immediately to real image data”). Exercises use digits + immunohistochemistry. Matches objectives.

Run All: Should work. Widgets in Exercise 3 call enable_custom_widget_manager(). No committed widget metadata (clean 32 KB file).

Actions:

1. Add a short Why this matters section (axis semantics on real images) before 1.1.
2. Run All screenshot on Every notebook teaches one concept clearly, on real data #44 — especially the interactive axis-reasoning widget.

[project-delphi]

@Laverde97

§02 · Thinking in N dimensions

Verdict: ⚠️ Best notebook pedagogically — biggest facilitator mismatch.

Structure: Why this matters → 3 exercises → What just happened → Done. Exercise triple consistent. Bilingual prose throughout.

Data: Real digits, astronaut photo, pinned Almadrava video (SHA-256 verified), order-5 padded batch with validity mask. rng.permutation(8) is the pedagogical tool, not fake data. Explicitly rejects the old np.zeros shape ladder. Issue #44 synthetic-data complaints fully addressed.

Run All: Will work but is heavy — Setup %pip installs ffmpeg and downloads ~5.9 MB video. First open in Colab may take 1–2 min. Worth noting in screenshot evidence.

Blocker — deck/handbook drift: Notebook is now a coding exercise; _variables.yml still has format_en: group, EN deck says “Part II · group discussion · no code”, handbook still teaches np.zeros shape ladder and “Go to your breakout channel. No code.” Facilitators and students will see different sessions.

Actions:

1. Either update slides + handbook + format_en to match this notebook or revert §02 to discussion-only.
2. Run All screenshot on Every notebook teaches one concept clearly, on real data #44 — video download + shuffle exercise + padded batch plot.

[project-delphi]

@Laverde97

§03 · Indexing and broadcasting real data

Verdict: ✅ Ready.

Structure: Full standard shape. 3 exercises, What just happened, Done. All markdown cells have 🇪🇸 except generated footer.

Data: Breast cancer (column by name, boolean masks) + digits (broadcasting standardization, zero-variance trap). Prose calls out why synthetic random data would miss the trap. Fully real in exercises.

Run All: Clean file (13 KB). Widgets + enable_custom_widget_manager(). Self-contained Setup loads breast cancer + digits.

Actions: Run All screenshot on #44 — boolean mask exercise + zero-variance pixel discovery.

[project-delphi]

@Laverde97

§04 · Reshape and transpose real images

Verdict: ✅ Ready — model notebook for structure.

Structure: Why this matters → 3 exercises → What just happened → Kahoot footer (correct for §04). Every markdown cell bilingual including Kahoot block.

Data: Three distinct skimage RGB images (astronaut, immunohistochemistry, coffee). HWC↔CHW, NHWC↔NCHW, reshape-vs-transpose trap on real pixels. No synthetic arrays.

Run All: Lightweight (14 KB). No widgets. Should run fast in Colab.

Actions: Run All screenshot on #44 — side-by-side transpose vs reshape on a real image.

[project-delphi]

@Laverde97

§05 · Video pipeline design

Verdict: ✅ Strong — clean committed output.

Structure: Why this matters → 3 exercises → What just happened → Done. Predict → Run → Explain pattern. Exercise triple consistent.

Data: Pinned Almadrava clip end-to-end — decode to (T,H,W,C), frame-sampling loss, two pipeline designs, pad-vs-sample trade-off. All real.

Run All: Setup decodes full video in Colab (~same cost as §02). Widgets for pipeline exploration. metadata.widgets ≈ 300 KB committed (Plotly snapshot from prior run) — strip before merge.

Actions:

1. Drop metadata.widgets via normalizer.
2. Run All screenshot on Every notebook teaches one concept clearly, on real data #44 — sampling timeline visual + interactive pipeline widget.

[project-delphi]

@Laverde97

§06 · Contraction with einsum

Verdict: ✅ Excellent rewrite — watch file size.

Structure: Why this matters (one rule + Predict→Run→Explain) → 3 rich exercises → What just happened → Done. Teaching subheadings (“What are you looking at?”, “What should you try?”) are a nice pattern for 06–10.

Data: Microscopy RGB contraction, digit pixel patches for trace/transpose/matmul, full 1,797-image similarity matrix with cosine vs dot product. All real. Objectives match exercises closely.

Run All: Exercise 3 builds ~3.2M similarities — may take 10–20 s in Colab but should complete. metadata.widgets ≈ 461 KB committed (largest widget payload besides §11). Strip before merge.

Actions:

1. Drop committed widget state.
2. Run All screenshot on Every notebook teaches one concept clearly, on real data #44 — RGB sliders + digit retrieval scatter/interactive chart.

[project-delphi]

@Laverde97

§07 · Inverses and the pseudoinverse

Verdict: ✅ Issue #44 gap fully closed.

Structure: Why this matters → 3 exercises → What just happened → Kahoot footer. Bilingual including Kahoot block and folded-solution instructions.

Data: Old rng.standard_normal exercises are gone. Exercise 1: singular matrix from duplicated real housing columns. Exercise 2: 20,433×7 California housing least squares with interactive geometry. Exercise 3: unfold real digit tensor, pseudoinverse, fold back. Exactly what #44 asked for.

Run All: Housing CSV download in Setup. Plotly widgets for wide/square/tall exploration. metadata.widgets ≈ 50 KB committed — strip.

Actions:

1. Drop widget metadata.
2. Run All screenshot on Every notebook teaches one concept clearly, on real data #44 — singular-matrix diagnosis + housing geometry slider + unfolded-digit heatmap.

[project-delphi]

@Laverde97

§08 · Recursion with matrices and vectors

Verdict: ✅ Synthetic spots now justified; real forecast added.

Structure: Why this matters → 3 exercises → What just happened → Done. Consistent with 06–07 template.

Data: Exercise 1: Fibonacci as matrix state update (controlled, not pretending to be domain data). Exercise 2: explicitly labeled synthetic 2×2 eigen-gap matrix — fixes #44 complaint. Exercise 3: real airline passengers + pseudoinverse AR forecast with compounding error. Old random RNN skeleton replaced.

Run All: Flights CSV in Setup. Widgets for Fibonacci steps and eigen-ratio slider. metadata.widgets ≈ 60 KB includes committed Fibonacci stdout + Plotly figure from a prior run — definitely strip.

Actions:

1. Drop widget metadata (includes hidden stdout/plot output).
2. Run All screenshot on Every notebook teaches one concept clearly, on real data #44 — power-iteration slider + recursive forecast plot on real passengers.

[project-delphi]

@Laverde97

§09 · Convolution and deconvolution

Verdict: ✅ Ready — minor synthetic noise is fine.

Structure: Standard 06–10 template. 3 exercises, What just happened, Done.

Data: Real camera/astronaut pixels for conv/correlate/Toeplitz; transposed conv shape demo; Richardson-Lucy on real blurred photograph. Solution adds 0.002 * standard_normal noise to blurred real image — synthetic noise on real signal, appropriate for deconv demo.

Run All: %pip install not needed. Widgets for kernel/shape exploration. metadata.widgets ≈ 395 KB committed — strip.

Actions:

1. Drop widget metadata.
2. Run All screenshot on Every notebook teaches one concept clearly, on real data #44 — Toeplitz matmul vs fftconvolve + RL recovery slider on blurred photo.

[project-delphi]

@Laverde97

§10 · Tucker decomposition on real data

Verdict: ✅ Ready — strong capstone for Part IV.

Structure: Standard template → What just happened → Kahoot footer (correct).

Data: NYC taxi CSV → genuine order-3 tensor, three-mode unfoldings, HOSVD via SVD+einsum only, rank trade-off sliders, temporal factor linked to hour-18 rush. All real, no random fill-ins.

Run All: %pip install tensorly in exercise if needed (notebook handles it). Tucker on full taxi tensor may take a few seconds. metadata.widgets ≈ 172 KB committed — strip.

Actions:

1. Drop widget metadata.
2. Run All screenshot on Every notebook teaches one concept clearly, on real data #44 — reconstruction error vs rank sliders + temporal factor plot.

[project-delphi]

@Laverde97

§11 · Wrap-up and take-homes

Verdict: ✅ Take-homes well scoped — file size is the main issue.

Structure: Why this matters (connecting idea) → 5 take-homes (A–E), each with TODO + folded solution → What just happened → Done. 5/5 exercise triple (appropriate for wrap-up).

Data: A: real breast cancer PCA scaling trap. B: synthetic Q/K/V explicitly justified (mechanics without tokenizer) — uses default_rng(6) not legacy np.random.seed. C: real taxi CP vs Tucker. D: synthetic covariance by design with portfolio consequence spelled out. E: real CC0 voice + synthetic noise for measurable SNR. All #44 callouts addressed.

Run All: Heaviest notebook — audio download, STFT/SVD/ISTFT, multiple %pip cells possible. metadata.widgets ≈ 1.32 MB committed (largest file in the PR at 1.36 MB total). Must strip before merge.

Actions:

1. Drop widget metadata — this alone saves ~1.3 MB.
2. Run All screenshot on Every notebook teaches one concept clearly, on real data #44 — at minimum one frame per take-home (5 screenshots or one long scroll).
3. Learning-outcome sign-off on Every notebook teaches one concept clearly, on real data #44 still pending after screenshots land.

[Laverde97]

Addressed the requested review fixes in commit 65cfa7a:

• Added Spanish learning objectives to all notebook headers.
• Added the Spanish non-Kahoot completion footer (Fin de esta sección).
• Removed top-level metadata.widgets during notebook normalization while preserving kernelspec/language/Colab metadata.
• Added the missing bilingual “Why this matters” section to Notebook 01.
• Added the missing Spanish note in Notebook 00.
• Aligned Section 02 across _variables.yml, EN/ES slides, handbook, and generated includes as a 20-minute coding demo using the real-data notebook flow instead of the old group/no-code framing.
• Regenerated notebooks/includes/docs with pinned Quarto 1.6.40.

Validation completed successfully:

• check_links.py --notebooks-only ✅
• full check_links.py ✅
• git diff --check ✅
• 12/12 source notebooks byte-identical to rendered docs/notebooks/ copies ✅
• 24/24 source + rendered notebooks valid, with no outputs, execution counts, or widget state ✅
• working tree clean after push ✅

Remaining manual items from Issue #44:

• the twelve Colab “Run all” screenshots;
• the learning-outcome sign-off.

Those manual items are intentionally not represented as local-code validation.

[Laverde97]

@Laverde97

§00 · Configuración y bienvenida

Veredicto: ✅ Casi listo — solo un pequeño retoque.

Estructura: Encabezado → Configuración → Por qué esto importa → 2 ejercicios (PENDIENTE + solución plegada) → Qué acaba de suceder → Hecho. Coincide con la forma n. ° 44. No hay Kahoot (correcto).

Datos: Todos reales: cáncer de mama, dígitos, fotos de Skimage, archivos CSV de alojamiento/taxis/vuelos, comprobación de accesibilidad de vídeo. La sección «Por qué esto importa» define el alcance de la afirmación del taller sobre el uso de datos reales y menciona excepciones sintéticas posteriores. Una buena solución para la antigua exageración de que «nada es inventado».

Ejecutar todo: debería funcionar en Colab. %pip install imageio[ffmpeg], tres descargas CSV, solicitud de rango en video. La primera celda que impactan los estudiantes es la más pesada; el mensaje de error en prosa es claro.

Comportamiento:

1. Agregue 🇪🇸 blockquote a “Una nota sobre cómo funcionan estos cuadernos” (solo falta el español en la nueva celda de Markdown).
2. Publica una captura de pantalla de Run All en Cada cuaderno enseña un concepto claramente, con datos reales  #44 que muestra impresiones de configuración + vistas previas de formas.

@Laverde97

§00 · Configuración y bienvenida

Veredicto: ✅ Casi listo — solo un pequeño retoque.

Estructura: Encabezado → Configuración → Por qué esto importa → 2 ejercicios (PENDIENTE + solución plegada) → Qué acaba de suceder → Hecho. Coincide con la forma n. ° 44. No hay Kahoot (correcto).

Datos: Todos reales: cáncer de mama, dígitos, fotos de Skimage, archivos CSV de alojamiento/taxis/vuelos, comprobación de accesibilidad de vídeo. La sección «Por qué esto importa» define el alcance de la afirmación del taller sobre el uso de datos reales y menciona excepciones sintéticas posteriores. Una buena solución par

a la antigua exageración de que «nada es inventado».

Ejecutar todo: debería funcionar en Colab. %pip install imageio[ffmpeg], tres descargas CSV, solicitud de rango en video. La primera celda que impactan a los estudiantes es la más pesada; el mensaje de error en prosa es claro.

Comportamiento:

1. Agrega 🇪🇸 blockquote a “Una nota sobre cómo funcionan estos cuadernos” (solo falta el español en la nueva celda de Markdown).
2. Publica una captura de pantalla de Run All en Cada cuaderno enseña un concepto claramente, con datos reales  #44 que muestra impresiones de configuración + vistas previas de formas.

Notebook 00 · Setup and welcome — Colab Run All evidence

Executed 00-setup-and-data.ipynb in Google Colab using Runtime → Run all from PR #50 branch feat/issue-44-notebooks at commit 65cfa7a.

• Run All completed in real Google Colab.
• Setup/data-loading outputs completed successfully.
• Dataset shape previews are visible in the screenshots.
• No execution error was observed during the completed run.
• The notebook outputs were not saved back to the repository.

Screenshots below show the executed Colab session and the relevant setup/shape outputs.

[project-delphi]

Super, let's get the rest of the notebooks done and completing the requests we want handled.

[Laverde97]

Colab Run-All evidence requested for Issue #44 is now posted here:

#44 (comment)

All 12 notebooks (NB00–NB11) passed Colab Runtime → Run all on commit 958ce0b6b3cdd6aa4545dfc73a7a9d3f17432053.

The Issue #44 comment includes 24 screenshots: one historical Before and one executed After for each notebook.