Add Real-World Project: Build a Hybrid Search Demo

docs/projects/hybrid-search/_category_.json

@@ -0,0 +1,4 @@
+{
+  "label": "HybridSearchDemo",
+  "position": 23
+}

docs/projects/index.mdx

@@ -17,6 +17,12 @@ They're optional and ungraded. Browse them any time — each project's intro say
 
 <ProjectChooser
   projects={mergeProjectMeta([
+    {
+      id: 'hybrid-search',
+      title: 'Build a Hybrid Search Demo',
+      summary:
+        'Run keyword (BM25-style), embedding-based, and hybrid retrieval side by side on the same small corpus, and see where each approach wins and loses — no API key, no LLM.',
+    },
     {
       id: '2027-dependency-freshness-checker',
       title: 'Build a Dependency-Freshness Checker',

examples/hybrid-search/.gitignore

@@ -0,0 +1,4 @@
+.venv
+__pycache__
+*.pyc
+data/index.npy

examples/hybrid-search/.python-version

@@ -0,0 +1 @@
+3.12

examples/hybrid-search/README.md

@@ -0,0 +1,55 @@
+# Hybrid Search Demo Example
+
+The local companion to the course's [Build a Hybrid Search Demo (Keyword + Embedding)](../../docs/projects/hybrid-search/index.md) project — a small CLI that runs **keyword** (a from-scratch BM25-style scorer), **embedding** (local `sentence-transformers`), and **hybrid** retrieval side by side over the same tiny corpus, so you can see where each approach wins and loses.
+
+No LLM, no API key, no `.env` — the only download is the small local embedding model, which `sentence-transformers` fetches on first run.
+
+<!-- TODO: update these badge links to point at main once this PR merges -->
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abderrahim-lectures/python-data-analysis-course/blob/main/examples/hybrid-search/notebook.ipynb)
+[![Open In Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/abderrahim-lectures/python-data-analysis-course/blob/main/examples/hybrid-search/notebook.ipynb)
+[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/abderrahim-lectures/python-data-analysis-course/main?filepath=examples%2Fhybrid-search%2Fnotebook.ipynb)
+
+## What's here
+
+- `main.py` — a single-file CLI with the three retrievers:
+  - `KeywordScorer` — a small BM25-style lexical scorer built from scratch (idf + term-frequency saturation), so you can read every line of the math instead of trusting a library.
+  - `semantic_scores()` — local embeddings via `all-MiniLM-L6-v2` (384-dim), ranked by cosine similarity.
+  - `hybrid_scores()` — min-max-normalized keyword and semantic scores combined with a configurable `--alpha` weight.
+- `data/corpus/` — eleven short `.txt` passages on a mix of topics (Neptune, espresso, deep sea, piano, cycling, sourdough, ...), deliberately written so **keyword and semantic retrieval disagree**: each "paraphrase" passage expresses the same idea as its neighbor without sharing its vocabulary.
+- `data/test_queries.json` — ten test queries, each with the document that *should* rank first and a note on whether it's an exact-match or a paraphrase query.
+- `notebook.ipynb` — the same comparison as a self-contained notebook (the corpus is embedded inline, so nothing needs downloading beyond the packages and the model). Click a badge above to run it in Colab, Kaggle, or Binder with no local setup.
+
+## How to run this
+
+```bash
+# Build the embedding index once (downloads all-MiniLM-L6-v2 on first run)
+uv run python main.py --build
+
+# Compare all three methods on a single query
+uv run python main.py "Neptune planet winds"
+
+# Run the bundled test queries and print the per-method winners table
+uv run python main.py --evaluate
+```
+
+Other things to try:
+
+```bash
+# Show more hits per method, and shift the hybrid blend toward keyword
+uv run python main.py "grand piano sustain pedal" --top-k 5 --alpha 0.7
+
+# Reuse an existing data/index.npy instead of re-embedding on every run
+uv run python main.py "bicycle hill climbing gears" --reuse-index
+```
+
+`uv run` reads `pyproject.toml`/`uv.lock` and creates an isolated environment for this project automatically on first run — no manual virtual environment setup needed.
+
+## What the output means
+
+For each query you get three small tables (keyword / semantic / hybrid), each showing the top hits with a score. The `--evaluate` winners table then counts, across all test queries, how often each method put the *expected* document at rank 1 — in this bundled corpus keyword wins the exact-match queries, embeddings win the paraphrase queries, and hybrid inherits both. That's the honest takeaway: **there is no single "best" retriever** — it depends on the query, and on your corpus.
+
+See the full [Build a Hybrid Search Demo lesson](../../docs/projects/hybrid-search/index.md) for the step-by-step walkthrough, including the BM25 math and how the scores get combined.
+
+## Built your own version?
+
+See [`examples/student-projects/`](../student-projects/) for how to share it with the class via a pull request — no git experience required, it walks through every step.

examples/hybrid-search/data/corpus/coldbrew.txt

@@ -0,0 +1 @@
+Cold brew is coffee steeped in cold water for twelve to twenty-four hours and then filtered. Because the water never turns hot, cold brew tastes smoother and less acidic than coffee brewed with heat.

examples/hybrid-search/data/corpus/cycling.txt

@@ -0,0 +1 @@
+Bicycles transfer pedal power to the rear wheel through a chain and a set of gears. A small front chainring makes steep hills easier, while a large one trades torque for speed on flat ground.

examples/hybrid-search/data/corpus/deepsea.txt

@@ -0,0 +1 @@
+The deep ocean below a thousand meters is cold, dark, and under crushing pressure. Its residents include anglerfish that dangle glowing lures and giant squids with eyes nearly as large as dinner plates.

examples/hybrid-search/data/corpus/espresso.txt

@@ -0,0 +1 @@
+An espresso is a concentrated coffee made by forcing hot water under nine bars of pressure through finely ground, tightly packed coffee. It forms the base of lattes, cappuccinos, and flat whites, and wears a layer of brown foam called crema on top.

examples/hybrid-search/data/corpus/neptune.txt

@@ -0,0 +1 @@
+Neptune is the eighth and farthest planet from the Sun, a deep-blue ice giant with the strongest recorded winds in the solar system. It was discovered in 1846 by astronomers who predicted its position mathematically from disturbances in the orbit of Uranus, long before any telescope had ever seen it.

examples/hybrid-search/data/corpus/paraphrase_espresso.txt

@@ -0,0 +1 @@
+Brewing an intense, tiny serving starts by pushing near-boiling water through very finely ground beans packed into a metal basket. Cafes steam milk on the side and pour the two together for their most popular hot drinks.

examples/hybrid-search/data/corpus/paraphrase_neptune.txt

@@ -0,0 +1 @@
+The most distant ice giant in our solar system spins far beyond Saturn and Uranus. Astronomers located it by working out where a hidden planet must be to explain a strange wobble in Uranus's motion, then pointed their telescopes there and found it within a degree of the predicted spot.

examples/hybrid-search/data/corpus/piano.txt

@@ -0,0 +1 @@
+A grand piano produces sound when felt-covered hammers strike steel strings. Pressing the sustain pedal lifts the dampers so the strings keep vibrating after the key is released, letting notes blend into one another.

examples/hybrid-search/data/corpus/sourdough.txt

@@ -0,0 +1 @@
+Sourdough bread rises through a slow fermentation of flour and water by wild yeast and lactobacillus bacteria. No commercial yeast is added, which is what gives the loaf its sour tang and open crumb.

examples/hybrid-search/data/corpus/telescope.txt

@@ -0,0 +1 @@
+A reflecting telescope gathers light with a curved primary mirror instead of a glass lens. Using a mirror avoids chromatic aberration, the colored fringing that afflicts refracting telescopes, and lets builders create much larger instruments for the same price.

examples/hybrid-search/data/corpus/tomatoes.txt

@@ -0,0 +1 @@
+Tomatoes color up best when daytime temperatures stay between twenty and twenty-five degrees Celsius. Above thirty degrees, ripening slows sharply and the fruit stays greenish even though it is fully mature inside.

examples/hybrid-search/data/test_queries.json

@@ -0,0 +1,62 @@
+[
+  {
+    "query": "Neptune planet winds",
+    "expected": "neptune.txt",
+    "kind": "keyword",
+    "note": "Exact-match query: the rare word 'Neptune' is only in neptune.txt, so keyword should win."
+  },
+  {
+    "query": "an icy mystery world charted by pure calculation",
+    "expected": "paraphrase_neptune.txt",
+    "kind": "semantic",
+    "note": "Paraphrase query: none of these words appear anywhere in the corpus, so only embeddings can connect it to the Neptune passage."
+  },
+  {
+    "query": "espresso crema",
+    "expected": "espresso.txt",
+    "kind": "keyword",
+    "note": "Exact-match query: both rare tokens live only in espresso.txt."
+  },
+  {
+    "query": "squeezing hot water through fine grounds for a quick cup",
+    "expected": "paraphrase_espresso.txt",
+    "kind": "semantic",
+    "note": "Paraphrase query: the words 'hot water' overlap the cold-brew passage and lead keyword astray, but the meaning matches the espresso paraphrase."
+  },
+  {
+    "query": "bicycle hill climbing gears",
+    "expected": "cycling.txt",
+    "kind": "keyword",
+    "note": "Exact-match query: 'bicycle', 'hill', and 'gears' are distinctive tokens."
+  },
+  {
+    "query": "grinding uphill against gravity",
+    "expected": "cycling.txt",
+    "kind": "semantic",
+    "note": "Paraphrase query: none of these words appear verbatim, but the meaning matches cycling.txt's steep-hills passage."
+  },
+  {
+    "query": "why did my overnight dough get that bite",
+    "expected": "sourdough.txt",
+    "kind": "semantic",
+    "note": "Paraphrase query: 'dough' and 'bite' are absent from the corpus; the tang of a long fermentation must be found by meaning."
+  },
+  {
+    "query": "grand piano sustain pedal",
+    "expected": "piano.txt",
+    "kind": "keyword",
+    "note": "Exact-match query: all three tokens are distinctive to piano.txt."
+  },
+  {
+    "query": "a freezing abyss where animals make their own glow",
+    "expected": "deepsea.txt",
+    "kind": "semantic",
+    "note": "Paraphrase query: no vocabulary overlap, but the deep-sea creatures with glowing lures are the intended match."
+  },
+  {
+    "query": "reflecting telescope mirror aberration",
+    "expected": "telescope.txt",
+    "kind": "keyword",
+    "note": "Exact-match query: distinctive telescope vocabulary."
+  }
+]