Question-bank pipeline: qbank/ (~2,500 questions, checked in, scales to 10k+)

[monikagadage]

What

Replaces the one-off rag.ipynb scrape (single GitHub README → 1,715 questions, ~27 of the source's own labels, transcription noise and all) with qbank/, a real build pipeline. questions_db.json is now a checked-in build artifact (~2,500 questions, 10 canonical topics) so the app runs with no build step.

Three inputs, one shape {topic: [question]}

Input	What it is
qbank/curated/*.txt	~1,700 questions hand-written for this repo, one per line, filed by the topic in the file name (OOP.txt, OOP.2.txt → OOP)
qbank/sources.py	4 public GitHub lists (teamlead, learning-zone, Devinterview, sudheerj), each with a format-specific parser returning (question, category_hint)
qbank/taxonomy.py + --expand	opt-in LLM generation per (topic, subtopic), cached & resumable — takes the bank past 10k

build.py orchestration

normalize (strip numbering/markdown/glued answers; reject headings & boilerplate) → _route to a canonical topic (source hint via CATEGORY_MAP, else the keyword classifier, else drop — unlabelled + unclassifiable is almost always noise, which is why the committed bank is ~2,500 not the ~3,300 raw) → global dedupe → bucket by topic.

• dedup.py is blocked, not O(n²): exact pass on a punctuation-insensitive key, then bucket each question by its 3 rarest tokens and drop any whose token-set Jaccard with a kept one is ≥ 0.85. Question stop words (difference, between, vs, java, …) are stripped first so "difference between X and Y" and "X vs Y in Java" collide.
• classify.py is an ordered-regex heuristic (most specific topic first), Java Core fallback, and build._route only trusts it when a rule actually fired.
• expand.py takes any .invoke(str) -> .content object (fake in tests, ChatGroq in build.py); per-subtopic cache under qbank/.cache/expand/; a flaky call yields [] for that subtopic instead of aborting.
• fetch.py caches downloads with a certifi TLS context (macOS system Python often can't verify GitHub otherwise).

Wiring

• questions_db.json committed; removed from .gitignore; qbank/.cache/ ignored.
• rag.ipynb → 2 cells that shell out to python -m qbank.build.
• corpus.py docstring updated; no code change to the retrieval path.

Tests / CI

tests/test_qbank.py — 38 hermetic tests: normalize, classify, blocked-fuzzy dedup, all 4 source parsers, _route, expansion against a fake LLM (+ cache hit), and a full offline build.build() with fixture Markdown monkeypatched over qbank.fetch.fetch. Needs only pytest. 99 tests total.

Verify

uv run pytest tests/ → 99 passed. python -m qbank.build → 2,558 questions across 10 topics. python -m qbank.build --stats works.

Note

Licensing of the scraped repos is unclear (no LICENSE files) — documented in qbank/SOURCES.md, questions are used as a heavily-transformed seed with attribution, and the build still works with the scraped sources removed (curated + --expand only).