How the pipeline works, why it's built this way, and where it stands. For setup/run commands see SETUP.md.
This is not speech-to-text. It never transcribes what was said. It answers one narrower question: does this audio match one of a fixed set of known products closely enough to call it a hit?
That reframing is the whole architecture in one sentence:
frozen encoder + a small bank of reference embeddings + nearest-neighbor lookup
No model in this pipeline is trained or fine-tuned. The only "training" step is enrollment — recording a handful of reference clips per product and converting them to vectors. Adding a new vegetable tomorrow means adding ~20 clips and re-running one script, not retraining anything.
raw .wav files single .wav
data/<class>/<lang>/ data/splits/enroll.jsonl │
│ │ ▼
▼ ▼ ┌────────────┐
┌──────────────┐ ┌──────────────────┐ │ raw_audio │
│ split_data.py│─────────────▶│ enroll.py │ └─────┬──────┘
└──────────────┘ manifests │ (encoder.py + │ ▼
│ │ raw_audio.py) │ ┌────────────┐
│ └─────────┬─────────┘ │ encoder │
│ ▼ └─────┬──────┘
│ bank/bank.npz ▼
│ bank/bank_meta.json ┌────────────┐
│ │ │ match.py │◀── bank/bank.npz
▼ ▼ └─────┬──────┘
data/splits/val.jsonl ┌──────────────────────┐ ▼
data/splits/test.jsonl │ evaluate.py │ prediction +
└───────────────▶│ (tune on val, │ ranked scores
│ lock, score on test) │ (src/infer.py)
└──────────┬───────────┘
▼
results/report.{json,md}
results/confusion_matrix.{csv,png}
results/score_histogram.png
Clips are loaded as-is (src/raw_audio.py) apart from resampling to audio.sample_rate — no VAD trim, normalization, or pad/crop. src/preprocess.py still implements the fuller behavior but is unused by the pipeline.
Every stage reads a file the previous stage wrote (.jsonl manifests, bank.npz) — nothing re-derives state from a stage two steps back, and every stage can be run and inspected in isolation.
A trained classifier (e.g. a small CNN with a softmax head over 10 classes) would need retraining every time a product is added or removed, needs meaningfully more data per class to generalize, and bakes the class list into the model's weights. None of that fits a supermarket scale where the product list changes.
The enrollment approach trades a small amount of raw accuracy for:
- O(seconds) to add a class — encode a few clips, append to the bank.
- No GPU, no training loop, no labels beyond "which folder was this recorded in."
- A bank that's human-inspectable — it's just vectors + labels, not opaque trained weights.
This is the same idea behind speaker verification and face recognition systems (enroll a few reference embeddings, compare new samples by distance) applied to word identity instead of speaker identity.
Fine-tuning would require labeled data at a scale this PoC doesn't have (5–20 clips/class), and would tie the model to exactly today's product list. Freezing the encoder (model.eval() + requires_grad_(False), wrapped in torch.no_grad() in encoder.py) means the same encoder generalizes to any future product without touching it again — the only thing that changes when products change is the bank.
src/encoder.py defines an Encoder protocol (embed_dim + .embed(waveform) -> vector) and a build_encoder(config) factory keyed off config.encoder.name. Swapping models is a config change, not a code change — any HuggingFace Wav2Vec2-family model (anything loadable via AutoModel + AutoFeatureExtractor with output_hidden_states=True) can be dropped in by setting hf_id/edge_hf_id in config.yaml. No caller of Encoder.embed() needs to know which model backs it.
Why these two specifically, to start:
xlsr(facebook/wav2vec2-xls-r-300m) — pretrained on 128 languages including Sinhala and Tamil, which matters here since most off-the-shelf speech models are English-only. Used as the accuracy reference — the ceiling we check the cheaper option against, not the deployment target.edge(ntu-spml/distilhubert) — ~24M parameters vs. XLS-R's ~300M (see §5). Chosen as the deployment candidate because it's the standard distilled-HuBERT baseline, small enough to plausibly fit the Raspberry Pi Zero 2 W's RAM budget.
To try a better encoder later: point config.encoder.hf_id (or edge_hf_id) at any other HF checkpoint, adjust layer/pooling if needed, rebuild the bank (enroll.py), and rerun evaluate.py. The bank's metadata records which encoder/layer built it, and match.py/infer.py refuse to run if the active config doesn't match — so a stale bank can't silently produce garbage predictions after a swap.
Mid-layer transformer activations tend to carry more phonetic detail than the final layer, which drifts toward whatever the model's original pretraining objective needed (masked-prediction targets, not word identity). Making layer a config value let us sweep it empirically instead of guessing. Pooling (mean vs max) collapses the variable-length sequence of per-timestep vectors into one fixed vector — mean averages the whole clip, max keeps each dimension's peak value regardless of when it occurred. Both are swept on validation data, never assumed.
One caveat this project surfaced: DistilHuBERT only has 2 transformer layers, so layer values above that clamp to the same output every time (min(self.layer, len(hidden_states)-1) in encoder.py). The layer sweep is only meaningful for deeper models like XLS-R.
Every embedding is normalized to unit length right after pooling. That makes a dot product between two vectors equal to their cosine similarity — so match.py's entire scoring step is one matrix multiply (bank.vectors @ query), no separate normalization at match time, no framework dependency. Cosine similarity also cares about direction, not magnitude — robust to a louder or quieter recording producing a longer vector, which raw Euclidean distance wouldn't be.
match.py has no torch/transformers import. This is deliberate: it's the code path that has to run on the actual scale. The encoder (heavy, needs a real ML runtime) only runs once per enrolled clip and once per query; matching is a single small matrix multiply against a bank that's kilobytes to low-single-digit megabytes (§5). Keeping this boundary sharp means the eventual on-device build only needs to port preprocessing + this file — the training/encoding infrastructure never has to run on the Pi.
Two bank strategies exist:
prototype— average all enrollment embeddings per class into one vector.all_samples— keep every enrollment embedding, score a class by its best-matching stored sample (nearest neighbor within class).
all_samples is what's active, because averaging blurs away exactly the variation (different speakers, languages, mic conditions) that enrollment is supposed to capture — a query only needs to resemble one good reference, not the centroid of all of them. The cost is a slightly bigger bank (still trivially small, §5) and a few more comparisons per query (still trivial — one matrix multiply either way).
Per class, clips are shuffled with a seeded RNG (config.seed) and sliced:
[ enroll (n_enroll) ][ val (n_val) ][ test (everything else) ]
Currently n_enroll=20, n_val=5 per class, out of 210–242 clips per class — so test sets run ~185–232 clips per class, ~2073 total.
Speaker-disjoint splitting was the design goal, but this dataset's filenames (Ladies fingers (13).wav, Recording (4).wav, ...) don't encode a parseable speaker ID. split_data.py detects this, falls back to a random per-class split, and prints a loud warning that results may be optimistic — the same speaker could plausibly appear in both enroll and test, which is an easier problem than a truly unseen speaker at inference time. This is a known, documented limitation, not an oversight.
Splits are written once to data/splits/{enroll,val,test}.jsonl. Every downstream stage reads these files — nothing re-globs the raw data/ folder, so the split is fixed for the life of those manifest files regardless of what changes later in data/.
- Enroll → builds the bank. Never scored against.
- Validation → the only data used to pick
encoder.layerandmatch.threshold(sweep()inevaluate.py). Sweeps every(layer, threshold)combination inconfig.eval.layer_sweep/threshold_sweep. - Test → touched exactly once, after the val sweep has already locked in a winning config (
evaluate_test()). No code path inevaluate.pylets test data influence which config gets chosen — that would silently inflate the reported number.
87.3% test accuracy (n = 2,073), locked config: encoder=edge (DistilHuBERT), pooling=max, n_enroll=20, bank.mode=all_samples.
| step | config | test accuracy |
|---|---|---|
| 1 | XLS-R-300M, mean-pool, layer 16, 5 enroll/class | 48.4% |
| 2 | (validation-only sweep across encoders/pooling/layers) | — |
| 3 | DistilHuBERT, max-pool, 5 enroll/class | 60.9% |
| 4 | DistilHuBERT, max-pool, 20 enroll/class | 87.3% |
Enrollment count was the dominant lever — far more than encoder choice or layer/pooling tuning. Going from 5 to 20 enrollment clips per class (step 3→4, no other change) added 26 points; switching the encoder architecture entirely (step 1→3) added 12.
| class | accuracy |
|---|---|
| carrots | 100.0% |
| leeks | 100.0% |
| green chillies | 96.4% |
| potatoes | 95.2% |
| pumpkin | 92.6% |
| long beans | 81.4% |
| tomatoes | 81.4% |
| onions | 81.1% |
| garlic | 73.7% |
| ladies fingers | 73.2% |
| language | n | accuracy |
|---|---|---|
| Tamil | 698 | 89.3% |
| Sinhala | 689 | 87.5% |
| English | 686 | 85.1% |
Contrary to the general assumption (that Sinhala/Tamil would trail English as lower-resource languages for the pretrained encoder), all three languages land within 4 points of each other. The gap is driven by two class×language outliers rather than a general language effect: tomatoes in English (48%) and garlic in Tamil (51%) — both well below that class's accuracy in the other two languages. Worth checking whether those specific cells share a recording session/mic before attributing it to the encoder.
- tomatoes ↔ potatoes (31/215 tomato clips) — both end in the same "-toes" sound; a genuine phonetic near-rhyme, plausibly a hard floor for an acoustic-only matcher.
- garlic → potatoes/onions (47/205) and ladies fingers ↔ green chillies/long beans (44/205) — not an obvious phonetic pair; unresolved, flagged for follow-up.
| component | size |
|---|---|
| embedding bank (200 vectors × 768 dims × float32) | ~0.6 MB |
| edge encoder (DistilHuBERT, int8) | ~25 MB |
| Raspberry Pi Zero 2 W usable RAM | ~350–400 MB |
| xlsr encoder (wav2vec2-300M, fp32) | ~1,200 MB |
The bank is never the memory problem, at any realistic product-catalog size (1,500 classes × 5 samples × 768 dims ≈ 23 MB float32). The encoder is the constraint — XLS-R alone exceeds the Pi's entire usable RAM, which is exactly why it's kept as the accuracy reference rather than the deployment target, and why the edge profile exists.
export/to_onnx.py— ONNX export + int8 quantization of the edge encoder, with a parity check against the PyTorch version.- Real latency/RAM measurement on physical Raspberry Pi Zero 2 W hardware (today's RAM numbers are spec sheets, not measurements).
- Root-causing the two open class×language confusions in §5.4/5.5.
- Full End-to-End Live Pipeline Integration: Implementing real-time audio streaming input from the physical microphone directly into the preprocessing, embedding, and matching routines for continuous live classification.