CNN-Math

Computer vision pipeline that recognizes, interprets, and solves handwritten mathematical expressions. Instead of one end-to-end model, the problem is split into four stages so each piece can be built and validated on its own:

1. Segmentation (src/cnn_math/preprocessing/) — OpenCV preprocessing (grayscale inversion, adaptive thresholding) and contour detection to isolate each handwritten character into its own bounding box.
2. Classification (src/cnn_math/model/) — a custom CNN (Keras/TensorFlow) trained on public digit/operator/letter datasets to label each cropped box.
3. Spatial parsing (src/cnn_math/parser/) — a geometric algorithm over bounding box centers that decides horizontal vs. stacked (fraction) layout and builds a linear expression string.
4. Solving (src/cnn_math/solver/) — hands the string to SymPy to evaluate arithmetic or solve for a target variable.

src/cnn_math/pipeline.py wires all four stages together end to end.

Project layout

CNN-Math/
├── configs/
│   └── config.yaml          # Central config: preprocessing/model/paths/parsing params
├── data/
│   ├── raw/                 # bhmsds training images, one folder per class (gitignored, regenerate via prepare_dataset.py)
│   ├── processed/           # label_map.json — class-name -> model output index
│   ├── real_samples/        # Ground-truth-labeled crops from real photos, used to fine-tune toward actual handwriting
│   └── samples/             # Example input images for manual/CLI testing
├── models/                  # Saved model weights (.keras) — gitignored, regenerate via train.py
├── notebooks/                # Exploratory notebooks
├── scripts/
│   ├── prepare_dataset.py   # Fetch bhmsds and lay it out under data/raw/
│   ├── train.py             # Train the CNN classifier from scratch on data/raw/
│   ├── finetune.py          # Continue training from a saved checkpoint on data/real_samples/ + bhmsds anchors
│   ├── evaluate.py          # Evaluate classifier accuracy on the held-out validation split
│   ├── confusion_matrix.py  # Plot a confusion matrix for the trained classifier
│   ├── run_pipeline.py      # Run the full image -> answer pipeline (CLI)
│   └── main.py              # Run the pipeline on an image and pop up an annotated result window
├── src/cnn_math/
│   ├── preprocessing/
│   │   ├── image_utils.py   # Grayscale/threshold/denoise helpers
│   │   └── segmentation.py  # Contour detection + bounding box extraction
│   ├── model/
│   │   ├── architecture.py  # Keras CNN model definition
│   │   ├── labels.py        # Label-map constants + (de)serialization (no TF dep)
│   │   ├── dataset.py       # TF dataset loading / augmentation
│   │   └── classifier.py    # Train/predict wrapper around the CNN
│   ├── parser/
│   │   └── spatial_parser.py# Bounding-box geometry -> expression string
│   ├── solver/
│   │   └── symbolic_solver.py # SymPy evaluation / equation solving
│   └── pipeline.py          # End-to-end orchestration
└── tests/                   # Unit tests per stage

data/raw/ and data/real_samples/ serve different roles: raw/ is the large, generic bhmsds training set (regenerable from a fresh clone, never touched by hand); real_samples/ is small, hand-curated, and specific to your handwriting — it's what finetune.py uses to adapt the model beyond bhmsds's clean, uniform samples.

Setup

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

Usage

# Download the dataset and lay it out for training
python scripts/prepare_dataset.py

# Train the CNN on the configured dataset
python scripts/train.py --config configs/config.yaml

# Evaluate the trained model
python scripts/evaluate.py --config configs/config.yaml

# Run the full pipeline on an image of a handwritten expression
python scripts/run_pipeline.py --image data/samples/eq1.jpg --variable x

# Same as above, but pop up a window with the detected boxes/labels drawn
# on the image and the parsed expression + answer as the title
python scripts/main.py --image data/samples/eq1.jpg --variable x

# Optional: fine-tune toward your own handwriting once you have some
# ground-truth-labeled crops in data/real_samples/<label>/
python scripts/finetune.py --config configs/config.yaml

Dataset

Classifier training data comes from the Basic Handwritten Math Symbols Dataset (MIT licensed): 27,000 images across 18 classes — digits 0-9, operators + - * /, and letters w x y z — 1,500 images each.

scripts/prepare_dataset.py clones the dataset and reorganizes its flat symbols/<name>-<id>.png files into the per-class folder layout load_dataset() expects (data/raw/<label>/), matching the keys in DEFAULT_LABEL_MAP (src/cnn_math/model/labels.py). Two classes use a filesystem-safe folder name instead of the literal character (star for *, slash for /); LABEL_TO_CHAR translates these back to the real character before the parser/solver stages ever see a label.

Status

End-to-end pipeline is implemented and working: trained on bhmsds (~98.5% validation accuracy on an 80/20 split), then fine-tuned on real handwritten photo crops via scripts/finetune.py. Verified against a real photo of a handwritten quadratic equation, read and solved correctly.

Symbols bhmsds doesn't cover are handled structurally in spatial_parser.py (geometry, not classification — the CNN's raw prediction for these is never trusted directly):

• = — bhmsds has no equals-sign class, so each stroke of a handwritten = gets classified as its own -. Two - predictions stacked closely with overlapping x-ranges are merged into one = token (_merge_equals_signs).
• Exponents — a small character positioned above-and-right of another (e.g. the "2" in handwritten x2) is read as a superscript and folded into base**exp (_merge_superscripts), so x^2 parses correctly.
• Thousands-separator commas — , isn't a trained class and (unlike =) doesn't reliably resemble any one trained class either, so there's no classifier-output pattern to key off. Instead it's detected by shape — small area, sitting at/below the baseline — and dropped (_drop_thousands_separators). Verified against a real photo of 8,008,008 (correctly read as 8008008); the first threshold pass (calibrated on synthetic geometry alone) missed both real commas, so the area/position tolerances were widened based on that photo's actual box geometry — see test_thousands_separator_comma_dropped_real_geometry in tests/test_parser.py for the exact numbers that drove the fix.
• Numbers with a spurious leading zero (e.g. a misread producing 0087008) no longer crash the solver — symbolic_solver.py strips leading zeros before parsing, since Python's own grammar (which SymPy's parser relies on) rejects them as literals.

Known limitations, not yet addressed:

• Segmentation can fragment or merge touching strokes on real photos (e.g. a character's disconnected pen strokes read as two boxes, or two close-together characters merge into one) — adaptive-threshold tuning is per-image, not automatic.
• No fully held-out test set — only a train/validation split exists; evaluate.py reports accuracy on the same validation data used during training, not a separate set untouched until final reporting.