Skip to content

Phase 2 groundwork: Jetson detection, honest power, ESP32-S3 CNN inference - #31

Open
tarun101 wants to merge 3 commits into
mainfrom
claude/image-analyze-button-rtxg8l
Open

Phase 2 groundwork: Jetson detection, honest power, ESP32-S3 CNN inference#31
tarun101 wants to merge 3 commits into
mainfrom
claude/image-analyze-button-rtxg8l

Conversation

@tarun101

Copy link
Copy Markdown
Owner

Follows up the Jetson/Wyze/ESP32 camera support with the pieces that make those platforms actually usable for Phase 2 (Orin Nano + ESP-CAM). Three independent commits.

1. Detection on other cameras — calibration profiles (46ba05f)

Box-finding + warp were hard-coded to one imx708 capture, so detection couldn't locate the grid on any other camera. Lifted into a camera profile (detect/profiles/<name>.json), selected by PILLBOX_CAMERA_PROFILE (default imx708).

  • camera_profiles.py loads/validates the active profile; crop_cells.py exposes its fields under the same module names, so every caller is untouched.
  • profiles/imx708.json reproduces the original constants bit-for-bit (asserted) — the Pi deployment and all committed pillbox-data numbers are unchanged.
  • calibrate_camera.py generates a profile for a new camera from one empty-box photo: mark 4 corners (CLI --quad or interactive --click), anchors derived automatically.
  • To bring up a Jetson/Wyze cam: photograph the empty box, run the tool, set PILLBOX_CAMERA_PROFILE, and DoG/CNN/YOLO all work against it.

2. Power measurement — honest Orin Nano path (9e19b12)

The Jetson tegrastats reader returns None on the Orin Nano/NX (those modules drop the onboard INA3221 sensors), so the paper would silently get no power there.

  • Adds an external-meter source (PILLBOX_POWER_METER_CMD: any command that prints watts — USB meter, smart plug, bench PSU) as the highest-trust reader — the real way to get board power on the Orin Nano.
  • Pi 5 PMIC and Jetson tegrastats paths unchanged; paper_stats now reports the method used and tells the operator to set the meter var on a Jetson with no rails instead of implying power was captured.

3. ESP32-S3 on-device CNN inference (8e91125)

Phase-2 goal of running detectors on the board itself.

  • convert_cnn_to_tflite.py: pill_classifier.onnx → int8 TFLite → C header, int8-calibrated on real 6-channel cells.
  • main/main.cpp: esp-tflite-micro harness — runs the CNN on-device, reports latency / arena usage / prediction over serial.
  • ESP-IDF scaffolding (pinned esp-tflite-micro, PSRAM config) + README with a model-fit analysis measured from the shipped ONNX: Ref-CNN (79.6k params, ~78 KB int8) is the primary fit; DoG ports to C; YOLO (1.44 M) fits PSRAM but is a slow stretch goal.

Validation status

  • Tested here (40 unit tests, all pass): profile load/validate, anchor derivation, calibrate tool, PMIC/tegrastats/external-meter parsing + source priority, the conversion script's representative-set + C-header logic. imx708 profile asserted to reproduce originals; DoG pipeline smoke-tested end-to-end.
  • Needs hardware to validate (called out in code/README): the ESP32-S3 firmware (compile + arena/op tuning) and the TF-side ONNX→TFLite conversion. Written to standard APIs, not run on a board here.

🤖 Generated with Claude Code


Generated by Claude Code

claude added 3 commits July 26, 2026 01:14
Box-finding + warp were hard-coded to one imx708 capture, so the detection
stack could not locate the grid on any other camera. This lifts that
calibration into a profile: detect/profiles/<name>.json describing the
reference photo, the 21-cell quad, the match anchors and grid layout, selected
by PILLBOX_CAMERA_PROFILE (default imx708).

- camera_profiles.py loads + validates the active profile.
- crop_cells.py now sources REF_IMAGE/REF_QUAD/ANCHORS/MATCH_SCALE/CELL_*/DAYS/
  SLOTS from it under the same module names, so every caller is untouched;
  classify_cells and train_classifier derive REF_STEM the same way.
- profiles/imx708.json reproduces the original constants bit-for-bit — the Pi
  deployment and every committed pillbox-data number are unchanged (asserted).
- calibrate_camera.py generates a profile for a new camera from one empty-box
  photo: you mark the 4 corners (CLI --quad or interactive --click) and the two
  match anchors are derived automatically.
- draw_debug + grid dims are now grid-size-agnostic.

To bring up a Jetson/Wyze (or ESP32) camera: photograph the empty box, run
calibrate_camera.py, then PILLBOX_CAMERA_PROFILE=<name> and DoG/CNN/YOLO all
work against it. Tests cover profile load/validate, anchor derivation and the
calibrate tool. Script- and module-mode imports both handled.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9bSKH2BmvbZJD4up1jofL
The Jetson tegrastats reader returns None on the Orin Nano/NX because those
modules drop the onboard INA3221 rail sensors older Jetsons had — so the paper
would silently get no power there. Adds an external-meter source
(PILLBOX_POWER_METER_CMD: any command that prints watts — USB meter, smart
plug, bench PSU) as the highest-trust reader, which is the real way to get
board power on the Orin Nano. Pi 5 PMIC and Jetson tegrastats paths unchanged.

paper_stats now reports the external-meter method and, on a Jetson with no
tegrastats rails, tells the operator to set PILLBOX_POWER_METER_CMD instead of
implying power was captured. Tests cover PMIC V*I summing, tegrastats mW/W
parsing, the Orin-Nano no-rails case, external-meter parsing, and source
priority.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9bSKH2BmvbZJD4up1jofL
Phase-2 bring-up for running detectors on the ESP32-S3 itself. Adds:

- convert_cnn_to_tflite.py: detect/pill_classifier.onnx -> int8 TFLite -> C
  header, with an int8 representative set built from real 6-channel cells
  (cell RGB + empty-reference RGB) via onnx2tf + tensorflow. The
  representative-dataset builder and C-array emitter are unit-checked here; the
  TF conversion runs in a desktop/CI env.
- main/main.cpp: esp-tflite-micro harness that loads the model, runs the CNN on
  a 128x96x6 input, and reports latency, tensor-arena usage and prediction over
  serial. Op resolver covers the CNN's ops (Conv/MaxPool/ReLU/GAP/FC/softmax).
- ESP-IDF project scaffolding (CMakeLists, pinned esp-tflite-micro managed
  component, PSRAM sdkconfig) + .gitignore for the generated model header.
- README with the model-fit analysis measured from the shipped ONNX: Ref-CNN
  (79.6k params, ~78KB int8) is the primary fit; DoG ports to plain C; YOLO
  (1.44M, ~1.4MB) fits PSRAM but is a slow stretch goal. Includes the int8
  accuracy check and an on-hardware validation checklist.

Reference implementation: Python-side logic is tested; the firmware is written
to the esp-tflite-micro API but needs a board to compile/validate (called out
in the README).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9bSKH2BmvbZJD4up1jofL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants