Phase 2 groundwork: Jetson detection, honest power, ESP32-S3 CNN inference - #31
Open
tarun101 wants to merge 3 commits into
Open
Phase 2 groundwork: Jetson detection, honest power, ESP32-S3 CNN inference#31tarun101 wants to merge 3 commits into
tarun101 wants to merge 3 commits into
Conversation
Box-finding + warp were hard-coded to one imx708 capture, so the detection stack could not locate the grid on any other camera. This lifts that calibration into a profile: detect/profiles/<name>.json describing the reference photo, the 21-cell quad, the match anchors and grid layout, selected by PILLBOX_CAMERA_PROFILE (default imx708). - camera_profiles.py loads + validates the active profile. - crop_cells.py now sources REF_IMAGE/REF_QUAD/ANCHORS/MATCH_SCALE/CELL_*/DAYS/ SLOTS from it under the same module names, so every caller is untouched; classify_cells and train_classifier derive REF_STEM the same way. - profiles/imx708.json reproduces the original constants bit-for-bit — the Pi deployment and every committed pillbox-data number are unchanged (asserted). - calibrate_camera.py generates a profile for a new camera from one empty-box photo: you mark the 4 corners (CLI --quad or interactive --click) and the two match anchors are derived automatically. - draw_debug + grid dims are now grid-size-agnostic. To bring up a Jetson/Wyze (or ESP32) camera: photograph the empty box, run calibrate_camera.py, then PILLBOX_CAMERA_PROFILE=<name> and DoG/CNN/YOLO all work against it. Tests cover profile load/validate, anchor derivation and the calibrate tool. Script- and module-mode imports both handled. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9bSKH2BmvbZJD4up1jofL
The Jetson tegrastats reader returns None on the Orin Nano/NX because those modules drop the onboard INA3221 rail sensors older Jetsons had — so the paper would silently get no power there. Adds an external-meter source (PILLBOX_POWER_METER_CMD: any command that prints watts — USB meter, smart plug, bench PSU) as the highest-trust reader, which is the real way to get board power on the Orin Nano. Pi 5 PMIC and Jetson tegrastats paths unchanged. paper_stats now reports the external-meter method and, on a Jetson with no tegrastats rails, tells the operator to set PILLBOX_POWER_METER_CMD instead of implying power was captured. Tests cover PMIC V*I summing, tegrastats mW/W parsing, the Orin-Nano no-rails case, external-meter parsing, and source priority. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9bSKH2BmvbZJD4up1jofL
Phase-2 bring-up for running detectors on the ESP32-S3 itself. Adds: - convert_cnn_to_tflite.py: detect/pill_classifier.onnx -> int8 TFLite -> C header, with an int8 representative set built from real 6-channel cells (cell RGB + empty-reference RGB) via onnx2tf + tensorflow. The representative-dataset builder and C-array emitter are unit-checked here; the TF conversion runs in a desktop/CI env. - main/main.cpp: esp-tflite-micro harness that loads the model, runs the CNN on a 128x96x6 input, and reports latency, tensor-arena usage and prediction over serial. Op resolver covers the CNN's ops (Conv/MaxPool/ReLU/GAP/FC/softmax). - ESP-IDF project scaffolding (CMakeLists, pinned esp-tflite-micro managed component, PSRAM sdkconfig) + .gitignore for the generated model header. - README with the model-fit analysis measured from the shipped ONNX: Ref-CNN (79.6k params, ~78KB int8) is the primary fit; DoG ports to plain C; YOLO (1.44M, ~1.4MB) fits PSRAM but is a slow stretch goal. Includes the int8 accuracy check and an on-hardware validation checklist. Reference implementation: Python-side logic is tested; the firmware is written to the esp-tflite-micro API but needs a board to compile/validate (called out in the README). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q9bSKH2BmvbZJD4up1jofL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up the Jetson/Wyze/ESP32 camera support with the pieces that make those platforms actually usable for Phase 2 (Orin Nano + ESP-CAM). Three independent commits.
1. Detection on other cameras — calibration profiles (
46ba05f)Box-finding + warp were hard-coded to one imx708 capture, so detection couldn't locate the grid on any other camera. Lifted into a camera profile (
detect/profiles/<name>.json), selected byPILLBOX_CAMERA_PROFILE(defaultimx708).camera_profiles.pyloads/validates the active profile;crop_cells.pyexposes its fields under the same module names, so every caller is untouched.profiles/imx708.jsonreproduces the original constants bit-for-bit (asserted) — the Pi deployment and all committed pillbox-data numbers are unchanged.calibrate_camera.pygenerates a profile for a new camera from one empty-box photo: mark 4 corners (CLI--quador interactive--click), anchors derived automatically.PILLBOX_CAMERA_PROFILE, and DoG/CNN/YOLO all work against it.2. Power measurement — honest Orin Nano path (
9e19b12)The Jetson tegrastats reader returns
Noneon the Orin Nano/NX (those modules drop the onboard INA3221 sensors), so the paper would silently get no power there.PILLBOX_POWER_METER_CMD: any command that prints watts — USB meter, smart plug, bench PSU) as the highest-trust reader — the real way to get board power on the Orin Nano.paper_statsnow reports the method used and tells the operator to set the meter var on a Jetson with no rails instead of implying power was captured.3. ESP32-S3 on-device CNN inference (
8e91125)Phase-2 goal of running detectors on the board itself.
convert_cnn_to_tflite.py:pill_classifier.onnx→ int8 TFLite → C header, int8-calibrated on real 6-channel cells.main/main.cpp: esp-tflite-micro harness — runs the CNN on-device, reports latency / arena usage / prediction over serial.esp-tflite-micro, PSRAM config) + README with a model-fit analysis measured from the shipped ONNX: Ref-CNN (79.6k params, ~78 KB int8) is the primary fit; DoG ports to C; YOLO (1.44 M) fits PSRAM but is a slow stretch goal.Validation status
🤖 Generated with Claude Code
Generated by Claude Code