ToddleAI

On-device toddler gait observation — nothing about your child ever leaves the phone.

🏆 3rd place — ExecuTorch Hackathon 2026 (Qualcomm × Meta), San Francisco, June 27–28, out of 30 teams selected from hundreds of applicants.

---

What It Does

You record a short, guided video of your toddler walking. ToddleAI checks the recording quality in real time as you shoot — if the feet aren't visible, the camera is shaking, or your child isn't walking side-on, it tells you exactly what to fix before you even stop recording. Once you have a good clip, it extracts step-by-step timing from the walk (step time, cadence, left/right symmetry) and compares them to published pediatric norms. You can then ask an on-device assistant to explain the results in plain language. Every step — video, pose estimation, gait math, and chat — runs locally on the phone; the app requests no internet permission.

Demo

A sample walking clip used for development and testing is included at video/Toddler_walking_in_blue_dress_202606280214.mp4, and a second test clip lives at app/src/androidTest/assets/toddler.mp4. There is no separate rejection screen — a rejected recording routes to the Results screen, which shows why the clip was withheld and offers Record Again.

Demo flow as implemented:

1. Welcome — enter the child's age, then either Record Walking Video or Import Test Video From Device.
2. Capture — live camera preview with a MediaPipe skeleton overlay, a framing guide-zone, three quality dots ("feet" / "stable" / "body"), and a running step counter. Recording auto-stops at 15 seconds.
3. Analyzing — the recorded pose frames are re-run through the precision (two-pass) gait pipeline.
4. Results — either observation cards (cadence, symmetry, step-rhythm variability) or, if the clip didn't meet quality bar, a plain-language explanation of what to fix and a Record Again button.
5. Chat — tap Ask ToddleAI to ask the on-device Llama assistant about the results, grounded in the actual measurements from that clip.

No screenshots are checked into the repository; the flow above is derived directly from NavGraph.kt and the screen composables.

Architecture

                    ┌───────────────────────────── on-device, no network ─────────────────────────────┐
                    │                                                                                   │
  CameraX (30 fps) ─┼─► FrameProcessor ─► PoseEstimator ─► QualityGate.assessFrame ─► GuidanceEngine /  │
   Camera2Interop    │   (bitmap +          (MediaPipe          (per-frame:            FramingGuide     │
   pinned to 30fps   │    rotation)      PoseLandmarker,      feet/body/stability)    → skeleton +      │
                    │                    XNNPACK CPU/GPU)                              coaching overlay  │
                    │                                                                        │          │
                    │                buffered pose frames (recording, ≤15s)                  ▼          │
                    │                              │                              live guidance text,   │
                    │                              ▼                              step counter (UI)     │
                    │              ┌────────────────────────────────┐                                  │
                    │              │   ReplayAnalyzer (2nd pass)     │                                  │
                    │              │  best-segment extraction        │                                  │
                    │              │  → 5-frame smoothing            │                                  │
                    │              │  → GaitEventDetector (heel-     │                                  │
                    │              │     strike peak finder, L/R)    │                                  │
                    │              │  → QualityGate.assessRecording  │                                  │
                    │              │     (HIGH/MEDIUM/LOW/REJECT)    │                                  │
                    │              └───────────────┬──────────────────┘                                │
                    │                          REJECT? ──► withhold analysis, show why (Results screen) │
                    │                              │ else                                                │
                    │                              ▼                                                     │
                    │                  MetricComputer (cadence, step time,                               │
                    │                  L/R symmetry, step-time CoV)                                      │
                    │                              │                                                     │
                    │                              ▼                                                     │
                    │                  ObservationEngine + GaitNorms                                     │
                    │                  (Typical / Elevated vs age-band norms)                            │
                    │                              │                                                     │
                    │                              ▼                                                     │
                    │              ResultsScreen  ──►  ChatScreen                                        │
                    │                                   PromptBuilder → LlmEngine                        │
                    │                                   (Llama 3.2 1B, ExecuTorch                        │
                    │                                    XNNPACK/CPU, streamed tokens)                   │
                    └───────────────────────────────────────────────────────────────────────────────────┘

  Imported clip:  OpenDocument (video/*) ─► VideoFrameDecoder (linear MediaCodec, ~15 fps) ─► same
                  PoseEstimator + QualityGate + ReplayAnalyzer pipeline as above.

In-app path (what actually ships in the APK): pose estimation runs through MediaPipe Tasks PoseLandmarker, backed by TFLite + the XNNPACK CPU delegate (a GPU delegate is available but CPU is default). The on-device LLM chat runs through ExecuTorch's XNNPACK/CPU backend as well — see NPU / ExecuTorch Integration for why QNN isn't used for either workload in the shipping app.

Adb-shell NPU benchmark path (separate from the app): samples/pose/ and run_qwen_npu.sh drive Qualcomm-exported .pte models through qnn_executor_runner / qnn_llama_runner directly over adb shell, exercising the Hexagon NPU outside the installed app's sandbox. This is the dual-path design: one path is what a parent's phone actually runs; the other is the NPU capability proof that the installed app itself cannot reach (see below).

Two-pass design: during recording, FrameProcessor runs a lightweight per-frame quality check and feeds GuidanceEngine/FramingGuide for live coaching text and overlay color — no gait metrics are computed live. After recording (or after importing a clip), ReplayAnalyzer re-runs event detection twice: once on the raw buffered frames to pick the best contiguous "good" segment, then again on a 5-frame-smoothed, landmark-filtered version of that segment to compute the final metrics. Imported videos go through the exact same ReplayAnalyzer as live recordings.

Models

Model	Purpose	Framework	Backend	Size	File
PoseLandmarker (BlazePose Full, 33 landmarks)	Gait pose estimation — the only on-device model that outputs feet/lower-body landmarks, so it is the one actually used for gait	MediaPipe Tasks (TFLite)	XNNPACK CPU (default) or GPU	9.4 MB	app/src/main/assets/pose_landmarker_full.task
Llama 3.2 1B Instruct (SpinQuant)	On-device chat assistant that explains results	ExecuTorch	XNNPACK/CPU	— (pushed at runtime, not bundled)	/data/local/tmp/llm/llama32_1b_xnnpack_spinquant.pte + llama32_tokenizer.json
MediaPipe-Pose landmark stage (AI-Hub export, CPU)	NPU-benchmark reference only — outputs landmarks 0–24 (head→hips), no feet, cannot drive gait	ExecuTorch	XNNPACK/CPU	13.5 MB	app/src/main/assets/pose_landmark_cpu.pte
MediaPipe-Pose landmark stage (AI-Hub export, QNN)	NPU-benchmark reference only, same landmark limitation as above	ExecuTorch	Qualcomm QNN/NPU	17.3 MB	app/src/main/assets/pose_landmark_qnn.pte
MediaPipe-Pose detector stage (CPU / QNN)	Detector half of the two-stage AI-Hub pose pipeline used only by the samples/pose adb benchmark, not bundled in the app	ExecuTorch	XNNPACK/CPU, QNN/NPU	3.3 MB / 5.3 MB	build-artifacts/pose_detector_cpu.pte, build-artifacts/pose_detector_qnn.pte
Qwen3-1.7B (hybrid QNN)	Separate NPU-only chat benchmark, run outside the app via run_qwen_npu.sh	ExecuTorch	Qualcomm QNN/NPU (Hexagon, adb-shell domain)	— (external, not committed)	hybrid_llama_qnn.pte on-device at /data/local/tmp/qwenrun

No latency or throughput numbers are hard-coded or committed anywhere in this repository (checked the analysis, LLM, and settings code, the technical architecture doc, and the export-starter logs). The Settings screen has a benchmark comparison table wired to an optional qnn_benchmarks.json, but that file isn't present in this checkout — running the NPU benchmark scripts yourself is currently the only way to get real numbers on your device. See NPU / ExecuTorch Integration.

Key Features

• Guided Capture. CaptureScreen overlays a live MediaPipe skeleton, a dashed framing guide-zone, an animated corner/arrow prompt (FramingArrowOverlay) telling you to move back, tilt down, pan left/right, etc., and a step counter that pops as steps are detected. GuidanceEngine drives a running coaching message ("3 good steps — 2 more needed", "Great capture!") based on a rolling window of per-frame quality.
• Quality Gate. Rejects bad recordings and tells the parent exactly why — see its own section below. This is ToddleAI's signature feature.
• Temporal Gait Metrics. Computed per clip: cadence (steps/min, from median step time), left/right step-time symmetry (mean timing difference and asymmetry %), and step-time variability (coefficient of variation, a rhythm-consistency measure). All are temporal only — no stride length, joint angles, or walking speed are computed.
• On-Device LLM Agent. A Llama 3.2 1B Instruct model, run through ExecuTorch's LlmModule, answers parent questions grounded in that clip's actual observations (injected via PromptBuilder.gaitContext). It is a single grounded chat completion today, not a multi-tool agent — LlamaAgent.kt and ToolRouter.kt are present as stubs for a planned tool-calling layer but contain no implementation yet.
• Privacy by architecture. The manifest requests only CAMERA — no INTERNET permission exists anywhere in the app, so no gait data, video, or chat text can leave the device even if the code tried. The Settings screen verifies and displays this at runtime.

The Quality Gate

The quality gate is what makes ToddleAI trustworthy rather than just impressive: it withholds a result rather than guessing on a clip that isn't good enough to guess from.

Per-frame checks (QualityGate.assessFrame), evaluated on every posed frame:

• Feet visibility — both heels and both foot-index landmarks must have MediaPipe visibility > 0.5. If not, the frame is REJECTED outright.
• Major landmark visibility — hips, knees, ankles, heels, and foot-index (10 points) must all be > 0.4 for a frame to count as fully usable.
• Full-body framing — shoulders and ankles must all be > 0.3 visible (child isn't cropped out of frame).
• Camera stability — frame-to-frame shoulder-width and torso-height must each shift by no more than 18%; a bigger shift means the camera moved or shook, not the child.
• Mean landmark confidence — must exceed 0.4 overall or the frame is rejected regardless of the individual checks above.

Each frame is bucketed GOOD / PARTIAL / REJECTED from these checks.

Recording-level gate (QualityGate.assessRecording), evaluated once at the end of a clip:

• Counts detected steps and the ratio of GOOD frames, and finds the longest run of consecutive GOOD frames.
• HIGH confidence: ≥6 steps, >70% good frames, a ≥30-frame good segment.
• MEDIUM: ≥5 steps, >50% good frames, ≥30-frame segment.
• LOW: ≥3 steps, everything else looser.
• REJECT: fewer than 3 detected steps — analysis is withheld entirely; ReplayAnalyzer returns metrics = null and Results shows the reason instead of numbers.
• It also flags a likely wrong walking direction (child facing the camera instead of walking side-on) when a third of frames show both feet and full body visible but not all major landmarks confidently — the signature pattern of a front-facing shot — and tells the parent to "record from the side."

When the gate rejects, the app doesn't fail silently or show placeholder numbers — it shows the specific reason ("Feet were hidden in 42% of frames. Record from knee height.", "Camera movement detected. Hold the phone still or prop it up.", "Only 2 steps detected. Record a longer walking sequence.") and offers Record Again. This is deliberate: a system that can say "I don't have enough to go on" is more scientifically responsible than one that always produces a confident-looking answer.

Scientific Foundation

ToddleAI's approach — markerless, pose-estimation-based gait analysis from smartphone video — is grounded in peer-reviewed research on this exact class of method:

1. Anderson et al. (2025). Validation of markerless video-based gait analysis using pose estimation in toddlers. Frontiers in Digital Health. Validated in 112 toddlers; step-time correlation with instrumented ground truth reached r = 0.96.
2. Stenum, Rossi & Roemmich (2021). Two-dimensional video-based analysis of human gait using pose estimation. PLOS Computational Biology.
3. Gao et al. (2023). Automating General Movements Assessment with quantitative deep learning to facilitate early screening of cerebral palsy. Nature Communications. Reported AUC = 0.967 for movement-assessment classification.

Cadence reference ranges used by GaitNorms.kt to contextualize an individual clip's cadence are drawn from Rygelova et al. (2023, PLOS ONE), Sutherland's walking-maturation literature, and GAITRite pediatric normative data.

ToddleAI is a research prototype inspired by this work. It is not itself clinically validated. The papers above validate the feasibility of the approach — markerless pose-based gait analysis — under controlled research conditions with different models, cohorts, and hardware. ToddleAI has not been independently validated against instrumented ground truth, and this README does not and should not claim it has.

NPU / ExecuTorch Integration

ToddleAI targets ExecuTorch as the on-device inference runtime for both pose and LLM workloads, but the installed app and the Hexagon NPU benchmark currently take two different, deliberately separated paths — worth explaining honestly rather than glossing over:

• Why not QNN in the installed app: the target device (Samsung Galaxy S25 Ultra, Snapdragon 8 Elite / SM8750) blocks the installed application sandbox from the Hexagon cDSP under Samsung's SELinux policy. The adb shell domain is not subject to that restriction. This is documented directly in code comments (LlmEngine.kt, run_qwen_npu.sh, samples/pose/README.md) — it's the reason the in-app LLM and in-app pose model both run on XNNPACK/CPU rather than QNN/NPU, and why the NPU proof lives in adb-shell tooling instead.
• .pte export path (toddleai_pte_export_starter/): a staged Docker workflow that (1) builds a Linux x86_64 container with the Qualcomm QNN SDK mounted in, (2) clones and builds ExecuTorch v1.3.1 with the Qualcomm backend, (3) exports the MediaPipe Pose detector and landmark stages via scripts/export_mediapipe_pose.py, with thin wrappers export_xnnpack.py (XNNPACK backend) and export_qnn_fp16.py (QNN backend, targeting SoC SM8750), and (4) builds the Android ExecuTorch AAR with QNN support via scripts/build_android_aar_qnn.sh, which wraps upstream's scripts/build_android_library.sh.
• QNN NPU benchmark (adb-shell path): samples/pose/run_pose.py runs the two-stage AI-Hub MediaPipe-Pose export end to end on-device: pose_detector_qnn.pte finds the person and ROI, pose_landmark_qnn.pte produces 25 landmarks, both executed via qnn_executor_runner pushed to /data/local/tmp/poserun, with all pre/post-processing (letterboxing, NMS, ROI affine mapping, skeleton drawing) done host-side in Python. run_qwen_npu.sh does the equivalent for a Qwen3-1.7B hybrid QNN LLM via qnn_llama_runner at /data/local/tmp/qwenrun.
• QNN SDK / toolchain, as configured in this checkout: QNN SDK 2.47.0 (backend API 2.18.0), ExecuTorch v1.3.1, Android NDK 26c (upstream-validated; the export-starter's NDK 27.2.12479018 is a fallback that may need swapping), target SoC SM8750, Hexagon v79 HTP libraries (libQnnHtp.so, libQnnHtpV79Skel.so, libQnnHtpV79Stub.so, libQnnSystem.so in app/src/main/jniLibs/arm64-v8a/).
• Measured CPU-vs-NPU numbers: none are committed in this repository. The Settings screen (SettingsScreen.kt) has a benchmark comparison table designed to read an optional qnn_benchmarks.json, but that file is not present in this checkout — the honest state is "the plumbing for a CPU/NPU comparison exists; the numbers have to be generated by running the benchmark scripts on your own device." See the ExecuTorch Qualcomm backend docs for how delegation and profiling are normally captured: executorch/docs/source/backends-qualcomm.md (bundled in this repo's executorch/ checkout) or the upstream ExecuTorch Qualcomm backend documentation.

Tech Stack

• Language: Kotlin 2.0.20
• UI: Jetpack Compose (Compose BOM 2024.09.02), Material 3, Navigation Compose 2.8.0
• Camera: CameraX 1.4.2 (camera-core, camera-camera2, camera-lifecycle, camera-view), pinned to 30 fps via Camera2Interop
• On-device inference: ExecuTorch (executorch.aar, XNNPACK backend in-app), MediaPipe Tasks Vision 0.10.14
• Models: MediaPipe PoseLandmarker Full (BlazePose, 33 landmarks), Llama 3.2 1B Instruct (SpinQuant, ExecuTorch)
• Async/state: Kotlin Coroutines 1.10.2, AndroidViewModel + StateFlow
• Persistence: Jetpack DataStore Preferences 1.1.1 (inference-backend preference only)
• Build: Android Gradle Plugin 8.5.2, Gradle Kotlin DSL, compileSdk 34, minSdk 26, targetSdk 34
• ABI: arm64-v8a only (ExecuTorch/QNN ship arm64-only and the target device is arm64; restricting ABIs avoids bloating the APK with unusable MediaPipe x86/armeabi-v7a code)
• Target/dev device: Samsung Galaxy S25 Ultra, Snapdragon 8 Elite (SM8750), Hexagon NPU v79
• Java target: 17

Build & Run Instructions

1. Prerequisites

• Android Studio (recent stable) or the Android command-line tools
• JDK 17 (export JAVA_HOME=$(/usr/libexec/java_home -v 17) on macOS) — required by AGP 8.5.2
• Android SDK with compileSdk 34 / minSdk 26 installed
• A physical arm64 Android device for testing (an emulator cannot run the ExecuTorch/QNN native libraries or exercise the camera-driven capture flow meaningfully)

2. Clone the repo

git clone <this-repo-url>
cd toddle-ai

3. Model setup

The MediaPipe pose model is already checked into app/src/main/assets/pose_landmarker_full.task. If it's ever missing, re-download it (~9 MB):

curl -L -o app/src/main/assets/pose_landmarker_full.task \
  https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_full/float16/latest/pose_landmarker_full.task

For the ExecuTorch pieces, this repo is app source only — three artifacts are supplied separately because they're large/proprietary and not committed:

Artifact	Where it goes	Source
executorch.aar (built with QNN support)	build-artifacts/executorch.aar, or set executorchAarPath / EXECUTORCH_AAR_PATH	Build with toddleai_pte_export_starter/scripts/build_android_aar_qnn.sh
Qualcomm QNN .so libs (Hexagon v79)	app/src/main/jniLibs/arm64-v8a/, auto-staged from QNN_SDK_ROOT during preBuild if set	QNN SDK 2.47.0
llama32_1b_xnnpack_spinquant.pte + llama32_tokenizer.json	pushed to the device, not the APK: adb push ... /data/local/tmp/llm/	your exported Llama 3.2 1B Instruct SpinQuant package

In local.properties (already gitignored):

sdk.dir=/absolute/path/to/Android/sdk
qnnSdkRoot=/absolute/path/to/qairt/2.47.0.260601
# optional override if you keep the AAR elsewhere:
# executorchAarPath=/absolute/path/to/executorch.aar

4. Build

export JAVA_HOME=$(/usr/libexec/java_home -v 17)
./gradlew :app:installDebug

Or open the project folder in Android Studio and hit Run.

5. Install on device

installDebug above installs directly over USB debugging. Alternatively, build the debug APK (./gradlew :app:assembleDebug) and adb install app/build/outputs/apk/debug/app-debug.apk.

6. Push the LLM model

adb shell mkdir -p /data/local/tmp/llm
adb push llama32_1b_xnnpack_spinquant.pte /data/local/tmp/llm/
adb push llama32_tokenizer.json /data/local/tmp/llm/

7. First run

Launch the app, enter the child's age, and record (or import) a side-view clip with both feet visible for the full walk. Without the LLM files pushed, the Chat screen will show a "model missing" status, but the capture/quality-gate/metrics flow works independently.

Project Structure

app/src/main/kotlin/com/toddleai/app/
├── MainActivity.kt              # single-activity host; discards saved nav state on restore
├── ToddleAIApp.kt                # Application class (currently an empty stub)
├── ToddleAISessionViewModel.kt   # owns the whole capture → analysis → chat pipeline
├── navigation/
│   └── NavGraph.kt               # Welcome → Capture → Analyzing → Results → Chat, + Settings
├── capture/
│   ├── CameraManager.kt          # CameraX Preview + ImageAnalysis, 30fps pin, permission handling
│   ├── FrameProcessor.kt         # per-frame pose → quality → guidance pipeline (live pass)
│   └── VideoRecorder.kt          # stub — analysis runs off buffered pose frames, not a saved file
├── analysis/
│   ├── PoseEstimator.kt          # routes .task/.tflite/.pte to the right backend
│   ├── MediaPipePoseLandmarker.kt# the model actually used: BlazePose full, XNNPACK/GPU
│   ├── QualityGate.kt            # per-frame + per-recording quality checks (see above)
│   ├── GaitEventDetector.kt      # heel-strike peak detection, L/R, walking-direction estimate
│   ├── MetricComputer.kt         # cadence, symmetry, step-time CoV
│   ├── ObservationEngine.kt      # metrics → parent-facing Typical/Elevated observations
│   ├── ReplayAnalyzer.kt         # two-pass precision analysis (post-recording)
│   ├── ImportedVideoAnalyzer.kt  # decodes an imported clip, feeds the same ReplayAnalyzer
│   ├── VideoFrameDecoder.kt      # linear MediaCodec decode (avoids a known seek deadlock)
│   ├── FramingGuide.kt           # live single-hint framing coaching (move back/closer/pan/etc.)
│   └── GuidanceEngine.kt         # live step-count coaching + quality-level state
├── data/
│   ├── GaitNorms.kt               # published pediatric cadence bands by age
│   ├── SessionRepository.kt       # stub — no persistence implemented yet
│   └── models/                    # PoseFrame, FrameQuality, GaitEvent, TemporalMetrics, Observation, CaptureAssessment
├── llm/
│   ├── LlmEngine.kt               # wraps ExecuTorch LlmModule, single-thread, XNNPACK/CPU
│   ├── PromptBuilder.kt           # system persona + gait-context injection, Llama-3 chat template
│   ├── LlamaAgent.kt              # stub — planned session/runtime management layer
│   └── ToolRouter.kt              # stub — planned tool-calling layer
├── settings/InferenceSettings.kt  # persisted XNNPACK/QNN backend preference (DataStore)
└── ui/
    ├── screens/                   # WelcomeScreen, CaptureScreen, AnalyzingScreen, ResultsScreen, ChatScreen, SettingsScreen
    └── components/                # SkeletonOverlay, GuidanceOverlay, FramingArrowOverlay, ObservationCard, ...

toddleai_pte_export_starter/   # Docker-based ExecuTorch/QNN .pte export pipeline (pose models, AAR build)
samples/pose/                  # standalone adb-shell QNN NPU benchmark for the two-stage pose model
TODDLEAI_TECHNICAL_ARCHITECTURE.md  # full end-to-end architecture/design spec written pre-implementation

Limitations

• Temporal metrics only. Cadence, step time, and left/right symmetry are computed; there is no stride length, walking speed, joint-angle, or distance-based metric in this build.
• Pose model trained on adults. BlazePose was trained primarily on adult gait; accuracy on a toddler's smaller body and faster, less mature gait pattern is reduced relative to the adult populations the model was validated on.
• The quality gate can reject valid recordings in low light, cluttered backgrounds, or unusual clothing that reduces landmark visibility, even when the walking itself was fine.
• The LLM assistant is a single-turn grounded chat, not a tool-using agent yet — LlamaAgent.kt and ToolRouter.kt are unimplemented stubs for a planned tool-calling layer. Responses may occasionally be generic or not perfectly on-topic.
• No video or session persistence. SessionRepository.kt is a stub; nothing is saved between app launches today.
• This is a research prototype, not a clinical tool. It is not intended for diagnosis or medical decision-making, and observations should never replace a pediatrician's assessment.

What We Learned

Building ToddleAI in a 36-hour hackathon window surfaced a few ideas we think matter beyond this project:

• Knowing when not to answer is a feature, not a gap. The quality gate — refusing to produce gait numbers from a clip that doesn't meet the bar, and explaining exactly why — turned out to be more technically and scientifically interesting than the metrics themselves.
• Privacy is an architectural decision, not a checkbox. Removing the INTERNET permission entirely, rather than promising not to use it, forced every design decision (model choice, storage, LLM) to actually be on-device from the start.
• Scientific rigor and demo simplicity pull in opposite directions. Citing real validation literature while being honest that this specific system hasn't been validated required constant discipline against the instinct to round up a hackathon demo into a bigger claim than it's earned.

Team

Built by preyashyadav and Cody.

License

MIT — see LICENSE.