On-Device Disease Detection (WebAssembly/ONNX)

[jpdevhub]

Currently, disease detection requires uploading an image to the Python backend. Convert our PyTorch ResNet18 model to ONNX format and run it directly in the browser using onnxruntime-web. This will allow offline disease scanning instantly on the farmer's phone.

[github-actions]

👋 Thanks for opening this issue, @jpdevhub!

We'll take a look soon. In the meantime, please make sure you've included steps to reproduce (for bugs) or a clear problem statement (for feature requests).

— The AgroNavis team

[Excalibur677]

Hey @jpdevhub,

I can take this one on! Running the ResNet18 model completely client-side via WebAssembly is going to make the offline functionality of the PWA so much better.

Since we need to bridge the PyTorch model over to the browser, here is how I plan to approach the implementation:

1. Model Conversion: I'll write a quick utility script in the backend to export the existing fine-tuned plant_disease_resnet18.pth weights into an .onnx format using torch.onnx.export.
2. Quantization (Optional but highly recommended): Since this needs to run smoothly on older mobile browsers, I can look into applying float16 or INT8 quantization to shrink the model file size down so it loads instantly over cellular data.
3. Frontend Integration (onnxruntime-web): On the Next.js side, I'll integrate onnxruntime-web to load the model file. I'll create a dedicated React utility/hook that handles the canvas image preprocessing (resizing the farmer's uploaded/camera photo to $224 \times 224$, normalizing the RGB channels) and runs the inference entirely on-device.

This will completely eliminate the latency of uploading raw images to the FastAPI server.

Could you assign this to me? I'll start with the Python export script first and share the model size benchmarks with you here.

[vedant-kawale-27]

Here is the high-level implementation strategy and architectural flow:

---

1. Model Export & Architecture Preparation

Before the browser can handle the neural network, the model must be translated into an open, cross-platform standard.

• Freeze the Model Weights: Ensure the PyTorch ResNet18 model is fully trained and switched to evaluation mode so that dynamic behaviors (like dropout or batch normalization) are locked down.
• Graph Conversion: Convert the model graph into the ONNX (Open Neural Network Exchange) format. This compiles the PyTorch mathematical operations into a standardized, optimized format that non-Python runtimes can read.
• Expose Standard Node Names: Clearly label the input layer (e.g., "input") and output layer (e.g., "output") during conversion so the JavaScript application knows exactly where to feed data and where to read predictions.

---

2. Client-Side Asset Strategy

Since the .onnx model file acts as a static asset (similar to a large image or video file), it needs to be delivered efficiently to the client side.

• Static Hosting: Place the exported ONNX file into the public asset directory of your frontend application.
• Progressive Web App (PWA) Caching: To ensure the system works deep in rural fields without cell reception, configure your application's Service Worker. The service worker must intercept the initial download and cache the entire ONNX model file directly in the phone’s local browser storage (Cache Storage API) upon first installation.

---

3. Client-Side Preprocessing Pipeline

A major hurdle in edge AI is preparing raw sensor data. The browser must precisely replicate how Python prepared the images during training, using native web tools:

• Image Resizing: Use an invisible HTML5 <canvas> element to grab an uploaded picture or a live camera frame and forcefully downscale it to the exact dimensions the ResNet18 model expects (typically $224 \times 224$ pixels).
• Pixel Extraction: Extract the raw layout array from the canvas context.
• Data Normalization: Translate the standard $0-255$ integer pixel channels into floating-point decimals scaled between $0.0$ and $1.0$. Apply the exact mathematical standard deviation and mean normalization offsets that PyTorch used during training.
• Channel Rearrangement: Reorder the image layout. Web browsers store images sequentially as interleaved pixel colors (Red, Green, Blue, Alpha, Red, Green, Blue, Alpha...). PyTorch models expect planar data structure format (all Reds first, then all Greens, then all Blues). Your frontend script must manually restructure this data array before feeding it onward.

---

4. Edge Inference Execution

Once the data array is structured correctly, the browser's execution engine takes over.

• Inference Session Initialization: Use the onnxruntime-web library to create a virtual runtime engine inside the browser tab, pointing directly to your cached model file.
• WASM Acceleration: Ensure the runtime leverages WebAssembly (WASM) and multi-threading capabilities. This allows the browser to execute the model's heavy matrix multiplications at near-native speeds directly using the phone's CPU/GPU capabilities rather than slow, interpreted JavaScript.
• Input Binding: Pass the structured multi-dimensional array into the input node defined during step one.

---

5. Post-Processing & UI Response

The model's output isn't a neat text string like "Tomato Late Blight"; it is an array of raw numerical scores (logits).

• Argmax Sorting: Run a basic array sweep on the output to locate the index containing the highest numerical value (the class the model is most confident in).
• Index Mapping: Map that index position directly to a local frontend array containing human-readable labels (e.g., Index 0 maps to "Healthy Leaf", Index 1 maps to "Leaf Rust").
• Immediate UI Update: Render the text result and confidence score instantly on the user's dashboard without initiating a single HTTP network request.

hee @jpdevhub this is my implementation steps, please assign this issue to me underr SSoC26

[arcgod-design]

Hi @jpdevhub! I'd love to work on this feature as part of SSoC '26.

My Approach:

• Export the existing ResNet18 disease classification model to ONNX format and quantize it for size/speed
• Load and run the model in-browser using onnxruntime-web via WebAssembly for true offline inference
• Update CropScanTab.tsx to detect network status and route to on-device or server-side inference accordingly
• Benchmark latency and accuracy parity vs. the server model; document results in the PR

Could you please assign this to me (@arcgod-design)? Thank you!

[jpdevhub]

@vedant-kawale-27 go ahead