A Project by Team Inspiration - 1st Place University Team in the 2023 Call for Code Global Challenge - Round 3 (https://developer.ibm.com/blogs/global_challenge_round_3_winners/)
- Project summary
- Technology implementation
- Presentation materials
- Additional details
- About this template
The sustainability challenge our solution aims to address is the global food insecurity caused by the destruction of grains by stored grain pests. PestDetect is designed to circumvent the expensive expertise currently required to identify these stored grain pests and minimize the use of harmful chemical insecticides in granaries.
An AI-powered mobile application that identifies stored grain pests and recommends sustainable management techniques.
Food insecurity is a present day crisis. Stored grain pests - accounting for 9% and 20% of the post-harvest losses in developed and developing countries respectively- are a major cause of the current food grain shortage. We decided to build a solution overcome these losses by directly supporting the people who connect nature to society - our farmers. A massive problem currently faced by them are the expensive resources and scarce expertise available to identify, rectify and prevent stored-grain pest infestations. Most farmers in developing regions of the world are unable to find experts who can identify and help them prevent stored grain pest infestations. Ultimately, this results in the indescriminate use of insecticides, which are harmful to both the environment and humans. However, these farmers would greatly benefit from a grassroots solution that is affordable and promotes sustainable prevention and management techniques specific to the type of pest their grains are being attacked by. Armed with this new-found knowledge, the farmers of the world would be unstoppable, and able to overcome any pest, any where, at any time of the year.
Our solution? - PestDetect! PestDetect is a mobile application that places the key to overcoming stored grain pest infestations into the hands of farmers - literally! With the frontend created using Expo React Native, PestDetect is a simple-to-use, seamless and deployment-ready application that provides two basic functionalities to farmers - PestSnap and PestBot. In PestSnap, farmers can either take or upload a photo of a stored grain pest they have found in their grainary or storage area during one of their routine checks. Once they have cropped the photo, it is sent for predicition to an Image Classification model that is deployed on Watson Machine Learning, through a Flask Web App deployed on IBM Cloud Kubernetes. The model is trained on 7000 images for 7 classes of pests - Grain Moths, Grain Borers, Grain Weevils, Grain & Flour Beetles, Flour Moths, Mealworms and Rodents - by fine-tuning and adding to the prebuilt MobileNetV3 model. Based on the features it analyses from the image, it predicts and returns to the app the most probable category the pest in the image falls under.
This prediction received by the app is sent to the Watson Assistant Chatbot we have created (actions of the chatbot). The chatbot (utilising the new action-based Watson Assistant) has actions to respond to a prediction of one of the seven pest categories, with suitable sustainable management techniques that promote prevention rather than extermination. The responses of the chatbot have been prepared using the latest research in the field of pest grain management (Ahmad R, Hassan S, Ahmad S, et al. (2022) Stored Grain Pests and Current Advances for Their Management. Postharvest Technology - Recent Advances, New Perspectives and Applications. IntechOpen. DOI: 10.5772/intechopen.101503.). However, the average farmer may require some additional clarity on some of the pest prevention and management techniques initially provided to them by the chatbot. In order to address the innumerable number of queries a farmer may have, we have empowered our chatbot using OpenAIs gpt-3.5-turbo model (to be replaced by an LLM from watsonx in the future). By building a custom extension, we call the model and supply it the following:
- a set of system instructions that explain how to respond to the farmer in a simple and comprehensible manner. We also instruct it to respond with a 'Sorry, that question is beyond my current scope' message when it is asked a question that is either irrelevant to the pest or unknown to it
- background information about the specific pest category to provide context and guide its response
- the user's query
In order to take advantage of the seamless and user friendly UI created by the team at Watson Assistant, we use a web view in the PestBot component of a Watson Assistants webchat that is deployed on an NGINX server in IBM Cloud Kubernetes. The use of Kubernetes and other IBM Cloud services throughout our application allows it to scale to meet the expected demand of millions of farmers around the world. Our app is designed with the concept of design and usability in mind - evidenced by our intuitive user flow and the various help notes that are provided to our users to guide them along the process of receiving the best prediction for the pest that attacks their grains.
How does our solution differ from existing solutions? Like night and day. Currently, the use of image classification to predict stored grain pests is still in the research phase - far from the farmers who are in dire need of a directly accessible application. Additionally, preexisiting apps on app stores only provide predictions focussing on field pests - ignoring the tremendous post harvest losses due to stored grain pests. Moreover, these applications are devoid of the feature of providing sustainable prevention and management techniques - the very feature that takes our app from principle to practice. Finally, we aim to release our app at the extremely low monthly cost of $0.23 per month - the price of two cups of tea in most developing countries. We are grateful to the team at IBM Call for Code 2023 for providing us the opportunity and resources to use our skills to create this awesome application. We certainly believe that PestDetect will go a long way in bringing these farmers the solution that they deserve.
More detail is available in our description document.
-
Watson Machine Learning - The Watson Machine Learning service is used to deploy the Image Classification Machine Learning model, used in the backend service of the app. The model has been trained on a dataset of 7000 images of several pest categories. The model was then uploaded to the IBM Cloud Object Storage. Model inferences for the app are now run in Watson Machine Learning.
-
Watson Assistant - Watson Assistant is used in both the PestSnap and PestBot feature of the app to serve as a chatbot farmer to recommend to them sustainable pest prevention and management techniques, based on different categories of pests . The initial set of management and prevention practices recommended to the user by Watson Assistant is provided through the set of predefined actions for each pest category. Follow up questions are then handled by invoking the gpt-3.5-turbo model. A set of system instructions, information about the pest and the user's question are sent through its API, and the response is provided to the user through the Watson Assistant Interface.
-
Watson Studio - The Image classification model is uploaded as an asset and deployed using the Watson Studio Deployments service on an extra small CPU and 2 GB RAM Instance.
Additionally, This project uses:
- IBM Cloud Kubernetes Service to deploy the Flask Backend and the Nginx Server onto pods which are exposed using the NodePort service.
- IBM Cloud Object Storage stores the Machine Learning Model on the Watson Studio
- IBM Container Registry stores the Docker Image of the Flask Backend and the Nginx Server.
The Deployment of the Backend service is done using the IBM Cloud CLI, Docker CLI and Kubernetes CLI.
- IBM Cloud CLI was used to login into docker and kubernetes.
- Docker was used to build the docker image of the Flask Backend and the Nginx Server.
- Kubernetes was used to deploy the docker image onto the IBM Cloud Kubernetes Service.
Diagram and step-by-step description of the flow of our solution:
- The user navigates to the PestSnap module of the app.
- User uploads the image through the PestSnap module and it is sent to a Flask web app that is deployed to the IBM Kubernetes service.
- After preprocessing the image is sent to the IBM Machine Learning for inferencing.
- The Machine learning model prediction is received by the PestBot module.
- The received prediction is sent to the NGINX server that is deployed to the IBM Kubernetes service by the app.
- The IBM Watson Assistant web chat lists the sustainable management practices for the identified pest.
- The Watson Assistant chatbot is integrated with the gpt-3.5-turbo model using a custom extension to answer a wide variety of follow-up questions from the farmer.
The project currently does the following things.
- PestSnap - Snap or use an image from gallery to identify a stored grain pest and receive targeted sustainable prevention and management techniques
- PestBot - Chat with Pest Bot to get more information about stored grain pests
In the future we plan to increase accessibility by adding more languages, integrating a voice reader to support illiterate farmers, and making region specific predictions and recommendations to the farmers. We will also replace the gpt-3.5-turbo model with an LLM from watsonx to improve recommendations and be even more cost-effective.
See below for our proposed schedule on next steps after Call for Code 2023 submission.
Instruction to run the project locally:
- Create an account in Expo (expo.dev)
- Install the EAS CLI: npm install --global eas-cli
- Run eas login , and enter your credentials
- Run
npm install( yarn install if using yarn) - Run
npm run start( yarn start if using yarn) - Enter 's' to switch to the 'Expo Go' mode
- Download Expo from your phone's app store, and log in using the account created
- Scan the second QR Code generated in the Expo app. When prompted "How would you like to open this project", select "Expo Go" Note: Ensure you are signed into the same Expo account in both the app and your computer. Verify that your computer is connected to the Internet, ideally the same Wi-Fi network as your mobile device
Instruction to deploy on Production: Prior to this eas will need to be Setup on the machine. Follow the instructions here to configure eas.
- Clone the repository
- Navigate to the pest-detect-app directory
- Run
eas build - Run
eas submitto submit the build to the app store
Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.
We used GitHub for versioning. For the versions available, see the tags on this repository
- Ajay Menon
- Jonathan John Thomas
- Siddh Tailor
This project is licensed under the Apache 2 License - see the LICENSE file for details.