OnMyPlate: calorie-counting food app prototype
It’s not that we use technology. We live technology – Godfrey Reggio
Technologies that seemed to be far-fetched only a decade ago are deeply penetrating into our routine. Mobile applications recording everything from steps walked to heart rate have become popular at an incomparable scale. The essence of them is to awake user healthy habits and enhance the quality of life.
Abto Software Research and Development department created a mobile application prototype named OnMyPlate . It began as an idea hatched by our software engineers to apply image and volume recognition methods to mainstream consumer life. The purpose of a mobile food recognition system is to estimate calorie intake and recording the eating habits of a user. All image recognition processes are performed on a smartphone in a real-time mode. There is no need to use any extra resources or equipment, just a smartphone. The user only has to download an app, take a photo of the plate and the rest is processed by the program.
Abto Software proposes a methodology for automatic food recognition, based on a neural network model. Our development team conducted an extensive technical investigation for the identification as well as the estimation of the corresponding parameters. Image classifiers are trained to identify and categorize individual food items on a plate from a single image. The technology is also planned to employ contextual clues, such as location, to incorporate data from menus of restaurants in which the image was taken. For the design and evaluation of the prototype system, a visual dataset with nearly 10000 food images is being created and organized into 100 classes.
During the course, the team built and tested successfully the system and achieved a precision of 75%, thus proving the feasibility of the proposed approach.
By the creation of this app we demonstrate the potential of computer vision, clustering and machine learning. We aim for implementing a mobile recognition system that can run in a real-time way, suitable for daily use with image features to be extracted.
The real trailblazer of Abto’s prototype is a self-consistent algorithm for meal recognition with the highest level of simplicity while providing the right functionality.
The main features of the proposed method include:
- Identification of food items from images captured with a camera-enabled device
- The ability to quickly search and log food items.
- interactivity with no communication delays (no server needed, outside computational resources)
- Real-time calorie counting
- Available in Android and iPhone platform in the making
- Simple, lean and intuitive interface
- Advanced search engine
- Extensive food database
- Possibility to add your own food composition
- Automatic updates available
- Utilizes contextual clues, such as restaurant location and menus or the user’s profile, to further augment the information and improve calorie estimation
There were diverse challenges we came across in the process of development.
First, there exists a vast variety of food types that people consume to be included in the database. Second, a category of food usually contains multi-view variations due to how it’s prepared and served. Also, arbitrary lighting conditions, the volume of food may lead to varying visual appearance of meals. Besides, the quality of images taken with mobile phones makes the task even more complex.
Our system trains a neural network then decomposes descriptors.
The following pictures show the output of the algorithm:
These pictures have 3 segments each (ordered): initial image, calculated mask and plate image.
The following pictures show the result:
The advanced processing power of modern smartphones together with object recognition techniques make it possible to use a meal recognition system which runs on a regular smartphone. There are multiple mobile apps exploiting computer vision techniques. The primary technology, though, is image recognition.
In recent years food habit recordings for smartphones have become excessively popular. However, most of such services require selecting food items from complex menus by hand which is time-consuming and troublesome.
We propose a mobile meal recognition system prototype, the idea of which is to identify foods on the plate from images and determine the number of calories in a real-time mode. The app developed by Abto Software engineers is convenient and functional. It is also very promising in terms of availability, communication and server costs.
At the moment, the prototype is implemented and runs on Android and the iPhone version is in the making.
The technological progress and the well-being of humanity depend on each other. Accurate and user-friendly applications on basis of computer vision methods remain an open research issue. The final objective of Abto Software work is to help end-users lead a healthy lifestyle by simply snapping a photo of their plates to get the most of nutrition information.
You are welcome to try it out on the website onmyplateapp.com.