It’s not that we use technology. We live technology – Godfrey Reggio
Technologies that seemed to be far-fetched only a decade ago are deeply penetrating into our routine. Mobile applications recording everything from steps walked to heart rate have become popular at an incomparable scale. The essence of them is to awake user healthy habits and enhance the quality of life.
Abto Software Research and Development department created a mobile application prototype named OnMyPlate . It began as an idea hatched by our software engineers to apply image and volume recognition methods to mainstream consumer life. The purpose of a mobile food recognition system is estimating calorie intake and recording eating habits of a user. All image recognition processes are performed on a smartphone in a real time mode.There is no need to use any extra resources or equipment, just a smartphone. The user only has to download an app, take a photo of the plate and the rest is processed by the program.
Abto Software proposes a methodology for automatic food recognition, based on neural network model. Our development team conducted extensive technical investigation for the identification as well as the estimation of the corresponding parameters. Image classifiers are trained to identify and categorize individual food items on a plate from a single image. The technology is also planned to employ contextual clues, such as location, to incorporate data from menus of restaurants in which the image was taken. For the design and evaluation of the prototype system, a visual dataset with nearly 10000 food images is being created and organized into 100 classes.
During the course, the team built and tested successfully the system and achieved precision of 75%, thus proving the feasibility of the proposed approach.
By creation of this app we demonstrate the potential of computer vision, clustering and machine learning. We aim for implementing a mobile recognition system which can run in the real-time way, suitable for daily use with image features to be extracted.
The real trailblazer of Abto’s prototype is self-consistent algorithm for meal recognition with the highest level of simplicity while providing the right functionality.
Main features of the proposed method include:
- Identification of food items from images captured with a camera-enabled device
- The ability to quickly search and log food items.
- interactivity with no communication delays (no server needed, outside computational resources)
- Real time calorie counting
- Available in Android and iPhone platform in the making
- Simple, lean and intuitive interface
- Advanced search engine
- Extensive food database
- Possibility to add your own food composition
- Automatic updates available
- Utilizes contextual clues, such as restaurant location and menus or the user’s profile, to further augment the information and improve calorie estimation
There were diverse challenges we came across in the process of development.
First, there exists a vast variety of food types that people consume to be included into the database. Second, a category of food usually contains multi-view variations due to how it’s prepared and served. Also, arbitrary lighting conditions, volume of food may lead to varying visual appearance of meals. Besides, the quality of images taken with mobile phones makes the task even more complex.
Our system trains neural network then decomposes descriptors.
The following pictures show the output of the algorithm:
These pictures have 3 segments each (ordered): initial image, calculated mask and plate image.
The following pictures show the result:
Advanced processing power of the modern smartphones together with object recognition techniques make it possible to use meal recognition system which runs on the regular smartphone. There are multiple mobile apps exploiting computer vision techniques. The primary technology, though, is the image recognition.
In recent years food habit recordings for smartphones have become excessively popular. However, most of such services require selecting food items from complex menus by hand which is time consuming and troublesome.
We propose a mobile meal recognition system prototype, the idea of which is to identify foods on the plate from images and determine the amount of calories in a real-time mode. The app developed by Abto software engineers is convenient and functional. It is also very promising in terms of availability, communication and server costs.
At the moment, the prototype is implemented and runs on Android and the iPhone version is in the making.
Technological progress and well-being of humanity depend on each other. Accurate and user-friendly applications on basis of computer vision methods remain an open research issue. The final objective of Abto Software work is to help end-users lead a healthy lifestyle by simply snapping a photo of their plates to get the most of nutrition information.
You are welcome to try it out