Image recognition: finding furniture & appliances in photos
The task of object identification in an image is becoming more and more demanded in various industries. With the help of modern gadgets and smartphones equipped with cameras, image recognition is used now both for industrial and consumer applications.
The image recognition process involves low-level feature extraction to locate lines, regions or areas with specific textures. During this process, many problems can arise due to a different view of an object from different angles, occlusion of different parts of the objects in different images, shadows, and background mixing with the features etc. Humans perform these tasks almost unconsciously. However computers require high-quality programming and lots of processing power to recognize objects accurately.
Recently, our engineers received an interesting task: we had to investigate whether it is possible to process images with different types of kitchen appliances and furniture in them and create a list of objects present in these pictures. This kind of image recognition task is usually called image annotation.
Such a ready-to-use software solution, especially one that will provide a high production-level accuracy, is unlikely to be found on the market today. However, our specialists accepted the challenge and successfully delivered the Proof of Concept.
First of all, we wanted to find out how much time this task will take and what methodologies we should apply.
A moderate dataset of about 300 images per each of five object classes was created (the photos were collected from the Internet). Here’s the list of objects classes with sample images:
Since we were limited in time and budget, we decided to approach the image annotation task by using more classic methods: we used DenseSIFTs and Fisher Vectors retaining some spatial information. In total, we tested five different experimentation approaches, all of which proved to give fair results (70-80% accuracy) in quite a short period of time – the whole setup and experimentation process took just about one day of work!
Despite the substantial difference in objects classes almost 80% accuracy was achieved for every class. The best results were achieved with granite countertops and hardwood flooring recognition (which have large areas with very distinctive patterns); kitchen islands appeared to be the most challenging objects to recognize (since the images with kitchen islands usually are more ‘panoramic’ and include a lot of other classes objects). Confusion matrix below illustrates results of the most accurate experiment.
And here’s the mean average precision (mAP) diagram for the same experiment:
Taking kitchen object recognition further with deep learning
The initial experiment proved that automatic kitchen object recognition was feasible even with traditional computer vision techniques. Achieving 70-80% accuracy in just one day of experimentation demonstrated the potential of the approach, but production-ready applications often require even higher precision.
Following this research, our engineers continued investigating deep learning approaches to improve recognition accuracy and make the solution suitable for commercial use.
We thoroughly researched and evaluated VGG, a convolutional neural network model proposed by K. Simonyan and A. Zisserman from the University of Oxford for object recognition. The original model achieved 92.7% top-5 test accuracy on the ImageNet dataset containing more than 14 million images across 1,000 object classes.
To adapt the model for kitchen object recognition, our computer vision engineers developed an image post-processing algorithm that significantly improved object recognition performance for the selected dataset.
Deep learning architecture
The overall architecture of the proposed network is shown below. During the preprocessing stage, the RGB image is normalized by subtracting the mean image values. Based on the VGG 16-layer network, a multilayer deconvolution network is added on top of the convolutional backbone to generate an accurate semantic segmentation map for the input image.
Recognizing more kitchen objects
Once a dataset is created or selected according to a customer’s requirements, the model can be adapted to recognize virtually any required object classes.
For this stage of the project, we expanded the recognition capabilities to include:
- Dining table
- Stove
- Microwave oven
- Dishwasher
- Washing machine
- Refrigerator
- Plate rack
- Waffle iron
- Espresso maker
- Cocktail shaker
Below are several examples of the recognition results produced by the model.
Example 1
Photo processing result:
- 44% coffee pot
- 47% espresso maker
Example 2
Photo processing result:
- 97% dining table
Example 3
Photo processing result:
- 64% stove
- 33% microwave oven
Example 4
Photo processing result:
- 29% toaster
Commercial applications of automatic image annotation
The ability to automatically recognize furniture, appliances, and other objects in images opens opportunities across numerous industries.
For example, in the real estate industry, property photos can be analyzed automatically to identify kitchen appliances, furniture, and room features, helping create listings faster while reducing manual effort. Similar solutions can also improve image organization, search, and analytics wherever businesses manage large collections of visual data.
Beyond real estate, the same approach can be adapted for virtually any domain that requires automatic object recognition and image annotation by training the model on domain-specific datasets.
Summing up
Our initial experiment demonstrated that traditional computer vision methods such as DenseSIFT and Fisher Vectors can provide surprisingly strong results for kitchen object recognition, reaching approximately 70-80% accuracy with a relatively small dataset and minimal development time.
Building on those findings, we continued our research using convolutional neural networks and semantic segmentation techniques based on the VGG architecture. By combining deep learning with custom image post-processing, we demonstrated that automatic kitchen furniture and appliance annotation can be implemented on a commercial level while supporting a much broader range of recognizable objects.
As computer vision technologies continue to evolve, image annotation solutions are becoming increasingly practical for businesses that need to process large numbers of images efficiently. Whether used in real estate or other industries, automated object recognition can reduce manual work, accelerate data processing, and provide a scalable foundation for intelligent image analysis.