Soup is (not) served: Bulgaria

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from Bulgaria.

Key takeaway

The IR systems performed poorly in detecting as well as labeling one meal. The name of the dish nor any of its ingredients were correctly identified.

The above table includes only detections and labels of 80%+ confidence level, for lower confidence levels see the tables further below.

Insights

Object Detection

The object detection systems performed poorly. The dish nor any of its ingredients were predicted. Only general descriptions (e.g. bowl, food, etc.) were given and Rekognition again described the dish as ice cream in both images.

Labeling

The labeling systems performed similarly poorly. Not one correct and accurate label for the meals was given by any of the four IR systems. Most labels remained too general (e.g. bowl, food, etc.).

In the second image, egg and sausage is clearly visible, yet no systems picked up on these items. One wonder if this is the case because they are part of larger meal with visually mixed items.

Many labels clearly also (culturally) misrepresented the meal. For instance, Vision and Watson both described the dish as soup, which it clearly is not. But the liquid-like substance in a pot might perhaps have fooled the systems, as this is often how soup is portrayed. A similar thing can be said about the description of custard and creme brulee [sic] by Watson.

Finally, Vision also provided wrong cultural and origin specific descriptions (e.g. Arancini, Kai Yang, Chiboust Cream, American Food, Skyr). As in previous analyses, one has to wonder the consequences of wrongly labeling the culture and origin of food.

Suggestions for improvement

• Provide more specific and relevant labels for Sirene Po Shopski, Egg, and Sausage.
• Address (cultural) misrepresentations (i.e. Sirene Po Shopski is not soup or creme brulee [sic]);

Results

Two images of one meal from Bulgaria were available:

• Meal 1: Sirene Po Shopski (Cheese, Tomatoes, butter, eggs, and sausage)(Dinner)

• 

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

• 

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results: