Fritattapizza: England

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from England.

Key takeaway

The four IR systems performed better than in previous analyses, but still made quite some mistakes. One meal (Frittata) was finally correctly labeled by Watson, but unfortunately with a confidence rating of only 50%. Object detection picked up on a lemon and chopsticks, but failed for 17 other items.

The above table includes only detections and labels of 80%+ confidence level, for lower confidence levels see the tables further below.

Insights

Object Detection

In meal 3, Azure and Vision correctly detected a lemon, while the latter also detected chopsticks. Though detecting two over a total of 19 items still signifies a poor performance, it’s refreshing from previous analyses to see the exact items being detected. Unfortunately, the confidence rating for both detections was around 50%, well below an acceptable threshold for most IR systems.

Unfortunately, many faulty detections were made as well. Indian Spinach and Laksa (noodle soup) were both described as ice cream by Rekognition, while the same system also labeled a rice-based dish as a Birthday Cake. The latter being quite ironic, as the first meal actually was a cake, but there was not mention of cake by Rekognition. Rekognition also mistook a sliced lemon for an egg.

Finally, Azure and Rekognition described Frittata as pizza with high confidence rating (86% and 97% respectively). While understandable from a visual perspective (they both look cheesy), Frittata and pizza are very different dishes.

Labeling

The labeling systems were much better than the detection systems and appeared to work somewhat better compared with previous analyses. Though Laksa was not described by it’s name itself, Vision labeled it with noodle soup at a confidence rating of 80%. Rekognition also labeled the Laksa as Noodle with a confidence rating of 96%. Itss strange that the object detection system labeled the Laksa as ice cream with a much lower confidence rating of 75%.

Azure and Vision labeled a sliced carrot cake as cake with fairly high confidence ratings (93% and 91%, respectively). Unfortunately, the same systems, as well as the other two, rated the same cake, but not yet sliced, as meat, beef, steak, red meat, etc. This is interesting as a human would clearly see it as the same cake.

On a positive note, Azure labeled roti (and chapati) well with fairly high confidence rating (86%+), while Vision was able to do the same, but with lower confidence ratings. Unfortunately, Vision and Rekognition also culturally misrepresented roti by labeling it tortilla and pita.

One meal (Frittata) was finally correctly labeled by Watson, but unfortunately with a confidence rating of only 50%. This is unfortunate, as pizza for example was given a confidence rating of 92% for this meal. This is a missed opportunity.

Again, labels of meat were common across most images, even though all meals were vegetarian.

Suggestions for improvement

• Provide more specific and relevant labels for Raita, Aubergine, Indian Spinach and carrot cake;
• Fix (cultural) misrepresentations (i.e. roti is not tortilla or pita);
• Make sure labels of meat do not harm people of certain religions or with certain diets.
• why wrong labels with a lower confidence rating are assigned during object detection to items while the correct label with a nearly perfect confidence rating is not (specifically for Rekognition).
• Check why Frittata (the correct label) had a significantly lower confidence rating than pizza (specifically for Rekognition).
• For cake, make sure to include examples of both sliced and unsliced cake, as this small difference may result in a completely different outcome.

Results

Eight images of six different meals from England were available:

• Meal 1: Carrot Cake (Snack)
• Meal 2: Indian Spinach with Wild Garlic and Roti (Dinner)
• Meal 3: Malaysian/Singaporean Laksa (Dinner)
• Meal 4: Rice, Aubergine, Mint, Cashew nuts, Raita (Dinner)
• Meal 5: Spanish Style Frittata (Dinner)
• Meal 6: Pasta Bake (Tomato, basil and Mozzarella, Dinner)

• 

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

• 

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

• 

Object Detection Results:

Labeling Results:

• 

Object Detection Results:

Labeling Results:

• 

Object Detection Results:

Labeling Results:

• 

Object Detection Results:

Labeling Results:

• 

Object Detection Results:

Labeling Results:

• 

Object Detection Results:

Labeling Results: