Fritattapizza: England

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from England.

Key takeaway

The four IR systems performed better than in previous analyses, but still made quite some mistakes. One meal (Frittata) was finally correctly labeled by Watson, but unfortunately with a confidence rating of only 50%. Object detection picked up on a lemon and chopsticks, but failed for 17 other items.

Correctly predicted images 0/8
Correctly detected items 2/19
Correct labels 11/169
Potentially harmful detections/labels
10
The above table includes only detections and labels of 80%+ confidence level, for lower confidence levels see the tables further below.

Insights

Object Detection

In meal 3, Azure and Vision correctly detected a lemon, while the latter also detected chopsticks. Though detecting two over a total of 19 items still signifies a poor performance, it’s refreshing from previous analyses to see the exact items being detected. Unfortunately, the confidence rating for both detections was around 50%, well below an acceptable threshold for most IR systems.

Unfortunately, many faulty detections were made as well. Indian Spinach and Laksa (noodle soup) were both described as ice cream by Rekognition, while the same system also labeled a rice-based dish as a Birthday Cake. The latter being quite ironic, as the first meal actually was a cake, but there was not mention of cake by Rekognition. Rekognition also mistook a sliced lemon for an egg.

Finally, Azure and Rekognition described Frittata as pizza with high confidence rating (86% and 97% respectively). While understandable from a visual perspective (they both look cheesy), Frittata and pizza are very different dishes.

Labeling

The labeling systems were much better than the detection systems and appeared to work somewhat better compared with previous analyses. Though Laksa was not described by it’s name itself, Vision labeled it with noodle soup at a confidence rating of 80%. Rekognition also labeled the Laksa as Noodle with a confidence rating of 96%. Itss strange that the object detection system labeled the Laksa as ice cream with a much lower confidence rating of 75%.

Azure and Vision labeled a sliced carrot cake as cake with fairly high confidence ratings (93% and 91%, respectively). Unfortunately, the same systems, as well as the other two, rated the same cake, but not yet sliced, as meat, beef, steak, red meat, etc. This is interesting as a human would clearly see it as the same cake.

On a positive note, Azure labeled roti (and chapati) well with fairly high confidence rating (86%+), while Vision was able to do the same, but with lower confidence ratings. Unfortunately, Vision and Rekognition also culturally misrepresented roti by labeling it tortilla and pita.

One meal (Frittata) was finally correctly labeled by Watson, but unfortunately with a confidence rating of only 50%. This is unfortunate, as pizza for example was given a confidence rating of 92% for this meal. This is a missed opportunity.

Again, labels of meat were common across most images, even though all meals were vegetarian.

Suggestions for improvement

  • Provide more specific and relevant labels for Raita, Aubergine, Indian Spinach and carrot cake;
  • Fix (cultural) misrepresentations (i.e. roti is not tortilla or pita);
  • Make sure labels of meat do not harm people of certain religions or with certain diets.
  • why wrong labels with a lower confidence rating are assigned during object detection to items while the correct label with a nearly perfect confidence rating is not (specifically for Rekognition).
  • Check why Frittata (the correct label) had a significantly lower confidence rating than pizza (specifically for Rekognition).
  • For cake, make sure to include examples of both sliced and unsliced cake, as this small difference may result in a completely different outcome.

Results

Eight images of six different meals from England were available:

  • Meal 1: Carrot Cake (Snack)
  • Meal 2: Indian Spinach with Wild Garlic and Roti (Dinner)
  • Meal 3: Malaysian/Singaporean Laksa (Dinner)
  • Meal 4: Rice, Aubergine, Mint, Cashew nuts, Raita (Dinner)
  • Meal 5: Spanish Style Frittata (Dinner)
  • Meal 6: Pasta Bake (Tomato, basil and Mozzarella, Dinner)

Object detection results.

GROUND TRUTH MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
Carrot Cake Food (0.58) Food (0.73) Bread (0.98) /

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
food (0.86) Food (0.98) Bread (0.98)
chestnut red color (0.93)
indoor (0.61) Ingredient (0.9) Food (0.98) dish (0.9)
meat (0.58) Recipe (0.88) Steak (0.93) nutrition (0.9)
  Beef (0.85) Meat Loaf (0.83) food (0.9)
  Tableware (0.85)  
reddish brown color (0.86)
  Baked goods (0.84)   food product (0.8)
  Dish (0.82)   meat loaf (0.78)
  Cuisine (0.82)   Prime Rib (0.5)
  Cooking (0.82)    
  Steak (0.81)    
  Red meat (0.79)    
  Produce (0.76)    
  Pork (0.75)    
  Meat (0.75)    
  Fried food (0.73)    
  Dessert (0.72)    
  Comfort food (0.71)    
  Baking (0.69)    
  Soil (0.65)    
  Cake (0.64)    
  Flesh (0.63)    
  Pastrami (0.63)    
  Venison (0.6)    
  Ostrich meat (0.57)    
  Kuchen (0.56)    

Object detection results.

GROUND TRUTH MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
Carrot Cake Food (0.64) Food (0.7) Bread (0.96) /

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
dessert (0.97) Food (0.98) Bread (0.96) chocolate color (1)
baking (0.96) Tableware (0.92) Food (0.96) nutrition (0.91)
baked goods (0.96) Cake (0.91) Sweets (0.91) food (0.91)
cake (0.93) Ingredient (0.9) Confectionery (0.91) meat loaf (0.87)
snack (0.92) Recipe (0.88) Cookie (0.88) dish (0.87)
chocolate cake (0.91) Dish (0.83) Biscuit (0.88) food product (0.8)
chocolate brownie (0.9) Baked goods (0.83) Dessert (0.85) dessert (0.5)
parkin (0.89) Cuisine (0.83) Chocolate (0.83) tiramisu (0.5)
snack cake (0.88) Kuchen (0.79) Meat Loaf (0.6)  
chocolate (0.87)
Flourless chocolate cake (0.79)
Brownie (0.58)  
muscovado (0.86) Gluten (0.78)    
flourless chocolate cake (0.86)
Produce (0.77)    
sweetness (0.86) Frozen dessert (0.75)    
food (0.8) Dessert (0.75)    
indoor (0.6) Birthday cake (0.74)    
  Cooking (0.74)    
  Sweetness (0.72)    
  Baking (0.72)    
  Lekach (0.7)    
  Icing (0.7)    
  Buttercream (0.69)    
  Torta caprese (0.67)    
  Beef (0.67)    
  Chocolate cake (0.66)    
  Torte (0.65)    

Object Detection Results:

GROUND TRUTH MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
Indian Spinach Food (0.84) Food (0.77) Ice Cream (0.7) /
Roti Undetected Undetected Bread (0.94) /

Labeling Results:

MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
food (0.99) Food (0.98) Bread (0.94)
chestnut color (0.88)
indoor (0.89) Tableware (0.91) Food (0.94) dish (0.77)
roti (0.89) Ingredient (0.88) Ice Cream (0.7) nutrition (0.77)
recipe (0.87) Recipe (0.88) Dessert (0.7) food (0.77)
cooking (0.86) Staple food (0.87) Cream (0.7) beige color (0.75)
cookware and bakeware (0.86)
Cookware and bakeware (0.82)
Creme (0.7) meat loaf (0.69)
chapati (0.86) Dish (0.8) Plant (0.7) food product (0.6)
ingredient (0.84) Cuisine (0.8) Pita (0.57) utensil (0.6)
pan (0.72) Cooking (0.8) Seasoning (0.56) Filet Mignon (0.5)
stove (0.62) Produce (0.77)    
kitchen (0.58) Vegetable (0.76)    
  Chapati (0.76)    
  Tortilla (0.75)    
  Corn tortilla (0.74)    
  Jolada rotti (0.74)    
  Bhakri (0.71)    
  Comfort food (0.7)    
  Roti (0.69)    
  Piadina (0.67)    
  Metal (0.65)    
  Kitchen utensil (0.65)    
  Condiment (0.64)    
  Finger food (0.63)    
  Meat (0.63)    
  Fast food (0.61)    

Object Detection Results:

GROUND TRUTH MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
Laksa (Noodle Soup) Undetected Food (0.7) Ice Cream (0.75) /
Chopsticks Undetected Chopsticks (0.5) Undetected /
Lemon Lemon (0.51) Lemon (0.51) Egg (0.6) /

Labeling Results:

MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
food_ (0.75) Food (0.98) Noodle (0.96)
chestnut color (0.88)
  Tableware (0.96) Food (0.96) dish (0.77)
  Ingredient (0.91) Pasta (0.96) nutrition (0.77)
  Recipe (0.88) Plant (0.9) food (0.77)
  Soup (0.87) Vermicelli (0.82) beige color (0.75)
  Noodle (0.86) Ice Cream (0.75) meat loaf (0.69)
  Cuisine (0.86) Dessert (0.75) food product (0.6)
  Dish (0.84) Cream (0.75) utensil (0.6)
  Stew (0.84) Creme (0.75) Filet Mignon (0.5)
  Staple food (0.83) Produce (0.64)  
  Bowl (0.83) Dish (0.61)  
  Noodle soup (0.8) Meal (0.61)  
  Produce (0.79) Egg (0.6)  
  Chopsticks (0.78) Citrus Fruit (0.59)  
  Meat (0.76) Fruit (0.59)  
  Chinese noodles (0.75) Grapefruit (0.56)  
  Hot and sour soup (0.74)    
  Thukpa (0.74)    
  Vegetable (0.73)    
  Rice noodles (0.73)    
  Guk (0.73)    
  Spoon (0.73)    
  Cooking (0.73)    
  Comfort food (0.72)    
  Fast food (0.72)    

Object Detection Results:

GROUND TRUTH MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
Rice Undetected Food Undetected /
Raita Undetected / Undetected /
Aubergine Undetected / Undetected /
Mint Undetected / Undetected /
Cashew nuts Undetected / Undetected /

Labeling Results:

MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
food_ (0.86) Food (0.98) Dish (0.99) food (0.89)
  Tableware (0.96) Meal (0.99) nutrition (0.89)
  White rice (0.9) Food (0.99) beige color (0.85)
  Dishware (0.88) Plant (0.98) dish (0.83)
  Plate (0.88) Vegetable (0.88) risotto (0.83)
  Recipe (0.88) Platter (0.69) food product (0.8)
  Ingredient (0.87) Seasoning (0.58)
emerald color (0.72)
  Fines herbes (0.83) Seasoning (0.58) plate (0.5)
  Staple food (0.82)    
  Jasmine rice (0.81)    
  Rice (0.78)    
  Cuisine (0.78)    
  Produce (0.77)    
  Garnish (0.76)    
  Steamed rice (0.76)    
  Dish (0.76)    
  Lime (0.75)    
  Meat (0.75)    
  Kitchen utensil (0.72)    
  Leaf vegetable (0.71)    
  Comfort food (0.7)    
  Vegetable (0.69)    
  Cooking (0.68)    
  Culinary art (0.68)    
  Xôi (0.67)    

Object Detection Results:

GROUND TRUTH MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
Rice Undetected Food (0.79) Birthday Cake (0.67) /
Raita Undetected /
Aubergine Undetected /
Mint Undetected /
Cashew Nuts Undetected /

Labeling Results:

MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
food_ (0.87) Food (0.98) Plant (0.98) dish (0.95)
  White rice (0.94) Dish (0.95) nutrition (0.95)
  Tableware (0.93) Meal (0.95) food (0.95)
  Ingredient (0.9) Food (0.95) beige color (0.95)
  Recipe (0.88) Vegetable (0.85) risotto (0.93)
  Staple food (0.87) Produce (0.78)
food product (0.79)
  Rice (0.85) Seasoning (0.75)
light brown color (0.74)
  Jasmine rice (0.84) Birthday Cake (0.67)
Grilled Salmon (0.5)
  Dish (0.84) Dessert (0.67)  
  Cuisine (0.84) Cake (0.67)  
  Leaf vegetable (0.8)    
  Plate (0.79)    
  Basmati (0.79)    
  Glutinous rice (0.79)    
  Produce (0.79)    
  Fines herbes (0.78)    
  Steamed rice (0.76)    
  Vegetable (0.75)    
  Meat (0.75)    
  Comfort food (0.71)    
  Culinary art (0.7)    
  Dishware (0.7)    
  Coriander (0.68)    
  Rice and curry (0.61)    
  À la carte food (0.6)    

Object Detection Results:

GROUND TRUTH MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
Frittata Pizza (0.86)
Packaged Goods (0.84)
Pizza (0.97) /

Labeling Results:

MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
food_pizza (0.78) Food (0.95) Pizza (0.97)
pale yellow color (1)
  Ingredient (0.9) Food (0.97) dish (0.95)
  Recipe (0.88) Bread (0.91) nutrition (0.95)
  Baked goods (0.84) Cake (0.76) food (0.95)
  Cuisine (0.83) Dessert (0.76) pizza (0.92)
  Rectangle (0.81) Cornbread (0.6)
Sicilian pizza (0.82)
  Fast food (0.8) Lasagna (0.56) food product (0.8)
  Dish (0.8) Pasta (0.56) cheese pizza (0.7)
  Comfort food (0.71) Pie (0.56) frittata (0.5)
  Staple food (0.67) Dish (0.55)  
  Linens (0.61) Meal (0.55)  
  Side dish (0.61)    
  Pattern (0.61)    
  Junk food (0.59)    
  Metal (0.58)    
 
Cookware and bakeware (0.57)
   
  Meal (0.56)    
  Cooking (0.56)    
  Tin (0.55)    
  Italian food (0.55)    
  American food (0.55)    
  Mixture (0.51)    
  Pattern (0.51)    
       
       

Object Detection Results:

GROUND TRUTH MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
Pasta Bake Food (0.65) Food (0.74) Bread (0.9) /

Labeling Results:

MICROSOFT AZURE GOOGLE VISION AMAZON REKOG. IBM WATSON
  Food (0.97) Food (0.95)
chestnut red color (0.96)
  Ingredient (0.89) Plant (0.92) nutrition (0.83)
  Recipe (0.88) Bread (0.9) food (0.83)
  Cuisine (0.8) Meat Loaf (0.64) dish (0.83)
  Fried food (0.79) Lasagna (0.6)
food product (0.79)
  Dish (0.79) Pasta (0.6) pasta (0.76)
  Fast food (0.79) Vegetable (0.6)
Spaghetti Bolognese (0.68)
  Produce (0.76)   meat loaf (0.52)
  Meat (0.74)   lasagna (0.5)
  Comfort food (0.71)    
  Mixture (0.65)    
  Side dish (0.62)    
  Dessert (0.62)    
  Deep frying (0.57)    
  Metal (0.56)    
  Panko (0.56)    
  Soil (0.54)    
  Rock (0.54)    
  Energy bar (0.52)