All Sausage, but no Spring Roll: Canada

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from Canada.

Key takeaway

Overall, the systems performed very poorly. No images were correctly described, nor were any items in the images correctly detected. Some labels such as “Rice” and “White Rice” described an item in the images, but in general the labels remained superficial. Many potentially harmful labels were found.

The above table includes only detections and labels of 80%+ confidence level, for lower confidence levels see the tables further below.

Insights

Object Detection

Across the two meals, the object detection systems performed very disappointingly. Most items were described as the general terms of “Food” or “Bowl”, while two descriptions were completely off: crab cakes as well as Kadyos were detected as “Ice Cream” by Rekognition, while sliced melon was recognized as “Banana”. Though difficult to verify, these mistakes might indicate a (cultural) misrepresentation in the training data (i.e. more images of bananas than melons).

Labeling

Labeling remained surface level as well. General labels such as “Food”, “Dish” and “Plate” were common, as well as irrelevant labels such as “Ingredient” and “Recipe”. As is, the value of these labels is questionable, though this of course depends on the applications of use.

More specific (though still somewhat general) labels were also found, including “Stew”, “Brown Sauce”, “Comfort Food”. As in previous cases, descriptions such as “Rice” and “White Rice” were plentiful as one image clearly depicted rice.

Unfortunately, the systems also gave many potentially harmful or simply wrong descriptions. In a first instance, in one instance, one meal was described as “German Food”. In other instance, the meals’ were typical meals from other origins (e.g. “Semur” [an Indonesian Dish] , “Dumpling”, “Varenyky”) than Canadian/Filipino.

Azure seemed to be really confused by the image of three spring rolls, giving it 18 different descriptions of sausages (e.g. “Bratwurst”, “Loukaniko”, etc.) and not one of “Spring Rolls”. Rekognition also described the spring rolls as “Hot Dog” with 0.99 certainty.

As in the analyses of previous countries, types of meats were also often used interchangeably, as in “Pork”, “Beef”, “Chicken meat”, “Clam” would all be given as a label. Depending on the context, this could be detrimental to people of certain religions or with certain diets that avoid certain meats.

My recommendation

• Provide more specific and relevant labels for “Kaydos”, “Sliced Melon”, “Crab cake” and “Spring Rolls”;
• Fix (cultural) misrepresentations (i.e. crab cakes are not ice cream, sliced melon is not a banana, spring rolls are not sausages);
• Make sure labels of meat do not harm people of certain religions or with certain diets.

Results

Five images of two different meals from Canada (with a Filipino origin) were available:

• Meal 1: White rice, Kaydos (beans, jackfruit, pork [usually beef]), Sliced Melon (lunch)
• Meal 2: Crab Cake, Spring Rolls (filled with crab meat), Sliced Melon (snack)

• 

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

• 

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

• 

Object Detection Results:

Labeling Results:

• 

Object Detection Results:

Labeling Results:

• 

Object Detection Results:

Labeling Results: