There’s more than Rice: Malaysia

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from Malaysia.

Key takeaway

Overall, the systems’ performances were very disappointing. Object detection was not able to accurately detect any part of a meal, while labeling detected rice and a fork, but was too general for other items. One instance of cultural misrepresentation was also found.

Correctly predicted images 0/1
Correctly detected items 0/7
Correct labels 2/19
Potentially harmful detections/labels
The above table includes only detections and labels of 80%+ confidence level, for lower confidence levels see the tables further below.


The object detection features performed very poorly on the selected image. Only the very general terms of “Food” and “Tableware” were used to describe items such as “Bowl of Rice”, “Steamed Egg”, “Spoon” by Vision. Furthermore, “Cucumber Soup” was detected as “Plate” and “Chinese Iced Tea” as “Tableware”. All other systems failed to detect anything.

Vision’s labeling feature performed better, but still unsatisfactory. For instance, it labeled “White Rice”, “Fork”, “Steamed Rice” and “Meat”, but failed to detect the “Chinese Iced Tea”, “Cucumber Soup”, “Steamed Egg”, etc. The other systems performed much worse, with only being able to label in a very general manner (e.g. “Food”, “Plate”, “Tableware”, etc.).

Watson also culturally misrepresented the food and labeled it as “Taco”.

My recommendation

Developers should use more specific labels in order for the system to be useful. As the object detection system did not work well at all, a deeper analysis of what went wrong is needed.


An image of one meal from Malaysia was available:

  • Meal 1: Rice, Steamed Egg, Sweet and Sour Pork, Cucumber Soup, Chinese Iced Tea (Dinner)
Object detection results*:
Ground Truth Microsoft Azure Google Vision Amazon Rekognition IBM Watson
Bowl of Rice Undetected Food (0.73) Undetected /
Steamed Egg Undetected Food (0.72) Undetected /
Sweet and Sour Pork Undetected / Undetected /
Spoon Undetected Tableware (0.56) Undetected /
Cucumber Soup Undetected Plate (0.84) Undetected /
Chinese Iced Tea Undetected Tableware (0.76) Undetected /
Fork Undetected Undetected Undetected /

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling results:
Food (0.99) Food (0.99) Plant (0.90) Nutrition (0.71)
Table (0.96) Tableware (0.97) Food (0.90) Food (0.71)
Plate (0.95) Ingredient (0.91) Breakfast (0.87) Dish (0.71)
Tableware (0.79) White Rice (0.90) Bowl (0.83) Taco (0.70)
Breakfast (0.66) Recipe (0.88) Meal (0.81) Shop (0.68)
Dish (0.61) Cuisine (0.86) Vegetation (0.77) Retail Store (0.68)
Fast food (0.60) Plate (0.86) Dish (0.69) Building (0.68)
Dinner (0.57) Kitchen Utensil (0.86) Lunch (0.66) Reddish Brown Color (0.66)
Bowl (0.52) Fork (0.85) Produce (0.65) Light Brown Color (0.64)
Dish (0.85) Vegetable (0.57) Food Product (0.60)
Nasi Kandar (0.81)
Produce (0.79)
Staple Food (0.76)
Jasmine Rice (0.74)
Meat (0.72)
Steamed Rice (0.71)
Serveware (0.69)
Vegetable (0.69)
Comfort Food (0.68)
Rice (0.67)
Fried Food (0.67)
Lime (0.66)
Condiment (0.66)
Hayashi Rice (0.63)
Papadum (0.61)