There’s more than Rice: Malaysia

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from Malaysia.

Key takeaway

Overall, the systems’ performances were very disappointing. Object detection was not able to accurately detect any part of a meal, while labeling detected rice and a fork, but was too general for other items. One instance of cultural misrepresentation was also found.

The above table includes only detections and labels of 80%+ confidence level, for lower confidence levels see the tables further below.

Insights

The object detection features performed very poorly on the selected image. Only the very general terms of “Food” and “Tableware” were used to describe items such as “Bowl of Rice”, “Steamed Egg”, “Spoon” by Vision. Furthermore, “Cucumber Soup” was detected as “Plate” and “Chinese Iced Tea” as “Tableware”. All other systems failed to detect anything.

Vision’s labeling feature performed better, but still unsatisfactory. For instance, it labeled “White Rice”, “Fork”, “Steamed Rice” and “Meat”, but failed to detect the “Chinese Iced Tea”, “Cucumber Soup”, “Steamed Egg”, etc. The other systems performed much worse, with only being able to label in a very general manner (e.g. “Food”, “Plate”, “Tableware”, etc.).

Watson also culturally misrepresented the food and labeled it as “Taco”.

My recommendation

Developers should use more specific labels in order for the system to be useful. As the object detection system did not work well at all, a deeper analysis of what went wrong is needed.

Results

An image of one meal from Malaysia was available:

• Meal 1: Rice, Steamed Egg, Sweet and Sour Pork, Cucumber Soup, Chinese Iced Tea (Dinner)

• 

Object detection results*:

_{*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant}

Labeling results: