Rice is not Ice Cream: the Philippines

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from the Philippines.

Key takeaway

Overall, the systems performed very poorly. Not a single item was correctly detected and severe cultural misrepresentations were made. Vision was able to pick up on the visual characteristics of the BBQ, but unfortunately ascribed it to many different cultures and countries except for the Philippines.

The above table includes only detections and labels of 80%+ confidence level, for lower confidence levels see the tables further below.

Insights

Object Detection

The object detection features failed to identify a single item across all three images. Descriptions that were given, were very general (e.g. “Food”). Also, Rekognition identified rice as “Ice Cream” across all three images, while Vision also identified the rice as “Ice Cream” and “Dessert”. Many of the items remained undetected.

Labeling

The BBQ is perhaps one of the most visually distinctive item in the three images. Vision clearly recognized this distinctiveness as it consistently labeled skewered BBQ like dishes such as Sate Kambing (Indonesian mutton satay), Shashlik (Caucasus/Central Asian skewered meat), Yakatori (Japanese skewered pork or chicken), and many more.

Unfortunately, none of these labels referred to the Philippines, which makes us question if any cultural misrepresentation is at play here (BBQ is a very common dish in the Philippines). It also makes us question as to in how far it is possible to distinguish visually very similar dishes between cultures based on an image without context.

While in previous analyses rice seemed somewhat easy to detect, Watson and Rekognition mistook rice as “Ice Cream” across multiple images. Vision also made this error (once as “Ice Cream” and another time as “Dessert”), though it did also label it as “Rice”. Here, again, it makes us wonder if any cultural misrepresentation is at play. Though one can imagine the visual similarities between the round shaped presentation of the rice (which is common in many Asian countries) in the images and ice cream, one can not help but feel ice cream was simply more represented in the training data.

As always, many labels were also too general or irrelevant.

My recommendation

• Provide more specific and relevant labels for “BBQ” and “[Cup of] BBQ Sauce”;
• Fix (cultural) misrepresentations (i.e. rice is not ice cream);
• Understand the limits of how well a IR system could distinguish between similar dishes of different countries without further context or input, and what consequences this limit could have.

Results

Three images of one meal from the Philippines were available:

• Meal 1: White rice and BBQ (dinner)

• 

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

• 

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

• 

Object Detection Results:

Labeling Results: