This is not India: Yemen

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from Yemen.

Key takeaway

Overall, the systems’ performances were disappointing. In one image, “Rice” was labeled by multiple systems (though not detected), but not in the other where the rice was much more visible. Labels were often too general or irrelevant. Some labels culturally misrepresented the Yemeni food and utensils as Indian or Western. One mislabeling instance was found that could disadvantage people with certain diets or religions.

The above table includes only detections and labels of 80%+ confidence level, for lower confidence levels see the tables further below.

Insights

The object detection feature failed to provide specific descriptions of the objects across all four systems. The descriptions that were given remained surface level (e.g. “Bowl” instead of “Bowl of Chicken”, or “Bowl” instead of “Cup of Vegetable Sauce”) and many objects simply remained undetected.

Interestingly, the labeling features of Azure and Vision gave Indian-origin labels (e.g. “Masala”) to the first image. One explanation for this could be the plate on which the meal is served, which is also commonly used in Indian cuisine (a “Thali”). Perhaps the prevalence of Indian meals in the training images (i.e. because of a higher population, more common use of English, etc.) could contribute to this misrepresentation of a Yemen meal as Indian food.

Several cultural misrepresentations were present as well. For instance, Rekognition labeled (presumably) the bowl of rice as “Oatmeal” and IBM Watson labeled the food as an “Irish Stew”. These misrepresentations should make us question if Yemeni food was represented enough in the training images.

The systems provided more correct labels on the second image (e.g. “Chicken”, “Rice”, and “Carrot”). This is an interesting outcome because, in the first image, all the items are separated in different bowls. One could assume separate items helps the systems to distinguish between the items, but this was not the case. It will be interesting to see if this happens on pictures of other countries as well, as serving food as separate items is common in many Asian countries while and less so in Western countries.

Finally, in one instance, Vision labeled the food as “Seafood”, which could disadvantage people with certain diets (e.g. pescatarian) or religions.

My recommendation

As stated in the analyses of previous countries, the object detection features need to become better in detecting all the objects as well as giving more specific descriptions. The latter is similarly true for the labeling features. Also, further attention is needed towards the idea that the recognition of Yemeni food is influenced by Indian training images. Developers should also be careful that the presentation of Yemeni food and Asian food in general (i.e. as separate items) does not impact their system’s performance. Vision appears to be very good in detecting “Carrot” (see also Vietnam), so congratulations to the developers for that.

Results

Two images of one meal from Yemen were available:

• Meal 1: Rice, Cooked Chicken, Raw Vegetables, Vegetable Sauce (Lunch)

• 

Object detection results*:

_{*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant}

Labeling results:

• 

Object detection results:

Labeling results: