Where are the Chopsticks? Vietnam

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from Vietnam.

Key takeaway

Overall, the systems’ performances were disappointing. Object detection failed miserably across all four systems. Labeling worked slightly better, though rice was labeled in the first meal, but generally not in the second two meals. The systems overall missed a lot of nuances and provided predictions that could harm religious people and vegetarians/vegans.

The above table includes only detections and labels of 80%+ confidence level, for lower confidence levels see the tables further below.

Insights

The object detection features performed very poorly on the selected images: only one piece of carrot and one spoon was correctly identified (only be Azure and Rekognition, respectively), other items remained largely undetected (especially by Rekognition) or were described in a too general manner.

As usual, the labeling features identified more items, but were also severely lacking in detail. For the first meal, the rice and vegetables were detected, though only Vision detected both. Nuances such “fried vegetables” or “mixed vegetables” were not present. The soup was detected by Vision and Watson, though did not recognize it as “Bok Choy Soup”. Rekognition was able to label a spoon, which other systems missed. The chopsticks were not labeled.

For the second meal, rice was only detected by Vision and not by other systems, which is surprising as rice seemed easily detectable in the previous meal and other cases. Perhaps this was due to the rather unusual and cut-off position the rice was in, though this is just a guess. Is is common, Vision provided more labels, but this led to clear (cultural) misrepresentations (e.g. labels of “Guk” and “Sauerkraut”). One misrepresentation of Vision could, in some contexts, also cause harm to religious people or vegetarians/vegans: mock meat was labeled as meat with a moderately high confidence score (0.73).

Finally, for the third meal, only “Tomato”, “Iceburg Lettuce” [sic], and “Grilling were identified, the chicken and rice were not. As with the mock meat, the chicken was actually labeled as both “Beef” and “Steak”, which could cause harm or confusion to religious people. Also, as was the case in an image of the first meal, the chopsticks were not labeled. It will be interesting to see (in future analysis of other countries) if this is due to cultural bias or if this is simply a coincidence. Rice was strangely not detected in the image though it is clearly visibele, perhaps this is due to it’s position in the image.

My recommendation

As in previous analysis, I recommend developers use more specific labels and make sure that the specific labels they use don’t cause any misrepresentation. In this case, mock meat and types of real meats were interchanged. Unfortunately, this could harm religious people and vegetarians/vegans. Developers should also be sure to check if their systems work properly on chopsticks and not just spoons and forks, though further analysis is needed for this. Finally, Vision developers should probably fix the typo in the “Iceburg Lettuce” label.

Results

Images of three different meals from Vietnam were available:

• Meal 1: Rice, Bok Choy Soup and Fried Mixed Vegetables (Dinner)
• Meal 2: Rice, Cucumber Soup, Mock Meat and Lettuce (Lunch)
• Meal 3: Rice, Grilled Chicken, and raw vegetables (Dinner)

• 

Object detection results*:

_{*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant}

Labeling results:

• 

Object detection results:

Labeling results:

• 

Object detection results:

Labeling results:

• 

Object detection results:

Labeling results: