Of course there’s Beer: Belgium

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from Belgium.

Key takeaway

Overall, the systems performed poorly, though Amazon Rekognition stood out by providing two meals with almost all necessary labels. Nonetheless, none of the meals were fully detected nor labeled. IBM Watson appeared to be the most specific, but this unfortunately mostly with the wrong labels.

The above table includes only detections and labels of 80%+ confidence level, for lower confidence levels see the tables further below.

Insights

Across all systems, kitchen utensils such as forks, knifes and cups appeared to be more easily recognized than the food itself. In terms of the food, the systems mostly used general terms (e.g. food, bottle, cup, etc.) and failed to provide specifics (e.g. fries, beer bottle, cup of coffee, etc.). As such, it appears that the systems were not prepared for the visual complexities meals inherently present.

While food recognition systems present no immediate harm, (cultural) misrepresentations were common (e.g. labeling vegetables as custard or creme brulee [sic]). IBM Watson was most guilty of misrepresentation, but at the same time was also the only system to provide more specific labels. In this sense, the IR systems need to be much more specific in order to be useful, but developers should be careful as this opens also up the space for harm through misrepresentation.

While the labeling features of these systems had some merit to them, the object detection feature of these systems often disappointed. They commonly failed to detect the meals at all and were too general in their description if they did.

Finally, some forks and knifes remained largely undetected though they were clearly visible and recognizable to the human eye. Though not explicitly tested, unfamiliar lighting conditions in the images may have had an impact on this.

My recommendation

Developers of all four systems need to significantly increase system performance. For Azure, Vision and Rekognition this means providing more specific labels, while for Watson this means getting the specific labels right. A lot of (cultural) nuance is currently lost in these systems.

Results

Images of three different meals from Belgium were available:

• Meal 1: Fries and peanut sauce with a Vedett Beer (lunch)
• Meal 2: Oatmeal, coffee, oranges, and mixed fruit (breakfast)
• Meal 3: Baked tofu, white beans, lettuce, sliced tomato, grated carrots (lunch)

• 

Object detection results*:

_{*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant}

Labeling results:

• 

Object detection results:

Labeling results:

• 

Object detection results:

Labeling results: