Waffles are Easy: Singapore

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from Singapore.

Key takeaway

Finally, the IR systems performed somewhat well! Though object detection was still lacking, several systems correctly labeled the meal in two different pictures.

Of course, the meal contained only a single, simple, and visually distinguishable item (a waffle), but nevertheless a meal was correctly labeled in full for the first time.

It was also almost detected correctly in one image, but waffle had a confidence rating of 76%, right below our cut off point of 80%. Still, many faulty labels were present as well.

The above table includes only detections and labels of 80%+ confidence level, for lower confidence levels see the tables further below.

Insights

Object Detection

For the first image, object detection did not work well. All three detections (of nine in total) were too general (e.g. food). However, for the second image, Azure and Rekognition (combined) detected all three items with a fairly high confidence rating.

Unfortunately, Rekognition also described waffle as a bread, which I feel is a missed opportunity and a clear misrepresentation. Vision’s descriptions remained too general for the second image as well.

Labeling

As always, the labeling systems performed better than the object detection systems. What stands out compared to previous analyses, is that both Vision and Rekognition labeled the meal in both pictures with (very) high confidence ratings (85%-100%). Of course, compared to these previous analyses, the meal consists only of a waffle, a fork, and a knife – all simple and visually distinguishable items. Nevertheless, they labeled them well.

In the first image, the waffle is spread open and clearly shows the texture of Kaya and Margarine. This detail was not picked up by the IR systems. While understandable due to it’s detailed nature, one has to wonder if we can expect IR systems to pick up on these details. And if we can expect this from these systems, how much visual similarities between different countries confuse such systems (and humans).

For instance, I personally never heard of Kaya, though its popularity in Singapore is undeniable. So, as a human coming from a Western country, I’d probably would have described it as butter. Butter visually looks very similar, yet clearly misses the mark. Therefore, this is a clear case where something looks visually similar, but – depending on your background and the context surrounding the image – is something substantially very different.

Some wrongly labeled items were also prominent. For the second image, Rekognition correctly labeled knife first with 99% confidence. However, the next three labels were weapon, blade, weaponry also with 99% confidence. While wrongly labeling a weapon as not a weapon would perhaps have worse consequences, one has to wonder the consequences of labeling a simple table knife as a weapon.

Finally, Vision labeled the waffle as Belgian waffle with the same confidence as a waffle (95%). One wonders if the fame of Belgian waffles influenced the prediction of Vision. Again, one also has to wonder to what degree an IR system can determine the origin of a meal.

Suggestions for improvement

• Address (cultural) misrepresentations (i.e. [not all waffles are Belgian waffles]);
• Understand the limits of IR systems and think about the consequences of these limits:
  • Can we expect IR systems to detect if, for example, a waffle has Kaya and Margarine on it simply based on an image without further context or input?
• Understand the consequences of labeling [a simple dinner knife] as a weapon with high confidence.

Results

Two images of one meal from Singapore were available:

• Meal 1: Waffle with Kaya and Margarine (Dessert)

• 

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

**It appears that the Azure labeling API is not giving back any results at the time of analysis (only _others with a confident rating of 0.004, model version 2021-05-01 [object detection API used is model version 2021-04-01]).

• 

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

**It appears that the Azure API is currently not giving back any results (only abstract_ with a confident rating of 0.004).