Archive for category: Recognizing Food Around the World

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from Ecuador.

Key takeaway

The IR systems performed very poorly for both object detection and labeling. Cuy was not recognized at all across four images. Several (cultural) misrepresentations were present.

Insights

Object Detection

The object detection systems failed to accurately describe the Cuy in any of the four images. Vision gave the general description of Food for the Cuy in all images, while Rekognition gave the description of Pizza for the Cuy in all four images. As such, object detection performed very poorly.

Labeling

The labeling systems performed very poorly as well. The labels given remained surface level and seemed to not even come close to describing the meal. Perhaps the most relevant label was Fried Food. This brings to question the usefulness of the results for the presented meal.

Furthermore, several cultural misrepresentations were present, the most obvious one being Rekognition consistently mistaking the Cuy for pizza. With labels such as Hendl and Britisch cuisine, Vision also gave descriptions that significantly mispresent the meal.

As in previous analyses, here too we have to address confusion (mostly by Vision) between meats. While a typical dish in Ecuador and neighboring countries, people from other cultures might prefer not to eat Cuy. Yet, the systems described the meal as chicken meat, duck meat, turkey meat, pork, etc. If someone would overly rely on the results of these labeling systems, they would perhaps eat Cuy while thinking it would be something else.

Finally, we see that Rekogntion primarily have labels for the laptop in the background. While obviously not wrong, I wonder if Rekognition found it easier to present results of something more common and visually simple/distinctive, and thereby failed to give a lot of results for the food.

Suggestions for improvement

• Provide more specific and relevant labels for Cuy;
• Address (cultural) misrepresentations (i.e. Cuy is not pizza, );
• Make sure labels of meat do not harm people of certain religions or with certain diets (i.e. Cuy is not duck meat or chicken meat).
• Check in how far the systems can distinguish between less relevant, yet visually simple background objects and meals in the foreground (especially for Rekognition).

Results

Four images of one meal from Ecuador were available:

• Meal 1: Cuy (Fried Guinea pig) (Lunch)

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from Singapore.

Key takeaway

Finally, the IR systems performed somewhat well! Though object detection was still lacking, several systems correctly labeled the meal in two different pictures.

Of course, the meal contained only a single, simple, and visually distinguishable item (a waffle), but nevertheless a meal was correctly labeled in full for the first time.

It was also almost detected correctly in one image, but waffle had a confidence rating of 76%, right below our cut off point of 80%. Still, many faulty labels were present as well.

Insights

Object Detection

For the first image, object detection did not work well. All three detections (of nine in total) were too general (e.g. food). However, for the second image, Azure and Rekognition (combined) detected all three items with a fairly high confidence rating.

Unfortunately, Rekognition also described waffle as a bread, which I feel is a missed opportunity and a clear misrepresentation. Vision’s descriptions remained too general for the second image as well.

Labeling

As always, the labeling systems performed better than the object detection systems. What stands out compared to previous analyses, is that both Vision and Rekognition labeled the meal in both pictures with (very) high confidence ratings (85%-100%). Of course, compared to these previous analyses, the meal consists only of a waffle, a fork, and a knife – all simple and visually distinguishable items. Nevertheless, they labeled them well.

In the first image, the waffle is spread open and clearly shows the texture of Kaya and Margarine. This detail was not picked up by the IR systems. While understandable due to it’s detailed nature, one has to wonder if we can expect IR systems to pick up on these details. And if we can expect this from these systems, how much visual similarities between different countries confuse such systems (and humans).

For instance, I personally never heard of Kaya, though its popularity in Singapore is undeniable. So, as a human coming from a Western country, I’d probably would have described it as butter. Butter visually looks very similar, yet clearly misses the mark. Therefore, this is a clear case where something looks visually similar, but – depending on your background and the context surrounding the image – is something substantially very different.

Some wrongly labeled items were also prominent. For the second image, Rekognition correctly labeled knife first with 99% confidence. However, the next three labels were weapon, blade, weaponry also with 99% confidence. While wrongly labeling a weapon as not a weapon would perhaps have worse consequences, one has to wonder the consequences of labeling a simple table knife as a weapon.

Finally, Vision labeled the waffle as Belgian waffle with the same confidence as a waffle (95%). One wonders if the fame of Belgian waffles influenced the prediction of Vision. Again, one also has to wonder to what degree an IR system can determine the origin of a meal.

Suggestions for improvement

• Address (cultural) misrepresentations (i.e. [not all waffles are Belgian waffles]);
• Understand the limits of IR systems and think about the consequences of these limits:
  • Can we expect IR systems to detect if, for example, a waffle has Kaya and Margarine on it simply based on an image without further context or input?
• Understand the consequences of labeling [a simple dinner knife] as a weapon with high confidence.

Results

Two images of one meal from Singapore were available:

• Meal 1: Waffle with Kaya and Margarine (Dessert)

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

**It appears that the Azure labeling API is not giving back any results at the time of analysis (only _others with a confident rating of 0.004, model version 2021-05-01 [object detection API used is model version 2021-04-01]).

Object detection results.

*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant

Labeling Results:

**It appears that the Azure API is currently not giving back any results (only abstract_ with a confident rating of 0.004).

I’m testing four of the most popular image recognition systems (Azure, Vision, Rekognition and Watson) on self-collected, original images of food from around the world. The results so far in numbers are:

• Correctly predicted images: 0/24 (0%)
• Correctly detected items 7/111 (6%)
• Correct labels: 37/516 (7%)
• Potentially harmful detections/labels: 49

Update: 29/06

I’ve tested the IR systems on five new countries: the Philippines, Canada, the US, Yemen, and Germany (6 meals and 13 images). The results were poor across all countries. If correct at all, detection and labeling descriptions overall remained too general (e.g. Food, Dish, etc.).

Results in numbers:

• Correctly predicted images: 0/13 (0%)
• Correctly detected items 1/45 (2%)
• Correct labels: 14/258 (5%)
• Potentially harmful detections/labels: 44

Insights

Watson, Rekognition and to a lesser degree Azure seem to be trigger happy to label food items (e.g. Rice and Kaydos) as Ice Cream. Is this due to a (Western) bias towards ice cream? Too early to tell really, but nonetheless an interesting direction to explore in future updates.

Though Rice was labeled as Ice Cream multiple times, at other times Rice seemed to be one of the most easily detected items, especially for Azure and Vision. So this leads us to wonder where the threshold for Rice and Ice Cream is located.

Unfortunately, potentially harmful descriptions were common, especially in the form of misplaced references to the origin of the meals as well as confusion between types of meats. More blatant forms of cultural misrepresentation were found too. For example Azure assigned 18 different sausage types to an image of spring rolls.

Finally, with labels such as Gluten and Sugar, one has to wonder what we can realistically expect from IR systems. How can these systems possibly know if a Bagel is gluten or sugar free without any context? Even most humans would find this task impossible.

Suggested improvements:

• Provide more specific and relevant labels for:
  • BBQ, [Cup of] BBQ sauce, Kaydos, sliced melon, crab cake, spring rolls, bagel, minestrone, bottle of wine, olives and pesto
• Fix (cultural) misrepresentations:
  • Rice, Kaydos and Crab Cakes are not ice cream, sliced melon is not a banana, spring rolls are not sausages, rice is not oatmeal;
  • A Thali (Indian serving plate) in an image possibly skews Yemeni food results towards Indian food results.
• Understand the limits of IR systems and think about the consequences of these limits:
  • Can we expect IR systems to distinguish between similar dishes of different countries without further context or input?
  • Can we expect IR systems to detect if, for example, a meal is gluten or sugar free simply based on an image without further context or input?

Update: 17/06

So far, I’ve tested the systems on food from Belgium, Myanmar, Vietnam and Malaysia (9 meals and 11 images). With these admittedly limited samples, I wanted to write an interim conclusion to make the results of the project easily digestible.

Thus far, the results have been disappointing to very disappointing. Object detection consistently failed across the four systems. The systems labeled the images better, though these labels also often included too general or irrelevant descriptions.

In some cases, these descriptions culturally misrepresented (e.g. mistake the country or culture of the dish) the food, which could be controversial in some contexts. In one instance, “Chicken” was described as “Beef”, which in some contexts could disadvantage people from certain religions. In another instance, “Mock Meat” was labeled as “Meat”, which could similarly disadvantage people from certain religions as well as vegetarians/vegans.

Note: all numbers on this page are only from detections and labels of 80%+ confidence level, for lower confidence levels see the countries’ individual analyses.

Image recognition (IR) systems often perform poorly once in the real world. In this post, I test four of the most popular IR systems on original real world images of food from around the world, this time from Yemen.

Key takeaway

Overall, the systems’ performances were disappointing. In one image, “Rice” was labeled by multiple systems (though not detected), but not in the other where the rice was much more visible. Labels were often too general or irrelevant. Some labels culturally misrepresented the Yemeni food and utensils as Indian or Western. One mislabeling instance was found that could disadvantage people with certain diets or religions.

Insights

The object detection feature failed to provide specific descriptions of the objects across all four systems. The descriptions that were given remained surface level (e.g. “Bowl” instead of “Bowl of Chicken”, or “Bowl” instead of “Cup of Vegetable Sauce”) and many objects simply remained undetected.

Interestingly, the labeling features of Azure and Vision gave Indian-origin labels (e.g. “Masala”) to the first image. One explanation for this could be the plate on which the meal is served, which is also commonly used in Indian cuisine (a “Thali”). Perhaps the prevalence of Indian meals in the training images (i.e. because of a higher population, more common use of English, etc.) could contribute to this misrepresentation of a Yemen meal as Indian food.

Several cultural misrepresentations were present as well. For instance, Rekognition labeled (presumably) the bowl of rice as “Oatmeal” and IBM Watson labeled the food as an “Irish Stew”. These misrepresentations should make us question if Yemeni food was represented enough in the training images.

The systems provided more correct labels on the second image (e.g. “Chicken”, “Rice”, and “Carrot”). This is an interesting outcome because, in the first image, all the items are separated in different bowls. One could assume separate items helps the systems to distinguish between the items, but this was not the case. It will be interesting to see if this happens on pictures of other countries as well, as serving food as separate items is common in many Asian countries while and less so in Western countries.

Finally, in one instance, Vision labeled the food as “Seafood”, which could disadvantage people with certain diets (e.g. pescatarian) or religions.

My recommendation

As stated in the analyses of previous countries, the object detection features need to become better in detecting all the objects as well as giving more specific descriptions. The latter is similarly true for the labeling features. Also, further attention is needed towards the idea that the recognition of Yemeni food is influenced by Indian training images. Developers should also be careful that the presentation of Yemeni food and Asian food in general (i.e. as separate items) does not impact their system’s performance. Vision appears to be very good in detecting “Carrot” (see also Vietnam), so congratulations to the developers for that.

Results

Two images of one meal from Yemen were available:

• Meal 1: Rice, Cooked Chicken, Raw Vegetables, Vegetable Sauce (Lunch)

Object detection results*:

_{*Green = the right prediction; Yellow= the right prediction, but too general; Red = potentially harmful prediction; White = largely not relevant}

Labeling results:

Object detection results:

Labeling results: