It is critical to test self-driving cars (SDCs) in real world conditions. However, it is often unclear which real-world conditions to actually test in. The following article presents a blueprint for testing SDCs in the urban environment of Vilnius, Lithuania.

Context

Self-driving cars (SDCs) have failed to recognize their surroundings in manners that baffle most humans. For instance, these cars failed to properly recognize kangaroos, traffic signs, and people with a dark skin.

SDCs fail to recognize such seemingly simple things because they learn to make sense of the world in a very different manner than humans do.

One of the biggest technical issues for self-driving cars right now, is recognizing all the obscure yet important elements they may encounter in real-world conditions (edge cases).

In this article, I provide a blueprint of simple, yet critical elements that SDCs need to deal with, but simultaneously could have difficulties with in the urban environment of Vilnius, Lithuania. As such, this blueprint helps plan for SDC testing in real-world conditions.

Approach

Through direct observations in Vilnius and secondary research (i.e. interviews, case analyses, etc.), urban elements of critical importance and of potential relevance to SDCs were identified.

An image recognition (IR) system* was then confronted with a set of original images of the identified elements.

Finally, noteworthy results of the previous step were translated into a concrete and actionable blueprint for testing SDCs based on the intimate knowledge of Vilnius gained in the first step. Three parts of this blueprint are summarized below.

Insights

Keep Vilnius’ youngest generations safe by detecting baby strollers, mini-scooters, and mini-bicycles.

In Vilnius’s city center, one quickly notices the many young families walking around with babies, toddlers and young children. Baby strollers, kids’ scooters, or kid’s bikes often accompany these families.

Unfortunately, out of 48 collected instances of 24 different baby strollers, not a single stroller was properly recognized. In 24 cases, the strollers were not recognized at all. In 8 cases, they were recognized as “luggage and bags”. In other cases, the system saw the strollers simply as an extension of a “person”, usually the parent standing next to or behind the stroller.

Furthermore, out of 42 collected instances of 19 different children with either mini-scooters or mini-bicycles, only two bicycles and not a single scooter were detected (the two bicycles with only 52% and 60% confidence). In most cases, the kids were labeled as “persons” with, in general, confidence rates above 80%. In one case a scooter was mislabeled as a “skateboard”, and in another case as a “home appliance”. It should be noted that adult bicycles do not present as much a problem to the IR system.

Why it matters: babies, toddlers and young children are very vulnerable road users with widely different abilities and movement patterns than adults. Strollers, mini-scooters and mini-bicycles should prompt SDCs to anticipate this difference in abilities and behavior. The fact that the IR system failed to recognize a single stroller is therefore very worrisome. This worry is exacerbated by the system’s inability to detect the mini-scooters and -bicycles these children often ride, as well as its inability to differentiate between adults and small children.

Recommendation**
Test for baby strollers, mini-scooters and mini-bicycles near public parks and popular walking locations, such as Bernardine Park on Maironio g. or Cathedral Square on Šventaragio g. Warm weather weekends and public holidays provide the best testing conditions.

Adapt to fast and unpredictable road users, such as people with e-scooters and skateboards.

People in Vilnius use a wide variety of transportation. As the inner city is relatively small and has many small, curvy streets, e-scooters enjoy a lot of popularity with young to middle aged people. Skateboards, inline skates and – in rarer cases – one wheeled Segways also appear on the streets.

Out of 60 collected instances from 39 different e-scooters, the IR system failed to recognize a single e-scooter, either with people or no people on them. With people on them, the system commonly labeled them simply “person” with a confidence rate above 80%. In three instance, one wheel of the e-scooters was detected. In a few other cases, the e-scooters were labeled as “luggage and bags” (n=1), a “Segway” (n=1), or “Home appliance” (n=2).

Furthermore, out of 29 collected instances of 14 different people on skateboards, only 1 instance of a skateboard was detected. For inline skaters, out of 12 instances of 5 different inline skaters, none was detected as an inline skater. Finally, from 12 instances of 4 different one-wheeled Segways, none was labeled as such.

Why it matters: People on e-scooters, skateboards, inline skates, and one-wheeled Segways all have unique movement and behavior patterns. In the urban environment of Vilnius this often means fast, quick-turning and often unpredictable movement. As the streets of Vilnius’ inner city are (almost literally) filled with these means of transportion, the IR system not being able to properly recognize them gives reason to worry.

Recommendations
Test for people riding e-scooters on smaller streets in the inner city, such as Arklių g. and Stiklių g. Don’t test on bigger streets with seperate bike lanes or large sidewalks such as Kauno g., no much interaction occurs here.
Testing for people riding skateboards can be done around V. Kudirkos a. and Upės g. after school hours, on weekends, or school holidays.
Create your own testing setup to test people riding inline skates and one-wheeled Segways. Both show up too irregularly and unpredictably in practice.
In all of the above cases, test during sunny weather. Testing in rain or snow greatly reduces your testing efficiency.

Customize your testing to local events.

Cities continuously flow and change. Obstacles not there yesterday, are here today, and gone tomorrow. The rather fickle form and temporary nature of these objects present a real challenge to SDCs.

For example, the yearly Vilnius City Days festival presents a number of such challenges. During this festival, a wide range of obstacles (e.g. tents, seating areas, etc.) are placed on the streets. Though cars are not allowed in these areas during the festival, several potential hazards exist. For instance, after the event when people are clearing up the area, miscellaneous objects remain on the streets while traffic control isn’t as strict any more. In the below images, you can see three of such objects with not so good detection results. More challenges were identified, such as visually different and temporary road signs as well as non-conventional pedestrian flows.

A second example of local events are curbside street markets. In the street market – only existing on Sunday mornings – on the two pictures below, vendors sit inches from the road on rather flimsy seats, while a huge variety of items such as luggage and books – depending on what’s for sale – lay right next to them. It does not take much imagination to see how either items or people can accidentally fall on the road or form problems for oncoming traffic. Given the wide variety of objects, preparing SDCs for this situation could prove difficult.

Why it matters: Cities flow where people flow. Whether curbside markets or large events, people – including children – often take the center stage. The large crowds as well as the many miscellaneous objects near or on the streets present unique challenges to SDCs. First, SDCs need to be able to recognize all these objects, but secondly also be able to take the right action when they become obstacles, even with large crowds around. Only a slightly off response of SDCs could easily result in a collision.

Recommendation
A customized testing plan is needed to account for highly temporal and changeable events such as the Vilnius Days’ Festival and street markets.

Conclusion

SDCs need to be tested on situations they will encounter when deployed in the real world. Unfortunately, edge cases make it difficult to predict what SDCs will actually encounter, making it hard to decide what, when and where to test.

In this article, three insights into testing SDCs in Vilnius, Lithuania were given. These insights reveal which urban elements SDCs could encounter in Vilnius, as well as where and when to test for them. Though edge cases can never be ruled out by definition, the presented insights do make testing more efficient, reliable and, ultimately, reduce the chances of accidents with SDCs.

For the full report on testing SDCs in Vilnius, Lithuania, please send me a message.

*Google Vision

**It goes without saying that testing with such vulnerable audiences in real world scenarios requires the highest safety precautions as well as the right permissions from local regulators. Dry testing first is encouraged.