In our first blog post, we explored the limitations of physics-based models for camera perception. Though the industry has undoubtedly made strides in advancing the ability of automated systems – end-to-end deep learning models’ complex structure poses challenges from a functional safety perspective. Similarly, physics-based models are interpretable, but lack the complexity needed to accurately predict human behaviour in all scenarios. It’s clear automated systems are missing something when it comes to crossing prediction. 

In February, we examined how physics-based models can be advanced by modelling the mind of the pedestrian. We provided evidence that our Behaviour Model prevents incorrect or delayed predictions that physics-based models often fall victim to. By incorporating psychology into probabilistic machine learning models, Humanising Autonomy is able to mitigate the limitations of physics-based models while keeping the positive attributes of a white box AI approach: interpretability, transparency, small model size and a trustworthy estimate of its prediction uncertainty.

This week, we’re looking at our Behaviour Model in context, and will provide real world examples that highlight where Behaviour AI makes a real, tangible difference.

Abrupt stops confuse physics-based systems

Have you ever rushed to the crosswalk, only to stop at the last second as cars start rolling through the intersection? This pedestrian “screech-to-a-halt” confuses physics-based systems, and therefore a majority of automated vehicles on the road today. Moving quickly towards the crosswalk is often incorrectly predicted as “will cross in front of the vehicle”. This behaviour is mostly observed when visibility is reduced, as is the case around narrow sidewalks and corners. 

This sequence of screenshots (L to R) shows the vehicle’s point of view when moving at a constant speed of approximately 30km/h. In the first two images, the pedestrian is looking at the vehicle. A red frame around the pedestrian means that the physics-based model predicts crossing. A blue frame around the pedestrian means that our Behaviour Model predicts crossing (which doesn’t happen above). It can be seen that the physics-based model falsely predicts crossing in the first two images, while our approach does not.

In our case study, we found false crossing prediction to be a frequent occurrence that lowers the overall precision of physics-based models. Consequences can be severe; and often result in emergency braking that harm both passenger and pedestrian, or lead to a rear collision. By accurately understanding the “screecher” pedestrian, Behaviour Models prevent emergency braking and lead to safer, smoother journeys overall, while also being less irritating and more trustworthy for drivers.

Turning the corner isn’t as easy for physics-based models

Turning the corner happens every day; and yet automated systems struggle to accurately predict and identify these scenarios. 

From the stalled vehicle’s point of view: a red frame around the pedestrian indicates that the physics-based model predicts crossing. A blue frame around the pedestrian means that our Behaviour Model predicts crossing. As evidenced by this sequence, our HA Behaviour Model predicts crossing much earlier than the physics-based model. 

Pedestrians often walk parallel or close to the vehicle’s path and turn their head and shoulders to verify if crossing is safe. Crossing will follow if the pedestrian perceives risk as small or non-existent. The physics-based model only predicts crossing after the pedestrian has started approaching the vehicle’s path, thereby delaying crossing predictions and lowering the overall recall of physics-based models.

Vehicles don’t understand the dynamic relationship between people, vehicles and infrastructure

Pedestrians’ awareness is an important factor in their decision to cross. For example, if you notice the vehicle is slowing down, if the driver indicates you’re okay to cross, or if other pedestrians have started crossing, you’ll often make the decision to cross. It’s crucial to understand this context – the pedestrian’s relationship with the vehicles on the road, the street level infrastructure, as well as other people on the street.

Here, the vehicle is slowing down while both pedestrians are looking at the vehicle. A red frame around the pedestrian shows the physics-based model predicts crossing. A blue frame around the pedestrian means that our Behaviour Model predicts crossing. It can be seen that the physics-based model struggles predicting crossing for the two approaching pedestrians while our Behaviour Model predicts crossing correctly. 

The example above highlights that a profound understanding of road dynamics is necessary to be able to accurately predict what a person may, or may not, do next. Subsequently, based on the prediction, the vehicle may want to accelerate to communicate “no yielding” to an aware pedestrian. On the other hand, if the pedestrian is distracted or unaware, the vehicle will have to decelerate. Physics-based models do not understand this, or take this into account when predicting pedestrian actions for decision making. This is a critical mistake: without this understanding, AVs and driver assistance systems cannot operate safely on pedestrian-dense roads where this happens almost constantly. 

The fact is that edge cases are every day

These scenarios are just the tip of the iceberg, and hint at a larger problem for physics-based models and automated vehicle systems that roam our roads today. What are so often referred to as “edge cases” are in fact everyday occurrences. While the physics-based model consistently fails, the Behaviour Model accurately predicts pedestrian intent in a transparent manner. By capturing the complex dynamics between people, vehicles and the environment, the Behaviour Model can be used in automated vehicles to make decisions on acceleration, deceleration or stopping completely. Thus, only by modelling the mind of a pedestrian can a smoother, safer ride truly be possible. 

This is the third in a series of blog posts by Senior Behavioural Data Scientist Dominic Noy. His webinar Beyond Physics: Tackling the Limitations of Camera Perception is available now for download. Contact [email protected] to learn more.