Toronto-based autonomous vehicle (AV) startup Waabi is calling for the industry to follow the road it has forged by developing a framework to measure the realism of AV simulators.
Waabi founder and CEO Raquel Urtasun outlined in a blog post this week how the company’s newly developed realism metric works, which is meant to evaluate how an AV simulator represents the physical world to an autonomous system.
Urtasun explained how the most safety-critical events are also the rarest, making it essentially impossible for an AV to be exposed to every potential real-world situation, and that recreating dangerous scenarios for testing purposes are also practically impossible and come with ethical problems.
“Advanced simulators can be used to understand how the system will perform and in which specific situations it will fail, helping to understand the deficiencies of the system,” Urtasun said. “But relying on simulation for these high stakes tests introduces a new, critical hurdle: ensuring the realism of the simulator.”
Waabi’s realism metric compares how AVs drive in simulated scenarios to how they drive in identical real-world scenarios. They do this by recreating a set of real-world scenarios within the simulator as a digital twin, and then measuring the difference between the AV’s trajectory in both. The closer a simulation is to an AV’s real-life reaction, like triggering a hand brake or detecting an encroaching vehicle, the more realistic and reliable a simulator can be considered. Waabi says its platform has a 99.7 percent realism score, by its own metric.
RELATED: Waabi inks partnership with investor Volvo to develop and deploy self-driving trucks
After unveiling Waabi’s new measurement, Urtasun called on the AV industry to prioritize simulator realism, saying that transparency and accountability are “absolutely paramount” for building public trust in AV technology.
“Going forward, all AV developers using simulation need to be able to publicly demonstrate the quantified realism of their simulators,” Urtasun said. “Just as we have safety standards for vehicles themselves, we must establish clear and measurable standards for the simulators on which their safety depends.”
Earlier this year, Waabi teamed up with one of its investors, Swedish automaker Volvo, to build and commercialize self-driving trucks. Urtasun told BetaKit at the time that the deal was “a massive step forward” and the last piece the startup needed in order to have a solution that can scale. The partnership follows Waabi’ $275-million CAD ($200-million USD) Series B round last June, when the company announced it planned to deliver fully driverless, generative AI-enabled trucks in 2025.
Feature image courtesy Waabi.