Getting safety metrics right for connected and autonomous vehicles

This report discusses the challenges and practicalities of implementing methods to test an Automated Driving System (ADS) in a range of simulated scenarios. It relates to the CertiCAV Software Framework, a proof-of-concept tool developed by CertiCAV – which has been released as open-source code.

The preceding work on CertiCAV explored important areas of safety assurance. This included defining a set of high-level driving performance criteria and proposing a structure for requirements definition (to ensure that requirements are measurable and useful). The outcomes of this work package were taken as the starting point for the software framework report and for framework design in general. Analysis in the report covers:

• The role of automated testing and its limitations
• Examples of tests which could be used to demonstrate that certain types of requirement are met
• The key challenges to be overcome when implementing different types of test
• Design of a software framework capable of downloading scenarios from a database, running them in a driving simulator, and evaluating the results
• Storage of simulator data output in a standards-aligned format

This report was written for the Department for Transport’s International Vehicle Standards team. It is likely to be of interest to others involved in designing, specifying, testing or assuring Connected and Autonomous Vehicles.

“It is becoming clear that collisions or even TTC (time-to-collision) are, by themselves, inadequate metrics for assessing the safety of autonomous vehicles. While stakeholders are starting to explore more sophisticated metrics, you get a deep understanding of the issues by actually implementing them, and this is what we’ve done and reported on here.” Zeyn Saigol, Principal Technologist at CPC and Technical Lead for CertiCAV

* The CertiCAV project, and the document shared here, is intended to be a thought-piece on the challenges of automated vehicle safety. The document does not intend to be exhaustive, present authoritative recommendations or final solutions. Where assumptions have been made, for the purpose of project delivery, these have been set without prejudice to alternatives. This document does not necessarily represent the views of the Department for Transport or any UK government body.