Cost-Effective Mission Assurance Engineering Through Simulation
Karl Siil, Aviel D. Rubin, Matthew Elder, Anton T. Dahbura, Matthew Green, Lanier A. Watkins · 2021
Cyber-physical systems have witnessed fantastic leaps in their capabilities, thanks to advances in artificial intelligence and machine learning. With these great capabilities, however, should come great assurance that they will behave as expected. For example, an autonomous vehicle (AV) must protect passengers, bystanders, property and itself. Safety alone is insufficient, however. The AV is built for a mission, and mission assurance must also be addressed, i.e., getting the AV’s job done despite foreseen and unforeseen circumstances. Mission assurance should begin as far left in the engineering lifecycle as possible, ideally before the first vehicle is assembled. If the many hours of operational experience that familiarize system builders and operators with the vehicle’s performance and potential risky behaviors could be accrued through simulation, rather than expensive prototypes, a better vehicle can be developed at significantly less cost. The purpose of this paper is to demonstrate the value of cost-effective open-source based simulation in exercising and analyzing AV algorithms. Our results with DESCRETE, a testbed we developed for engineering mission assurance in the maritime domain, show that sufficient fidelity can be realized practically in the lab for unforeseen, but realistic, situations to arise and be examined in a more controlled and less costly environment. Collision avoidance algorithms, for example, must consider complex interactions between multiple vehicles, trading off safety for mission efficiency. Our experimental results demonstrate the interplay between these two competing goals, and help inform what to deem appropriately safe by both eliminating the obviously unsafe situations and identifying what might be too safe, which necessitates either accepting some risks or changing the mission to avoid them.