Identifying Performance Regression Conditions for Testing & Evaluation of Autonomous Systems
Paul G. Stankiewicz, Marin Kobilarov · 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · 2021
This paper addresses the problem of identifying whether/how a black-box autonomous system has regressed in performance when compared to previous versions. The approach analyzes performance datasets (typically gathered through simulation-based testing) and automatically extracts test parameter clusters of predicted performance regression. First, surrogate modeling with quantile random forests is used to predict regions of performance regression with high confidence. The predicted regression landscape is then clustered in both the output space and input space to produce groupings of test conditions ranked by performance regression severity. This approach is analyzed using randomized test functions as well as through a case study to detect performance regression in autonomous surface vessel software.