Predicting Performance Variability
Mohammed Baydoun, Mohammad Sonji, Pedro Bruel, Dejan S. Milojicic, Eitan Frachtenberg, Izzat El Hajj · 2025
As computing systems grow increasingly more complex, application performance on these systems is becoming more variable and less deterministic. Scalar performance summaries such as mean or median run time do not adequately reflect the true behavior of an application that can only be gleaned from the complete performance distribution. However, measuring the distribution of an application’s performance on a system requires running the application many times on that system, which can be an expensive process. To address this challenge, we aim to answer the question: Can the performance distribution of an application on a system be predicted by learning from other representative applications?We aim to answer this question in the context of two use cases: predicting the performance distribution of an application on a system from a few runs of that application on the system, and predicting the performance distribution of an application on a system from a measured distribution of that application’s performance on a different system. To this end, we measure the performance distribution of a large set of representative benchmarks and use that information to train prediction models that predict the performance distribution of new applications. We consider different alternatives for formulating the prediction problem as well as different types of prediction models. Our evaluation compares these alternatives to identify the best formulation and model to use for each use case, and shows that many application performance distributions can be predicted with reasonable accuracy.