Optimized Assertiveness-Cost Evaluation: An Innovative Performance Measuring Method for Machine Learning Models
Lyanh Vinicios Lopes Pinto, André Vinícius Neves Alves, Adriano Madureira dos Santos, Flávio Rafael Trindade Moura, Walter Alexandre A. de Oliveira, Jefferson Magalhaes De Morais, Roberto Limão de Oliveira, Diego L. Cardoso, Marcos César da Rocha Seruffo · 2024
The increasing use of Machine Learning (ML) across various sectors has rendered model evaluation a progressively complex task. Ensuring models exhibit both high performance and usability necessitates in-depth evaluations of both assertiveness and computational cost. To address these needs, this study introduces the Optimized Assertiveness-Cost Evaluation (OACE) method, a novel general performance evaluation framework for ML that integrates joint analyses of assertiveness and cost metrics, based on a holistic evaluation that balances both aspects. The methodology and implementation extends to different ML contexts, allowing for comprehensive and consistent replication of results from a reproducible algorithm. In the case study, Deep Learning models were developed for the classification of three-dimensional shapes in real-world environments. The results indicated that the ResNet-50 architecture achieved the highest score with the proposed method, being 20% more better than the other architectures, demonstrating its effectiveness in identifying models that maximize assertiveness while minimizing computational cost.