Inference-Optimized Metaheuristic Approach for a Prospective AI Training/Inference Inversion Paradigm in Optimized Energy-Aware Computing
S. Chan · 2024
Energy consumption at the various steps of the Machine Learning (ML) model life cycle, as a constituent application of Artificial Intelligence (AI), is infrequently reported. While the Explainable AI (XAI) or Explainable ML (XML) movement has focused upon more explainable ML models, the AI Energy Consumption (AEC) facet has not progressed as rapidly and still remains quite translucent. For example, even the AEC ratios of the key AI stages (e.g., pre-training, fine-tuning, and inferencing) have not always accompanied the releases of new ML models. This lack of data has, perhaps, contributed to the dearth of analyses on effective compute (e.g., algorithmic efficiency versus hardware efficiency) for the newer models, and in modern times, AEC may be skewing to the inferencing side. This may necessitate revised architectures, particularly amidst the findings that generalized ML models for specific tasks have a much higher AEC when contrasted against task-specific ML models. It has also been reported that, within those same models, a higher number of parameters segues to a higher AEC. Higher accuracies also beget higher AECs, and “advanced anomaly detection” necessitates tasks that have an even higher AEC. Moreover, as it is now customary to run numerous instances of a pre-trained model over various instances in an ensemble fashion, the AEC is multiplied accordingly. Yet, there are opportunities to reduce AEC at the Metaheuristic Algorithm (MA) level (e.g., at the convolutional layer), and certain versatile constructs (that scale well across the AI stages) are amenable to such optimizations; furthermore, performance metric comparisons in the literature have, traditionally, been artificially constrained to a “fixed number of allowed function calls,” and this might have led to misinterpretations of MA performance in Real World Scenario (RWS) paradigms. These misinterpretations can skew research directions, particularly for RWS Multiple Objective Large Scale Nonlinear Programming Problems (MOLSNLP) and may also lead to an underestimation of the involved AEC. This paper presents a promising RWS-oriented Particle Swarm Optimization (PSO)-based MA with concomitant Rough Order of Magnitude (ROM) AECs.