What, When, Where to Compute-in-Memory for Efficient Matrix Multiplication During Machine Learning Inference
Tanvi Sharma, Mustafa Ali, Indranil Chakraborty, Kaushik Roy · IEEE Transactions on Emerging Topics in Computing · 2025
Matrix multiplication is the dominant computation during Machine Learning (ML) inference. To efficiently perform such multiplication operations, Compute-in-memory (CiM) paradigms have emerged as a highly energy efficient solution. However, integrating compute in memory poses key questions, such as 1)What type of CiM to use: Given a multitude of CiM design characteristics, determining their suitability from architecture perspective is needed. 2)When to use CiM: ML inference includes workloads with a variety of memory and compute requirements, making it difficult to identify when CiM is more beneficial than standard processing cores. 3)Where to integrate CiM: Each memory level has different bandwidth and capacity, creating different data reuse opportunities for CiM integration. To answer such questions regarding on-chip CiM integration for accelerating ML workloads, we use an analytical architecture-evaluation methodology with tailored mapping algorithm. The mapping algorithm aims to achieve highest weight reuse and reduced data movements for a given CiM prototype and workload. Our analysis considers the integration of CiM prototypes into the cache levels of a tensor-core-like architecture, and shows that CiM integrated memory improves energy efficiency by up to$3.4 \times$and throughput by up to$15.6 \times$compared to established baseline with INT-8 precision. We believe the proposed work provides insights intowhattype of CiM to use, andwhenandwhereto optimally integrate it in the cache hierarchy for efficient matrix multiplication.