Analog Softmax with Wide Input Current Range for In-Memory Computing
A. Dubé, Paul Manea, Paolo Gibertini, Erika Covi, John Paul Strachan · 2025
The Softmax activation function plays a pivotal role in both the attention mechanism of Transformers and in the final layer of neural networks performing classification. The Softmax function outputs probabilities by normalizing the input values, emphasizing differences among them to highlight the largest values. In digital implementations, the complexity of softmax grows linearly with the number of inputs. In contrast, analog implementations enable parallel computations with lower latency. In this work, we demonstrate that this approach achieves a more efficient linear scaling of latency as vector size increases logarithmically. This analog softmax circuits are implemented in TSMC 28 nm PDK technology, capable of driving up to 128 inputs and producing an analog current output spanning three orders of magnitude. The study examines the circuit’s power consumption, latency, and error, emphasizing its efficiency compared to the alternative approach of converting outputs to digital signals via ADCs and performing the softmax calculation digitally. By reducing reliance on these power-intensive operations, this work aims to significantly enhance energy efficiency in in-memory computing systems.