2.4 ATOMUS: A 5nm 32TFLOPS/128TOPS ML System-on-Chip for Latency Critical Applications
Chang-Hyo Yu, Hyoeun Kim, Sungho Shin, Kyeongryeol Bong, Hyun‐Suk Kim, Yoonho Boo, Jaewan Bae, Minjae Kwon, Karim Charfi, Jinseok Kim, Hongyun Kim, Myeongbo Shim, Changsoo Ha, Wongyu Shin, Jaesung Yoon, Miock Chi, Byungjae Lee, Sungpill Choi, Donghan Kim, Jeongseok Woo · 2024
The growing computational demands of AI inference have led to widespread use of hardware accelerators for different platforms, spanning from edge to the datacenter/cloud. Certain AI application areas, such as in high-frequency trading (HFT) [1–2], have a hard inference latency deadline for successful execution. We present our new AI accelerator which achieves high inference capability with outstanding single-stream responsiveness for demanding service-layer objective (SLO)-based AI services and pipelined inference applications, including large language models (LLM). Owing to low thermal design power (TDP), the scale-out solution can support multi-stream applications, as well as total cost of ownership (TCO)-centric systems effectively.