High Dimensional Analog Range In-3D-NAND Search Accelerator for Applications of Search in Few-Shot Learning Model and Retrieval in Retrieval Augmented Generation
Po-Hao Tseng, Shao-Yu Fang, Chi-Tse Huang, Hao-Wei Chiang, Feng-Ming Lee, Yu‐Hsuan Lin, Jhe-Yi Liao, Yu‐Yu Lin, An-Yeu Andy Wu, Hsiang-Yun Cheng, Ming-Hsiu Lee, Kuang-Yeu Hsieh, Keh-Chung Wang, Chih-Yuan Lu · 2024
We developed a novel analog range in-memory searching technology (AR-IMS) using high dimensional serial-parallel array architecture in 3D-NAND flash chip for applications in few-shot learning (FSL) models and retrieval augmented generation (RAG). A controller (or feature extractor) for analog vector extraction of 492 dimensions (features) was optimized from a common model by in-field data to improve system accuracy. The current-clamping function in the AR-IMS array significantly reduces the matching current variation and keeps the total matching current at low level to meet the design requirements for the sense amplifiers. A special analog range level quantization scheme was developed to ensure in-field performance and matching tolerance for high system reliability. The feature vector data bits stored in the AR-IMS are quantized up to 64 levels, and the query vector is quantized into 100 levels. The proposed integrated system (Optimizing controller + AR-IMS accelerator) enable high performance analog computing for continuous-learning FSL AI systems. As compared to the CPU stand-alone system, the AR-IMS accelerator with CPU or GPU integration provided high speed (57x to 258x), low power computing performance (70x to 158x) for RAG in large language model (LLM) generative process.