Predicting Accuracy for Quantized Neural Networks: An Attention-Based Approach with Single-Image
Wei Lu, Chaojie Yang, Shugang Zhang, Zhong Ma, Qin Yao, Wei Zheng · 2024
Quantization is an important technique for lightweight deployment of neural network models on resource-limited hardware devices, but current neural network model quantization methods suffer from large loss of accuracy for complex tasks and poor generalisation, which affects the application of deep learning models and large language models in real intelligent scenarios. The root cause of these problems lies in the unclear principle of generating computational accuracy loss in the process of neural network model quantization, and the lack of an effective method for evaluating the quantization performance of the model. To address the above problems, we construct a single-image input accuracy predictor based on the self-attention mechanism, which can quickly predict the computational accuracy of the quantized neural network model by generating a single input image representing the whole test set and the corresponding accuracy predictor through the Transformer encoder and decoder. The method can quickly assess the performance of the model during the model development process, which is conducive to the optimisation of the quantization parameters of the model iterations and promotes the practical application of neural network models.