Implementation and Evaluation of LLM-Based Conversational Systems on a Low-Cost Device
Koga Sakai, Y. Uehara, Shigeru Kashihara · 2024
The rapid evolution of artificial intelligence (AI) technologies has highlighted the potential of generative AI, particularly large language models (LLMs), to revolutionize various sectors by automating content creation, enhancing personalized education, supporting medical diagnostics, and creating immersive entertainment experiences. However, deploying these advanced models often requires substantial computational resources, posing a challenge in resource-limited environments. This paper explores implementing LLM-based conversational systems on a low-cost device, specifically the Raspberry Pi. We designed and implemented a conversational system utilizing four LLMs, ChatGPT, Bard, Llama, and Rinna, and evaluated their performance in terms of execution time and computational resource usage. Our findings reveal that API-based models (ChatGPT and Bard) are more efficient in processing time and resource consumption, making them suitable for real-time applications. In contrast, locally-run models (Llama and Rinna) provide the advantage of offline operation despite higher computational demands.