In-Vehicle Artificial Intelligence Agent System with Local Large Language Models for Agent Selection and Vehicle Control
Toru Furusawa · 2025
Voice User Interfaces (VUIs) in vehicles enhance driver experience by enabling hands-free interaction with in-vehicle systems. However, the increasing complexity of these systems poses challenges for intuitive use, as drivers must remember numerous voice commands. Recent advancements in large language models (LLMs) offer opportunities to create more natural and flexible voice interactions. In this paper, we propose an in-vehicle Artificial Intelligence (AI) agent system that utilizes a local LLM for agent selection and vehicle control, eliminating the need for manual agent selection and dependency on internet connectivity for vehicle Application Programming Interface (API) execution. The system analyzes user speech content to automatically select appropriate AI agents and executes in-vehicle APIs using a local LLM capable of function calling. We implemented the system on an NVIDIA Jetson AGX Orin Developer Kit to simulate in-vehicle devices and evaluated its performance in terms of response time, accuracy of in-vehicle control API execution, and function calling success rate. Experimental results demonstrate that the proposed system achieves in-vehicle API execution with acceptable latency, improving usability and safety by enabling seamless and natural voice interactions without relying on internet connectivity.