InternDrive: A Multimodal Large Language Model for Autonomous Driving Scenario Understanding
Ye Zhang, Yiming Nie · 2024
With the rapid development of autonomous driving technology, accurately understanding complex driving scenarios has become a critical challenge. Existing computer vision-based solutions exhibit limitations when dealing with dynamic driving environments. Therefore, this paper proposes a method for understanding autonomous driving scenarios using multimodal large language models. Firstly, we designed a set of questions to guide multimodal large language models in comprehensively understanding driving scenarios, and based on this, we constructed a multimodal driving scenario dataset. This dataset combines open-source nuScenes image data with natural language annotations automatically generated and manually reviewed via the OpenAI API. Subsequently, we conducted visual instruction tuning on the open-source multimodal large language model InternVL-1.5 and proposed the InternDrive model. Furthermore, this paper introduces an evaluation method based on a proprietary large model and conducts a comprehensive assessment of InternDrive's ability to understand driving scenarios. Experimental results demonstrate that InternDrive exhibits superior accuracy in multiple driving scenario understanding tasks. Our research provides new methods and perspectives for enhancing the scene understanding capabilities of autonomous driving systems and showcases the potential application of multimodal large language models in the field of autonomous driving.