Beyond Accuracy: Performance and Behavioral Evaluation of Multimodal AI for Suspicious Aerial Traffic Monitoring
Gabriel Dietzsch, Elcio Hideiti Shiguemori · Biblioteca Digital da Memória Científica do INPE (National Institute for Space Research) · 2026
This paper evaluates multimodal AI models (Gemini and ChatGPT) for visual-based aerial trajectory classification, comparing them against a neural network baseline. Beyond standard metrics, we introduce behavioral criteria to assess reliability in safety-critical contexts. Results show that multimodal AI significantly outperforms the baseline (96% vs. 75% accuracy). However, higher accuracy did not equate to safer behavior. The top-performing model displayed systematic overconfi-dence and conceptual hallucinations, while the second model exhibited useful doubt, effectively communicating uncertainty in ambiguous cases. We conclude that while high-recall models suit automated pipelines, uncertainty-aware models are superior for human-in-the-loop scenarios. This work demonstrates that behavioral evaluationincluding confidence calibration and semantic interpretationis crucial for deploying multimodal AI in aerial surveillance.