On-Premise Multimodal AI Assistance for Operator-in-the-Loop Diagnosis in Machine Tool Mechatronic Systems
Seongwoo Cho, Jongsu Park, Jumyung Um · Applied Sciences · 2026
Modern machine tools are safety-critical mechatronic systems, yet shop floor maintenance from abnormal events still relies heavily on scarce expert know-how and time-consuming manual searches across heterogeneous controller documentation. This paper presents an on-premise multimodal AI assistant. It integrates large language models with retrieval augmented generation and real-time machine signals to support operator-in-the-loop fault diagnosis. The proposed system provides three tightly coupled functions: (1) alarm-grounded guidance, which answers controller alarms and recommends corrective actions by grounding generation on manuals, maintenance procedures, and historical alarm cases; (2) parameter-aware reasoning, which injects live process and health indicators (e.g., spindle temperature, vibration, and axis states) into the reasoning context through an industrial data pipeline, enabling context specific troubleshooting; and (3) vision enabled support, which retrieves similar visual cases and generates concise visual instructions when text alone is insufficient. The assistant is deployed within an intranet environment to satisfy industrial security and privacy requirements and is orchestrated via lightweight tool calling for seamless integration with existing shop floor systems. Experiments on real machine tool alarm scenarios demonstrate that the proposed system achieves 82% answer correctness for alarm Q&A and improves response consistency and time-to-resolution compared with baseline keyword search and template-based guidance. The results suggest that grounded, multimodal chatbot assistants can act as practical AI-based feedback and decision support mechanisms for mechatronic production equipment, bridging human skill gaps while enhancing reliability and maintainability.