A Multi-Expert VLM Framework for Cognitive Robotic Manipulation in semi-structured Manufacturing
Petar Tesic, Oliver Petrović, Christian Brecher · Procedia CIRP · 2026
In industrial manufacturing, many seemingly simple handling and sorting tasks remain unautomated due to their semi-structured and variable nature. While such operations are common in assembly, logistics and commissioning, conventional automation solutions lack the flexibility and cognitive capabilities required to adapt to changing objects, layouts and task conditions. The recent emergence of large language models (LLMs) and vision language models (VLMs) offers new opportunities to overcome these limitations by enabling natural interaction, general reasoning and multimodal perception within robotic systems. This work introduces a cognitive robotics framework using foundation-level VLMs for flexible automation in semi-structured manufacturing through No-Code programming. The architecture unifies perception, interaction and planning through specialized experts and handles previously unseen parts while also leveraging available CAD/CAM information of known objects. Through multimodal input, shared memory and closed-loop reasoning, the system unifies language-guided planning and rule-based motion control into an adaptive framework.