Enhancing Visual-LLM through Prompt Engineering and Hybrid Retrieval-augmented Generation for Site Safety Compliance Checking
Kairou Guo, Peter Kok-Yiu Wong, Jack C.P. Cheng, Xingyu Tao, Pak-Him Leung Pak-Him Leung · Computing in construction · 2025
The increasing prevalence of safety incidents on construction sites states the urgent need for enhanced monitoring. This study proposes an innovative hybrid Retrieval-Augmented Generation (RAG) algorithm to compliance check accuracy for site images. By integrating the Visual Language Model (VLM), we developed an algorithm capable of mastering domain knowledge without fine-tuning and addressing the limitation of interpreting RAG technology with visual information. A three-phased prompting framework was designed to enhance the VLM's compliance analysis abilities. Experiments based on actual construction site in Hong Kong demonstrated 21.89% increase in retrieval accuracy.