LLM-Based Retrieval-Augmented Generation (RAG) Pipeline with Multi-Modal RGB-D Sensing Fusion and Deep Learning Computer Vision for Automated Monitoring, Scene Understanding, and Mitigation of Worker-Machinery Proximity Safety Issues on Construction Jobsites

Worker-machinery proximity is a key contributor to safety incidents in the construction industry. This study proposes a non-intrusive and cost-effective retrieval-augmentation generation (RAG) pipeline for proximity work safety monitoring, enhanced by automated unsafe scene understanding and mitigation solution recommendation. The proposed pipeline aims to automatically (1) perform worker and machinery detection and proximity estimation using deep learning computer vision and RGB-D sensing fusion, (2) conduct query on the imagery data for scene information generation through vision-to-text mechanism, and (3) identify the potential safety hazards with recommended solutions through large language models (LLMs)-based RAG. Experimental results indicated that the proposed pipeline could accurately detect workers and machinery in real time, effectively estimate proximity information, and reliably generate scene understanding information with identified safety hazards and mitigation actions/solutions. This study offers a new RAG solution to advancing proximity-related construction safety management practices towards a more cost-effective, less complex, and more informative manner.