Hierarchical Vision Transformer-Based Deep Learning Architecture, RGB-D Sensing Fusion, and Multimodal LLM-Based Generative AI Pipeline for Automated Pavement Pothole Segmentation, Quantification, and Repair Recommendations

This paper proposes an automated approach for pavement pothole segmentation, quantification, and repair solution generation. The proposed approach integrates a deep learning (DL) architecture with hierarchical vision transformer (HViT) based on the SAM2-UNet architecture, RGB-D sensing fusion, and a multimodal large language model (LLM). It is designed to automatically (1) perform pothole segmentation on RGB images using a HViT-based DL model that allows multi-scale capturing of pavement pothole features, (2) retrieve comprehensive pothole properties through RGB-D fusion and point cloud processing, and (3) generate pothole-induced pavement repair solutions using multimodal LLM. Experimental results showed that the developed HViT-based DL model (i.e., SAM2-UNet) yields superior performance of pothole segmentation with mDice of 0.965 and mIoU of 0.937, while the integrated LLM can generate reasonable pothole-induced pavement repair recommendations. This paper contributes to the body of knowledge by offering a novel approach to advance automated pavement inspection towards more accurate, intelligent, and actionable direction.