A Smart Cyber-Physical System Combining Large Language Models (LLMs) and Augmented Reality (AR) to Enhance Human Building Interaction during Building Operations and Maintenance Procedures

The operation and maintenance of buildings are often time-sensitive and accuracy-dependent, requiring rapid and precise information processing. Oral communication supported by cognitive assistance can facilitate accurate decision-making, improving user experiences with building facilities and overall human-building interaction (HBI). This paper presents a novel cyber-physical system (CPS) integrating augmented reality (AR) and large language models (LLMs) to enable users to communicate orally with LLMs, providing guidance in diagnosing and resolving issues in building facilities, such as MEP or HVAC systems. The process involves converting user speech to text via custom libraries, transmitting the text to an LLM hosted on a web server through wireless communication, and displaying step-by-step guidance in the AR headset. Results demonstrated that the proposed CPS significantly improves user interaction with building facilities, enhancing situational awareness, task efficiency, and user satisfaction. This research introduces an innovative AR-LLM integration CPS that promotes a more engaging, personalized, and efficient HBI framework.