Viso-Gen: A Wireless Gaze-Assisted and Voice-Controlled Generative AI System Leveraging Stable Diffusion for Real-Time Image Generation for Civil Engineering Design and Visualization

This research presents "Viso-Gen,"a novel system combining custom wireless eye-tracking technology and generative artificial intelligence (AI) to enable architects, designers, and engineers to visualize and iteratively refine design concepts in real time. Viso-Gen is capable of tracking gaze points, identifying areas of interest, and leveraging generative AI (stable diffusion)-based image inpainting and generation. The proposed system utilizes a custom-built eye tracker to capture user gaze points and overlays gaze data onto a synchronized scene camera feed. A dynamic mask is generated around the points of focus to define the area to be modified. Viso-Gen relies on voice commands and instructions (natural language processing-based speech transcription) to trigger the stable diffusion-based image inpainting algorithms. The inpainting and eye-tracking process is optimized to work with parallel processing and maintain a low-latency system. The results are rendered in real time and made accessible through a browser-based platform, allowing for visualization on mobile devices and browsers.