GazeLight: Understanding the Feasibility of Gaze-based Adaptive Streaming for AR-Mediated Human-Robot Systems

While recent advances in robotics have led to superior perception, reasoning, and decision making capabilities, two key challenges hinder widespread practical use, specifically, in the context of augmented reality (AR) mediated human-robot collaboration (HRC). First, mobile robots and AR devices, similar to any other mobile system, are constrained in terms of their hardware resources such as battery availability, processing power, and memory, which affects their performance (e.g., reduced operation time when using sophisticated DNNs for perception). Second, systems that are human-facing typically have stringent end-to-end latency requirements (e.g., ≤ 100 ms) for acceptable use. To this end, we posit that a combination of human context-driven and edge AI-driven optimizations can alleviate such challenges towards realizing collaborative, real-time systems. In this work, we present empirical insights from two HRC case studies, gesture-driven teleoperation, and remote robotic instruction, to demonstrate how gaze-based visual attention, an exemplar human context, can be used to optimize end-to-end point-of-view streaming latency, a key performance bottleneck in HRC systems. Our simple gaze context-based adaptation scheme shows that the total latency can be reduced to 121.36 ms (≈63.28% reduction from the full-sized image case) and is only slightly over the 100 ms target.