TY - GEN
T1 - DeepMix
T2 - 20th ACM International Conference on Mobile Systems, Applications and Services, MobiSys 2022
AU - Guan, Yongjie
AU - Hou, Xueyu
AU - Wu, Nan
AU - Han, Bo
AU - Han, Tao
N1 - Publisher Copyright:
© 2022 ACM.
PY - 2022/6/27
Y1 - 2022/6/27
N2 - Mobile headsets should be capable of understanding 3D physical environments to offer a truly immersive experience for augmented/mixed reality (AR/MR). However, their small form-factor and limited computation resources make it extremely challenging to execute in real-time 3D vision algorithms, which are known to be more compute-intensive than their 2D counterparts. In this paper, we propose DeepMix, a mobility-aware, lightweight, and hybrid 3D object detection framework for improving the user experience of AR/MR on mobile headsets. Motivated by our analysis and evaluation of state-of-the-art 3D object detection models, DeepMix intelligently combines edge-assisted 2D object detection and novel, on-device 3D bounding box estimations that leverage depth data captured by headsets. This leads to low end-to-end latency and significantly boosts detection accuracy in mobile scenarios. A unique feature of DeepMix is that it fully exploits the mobility of headsets to fine-tune detection results and boost detection accuracy. To the best of our knowledge, DeepMix is the first 3D object detection that achieves 30 FPS (i.e., an end-to-end latency much lower than the 100 ms stringent requirement of interactive AR/MR). We implement a prototype of DeepMix on Microsoft HoloLens and evaluate its performance via both extensive controlled experiments and a user study with 30+ participants. DeepMix not only improves detection accuracy by 9.1 - 37.3% but also reduces end-to-end latency by 2.68 - 9.15×, compared to the baseline that uses existing 3D object detection models.
AB - Mobile headsets should be capable of understanding 3D physical environments to offer a truly immersive experience for augmented/mixed reality (AR/MR). However, their small form-factor and limited computation resources make it extremely challenging to execute in real-time 3D vision algorithms, which are known to be more compute-intensive than their 2D counterparts. In this paper, we propose DeepMix, a mobility-aware, lightweight, and hybrid 3D object detection framework for improving the user experience of AR/MR on mobile headsets. Motivated by our analysis and evaluation of state-of-the-art 3D object detection models, DeepMix intelligently combines edge-assisted 2D object detection and novel, on-device 3D bounding box estimations that leverage depth data captured by headsets. This leads to low end-to-end latency and significantly boosts detection accuracy in mobile scenarios. A unique feature of DeepMix is that it fully exploits the mobility of headsets to fine-tune detection results and boost detection accuracy. To the best of our knowledge, DeepMix is the first 3D object detection that achieves 30 FPS (i.e., an end-to-end latency much lower than the 100 ms stringent requirement of interactive AR/MR). We implement a prototype of DeepMix on Microsoft HoloLens and evaluate its performance via both extensive controlled experiments and a user study with 30+ participants. DeepMix not only improves detection accuracy by 9.1 - 37.3% but also reduces end-to-end latency by 2.68 - 9.15×, compared to the baseline that uses existing 3D object detection models.
KW - 3D object detection
KW - augmented and mixed reality
KW - hybrid mobile vision
KW - mobile headsets
UR - http://www.scopus.com/inward/record.url?scp=85134082504&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85134082504&partnerID=8YFLogxK
U2 - 10.1145/3498361.3538945
DO - 10.1145/3498361.3538945
M3 - Conference contribution
AN - SCOPUS:85134082504
T3 - MobiSys 2022 - Proceedings of the 2022 20th Annual International Conference on Mobile Systems, Applications and Services
SP - 28
EP - 41
BT - MobiSys 2022 - Proceedings of the 2022 20th Annual International Conference on Mobile Systems, Applications and Services
PB - Association for Computing Machinery, Inc
Y2 - 27 June 2022 through 1 July 2022
ER -