Falcon: A Mobile Semantic Visual Perception Framework Enlightened by Human Vision Systems

In the realm of computer human visual perception, semantic perception means recognizing objects, people, and scenes. This involves not just detecting shapes and colors but understanding what those visual elements represent. Despite its vital role in enhancing various aspects of emerging applications such as safety for autonomous driving and immersion for mixed reality (MR), real-time segmentation on mobile and edge platforms is challenging due to the nature of dense pixel labeling. To address this issue, we propose Falcon, a lightweight focus-aware segmentation framework that effectively integrates multiple innovations to achieve real-time segmentation on resource-constrained mobile and edge devices. We design a novel, low-dimension feature for efficient pixel labeling with shallow neural networks, an agile focus-aware refining scheme to compensate for the coarse nature of holistic segmentation, and a modularized design to accommodate the diversity of mobile and edge platforms and ensure seamless integration with different segmentation models. We build a prototype implementation of Falcon that supports both on-device executions and edge-assisted offloading, and asynchronous segmentation and refinement. We extensively evaluate the performance of Falcon for autonomous driving and MR applications with real setups and standard datasets. Our results demonstrate that Falcon achieves real-time segmentation, with an impressive rate of up to 40 frames per second.