Human skeleton posture reconstruction is an essential component for human–computer interactions (HCI) in various application domains. Traditional approaches usually rely on either cameras or on-body sensors, which induce privacy concerns or inconvenient practical setups. To address these practical concerns, this paper proposes a low-cost contactless skeleton posture reconstruction system, mPose, which can reconstruct a user's 3D skeleton postures using a single mmWave device. mPose does not require the user to wear any sensors and can enable a broad range of emerging mobile applications (e.g., VR gaming and pervasive user input) via mmWave-5G ready Internet of Things (IoT) devices. Particularly, the system extracts multi-dimensional spatial information from mmWave signals which characterizes the skeleton postures in a 3D space. To mitigate the impacts of environmental changes, mPose dynamically detects the user location and extracts spatial features from the mmWave signals reflected only from the user. Furthermore, we develop a deep regression method with a domain discriminator to learn a mapping between the spatial features and the joint coordinates of human body while removing subject-specific characteristics, realizing robust posture reconstruction across users. Extensive experiments, involving 17 representative body postures, 7 subjects, and 3 indoor environments, show that mPose outperforms contemporary state-of-the-art RF-based solutions with a lower average joint error of only ∼30 mm, while achieving transferability across environments and subjects at the same time.
All Science Journal Classification (ASJC) codes
- Medicine (miscellaneous)
- Information Systems
- Health Informatics
- Computer Science Applications
- Health Information Management
- 3D skeleton posture reconstruction