mPose: Environment- and subject-agnostic 3D skeleton posture reconstruction leveraging a single mmWave device

Cong Shi, Li Lu, Jian Liu, Yan Wang, Yingying Chen, Jiadi Yu

Research output: Contribution to journalArticlepeer-review

28 Scopus citations

Abstract

Human skeleton posture reconstruction is an essential component for human–computer interactions (HCI) in various application domains. Traditional approaches usually rely on either cameras or on-body sensors, which induce privacy concerns or inconvenient practical setups. To address these practical concerns, this paper proposes a low-cost contactless skeleton posture reconstruction system, mPose, which can reconstruct a user's 3D skeleton postures using a single mmWave device. mPose does not require the user to wear any sensors and can enable a broad range of emerging mobile applications (e.g., VR gaming and pervasive user input) via mmWave-5G ready Internet of Things (IoT) devices. Particularly, the system extracts multi-dimensional spatial information from mmWave signals which characterizes the skeleton postures in a 3D space. To mitigate the impacts of environmental changes, mPose dynamically detects the user location and extracts spatial features from the mmWave signals reflected only from the user. Furthermore, we develop a deep regression method with a domain discriminator to learn a mapping between the spatial features and the joint coordinates of human body while removing subject-specific characteristics, realizing robust posture reconstruction across users. Extensive experiments, involving 17 representative body postures, 7 subjects, and 3 indoor environments, show that mPose outperforms contemporary state-of-the-art RF-based solutions with a lower average joint error of only ∼30 mm, while achieving transferability across environments and subjects at the same time.

Original languageEnglish (US)
Article number100228
JournalSmart Health
Volume23
DOIs
StatePublished - Mar 2022
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Medicine (miscellaneous)
  • Information Systems
  • Health Informatics
  • Computer Science Applications
  • Health Information Management

Keywords

  • 3D skeleton posture reconstruction
  • MmWave

Fingerprint

Dive into the research topics of 'mPose: Environment- and subject-agnostic 3D skeleton posture reconstruction leveraging a single mmWave device'. Together they form a unique fingerprint.

Cite this