On Fatigue Detection for Air Traffic Controllers Based on Fuzzy Fusion of Multiple Features

Yi Hu, Zhuo Liu, Aiqin Hou, Chase Wu, Wenbin Wei, Yanjun Wang, Min Liu

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


Fatigue detection for air traffic controllers is an important yet challenging problem in aviation safety research. Most of the existing methods for this problem are based on facial features. In this paper, we propose an ensemble learning model that combines both facial features and voice features and design a fatigue detection method through multifeature fusion, referred to as Facial and Voice Stacking (FV-Stacking). Specifically, for facial features, we first use OpenCV and Dlib libraries to extract mouth and eye areas and then employ a combination of M-Convolutional Neural Network (M-CNN) and E-Convolutional Neural Network (E-CNN) to determine the state of mouth and eye closure based on five features, i.e., blinking times, average blinking time, average blinking interval, Percentage of Eyelid Closure over the Pupil over Time (PERCLOS), and Frequency of Open Mouth (FOM). For voice features, we extract the Mel-Frequency Cepstral Coefficients (MFCC) features of speech. Such facial features and voice features are fused through a carefully designed stacking model for fatigue detection. Real-life experiments are conducted on 14 air traffic controllers in Southwest Air Traffic Management Bureau of Civil Aviation of China. The results show that the proposed FV-Stacking method achieves a detection accuracy of 97%, while the best accuracy achieved by a single model is 92% and the best accuracy achieved by the state-of-the-art detection methods is 88%.

Original languageEnglish (US)
Article number4911005
JournalComputational and Mathematical Methods in Medicine
StatePublished - 2022

All Science Journal Classification (ASJC) codes

  • Modeling and Simulation
  • General Biochemistry, Genetics and Molecular Biology
  • General Immunology and Microbiology
  • Applied Mathematics


Dive into the research topics of 'On Fatigue Detection for Air Traffic Controllers Based on Fuzzy Fusion of Multiple Features'. Together they form a unique fingerprint.

Cite this