Neural correlates of affective context in facial expression analysis: A simultaneous EEG-fNIRS study

Yanjia Sun, Hasan Ayaz, Ali N. Akansu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

16 Scopus citations

Abstract

We present a framework to measure the correlation between spontaneous human facial affective expressions and relevant brain activity. The affective states were registered from the video capture of facial expression and related neural activity was measured using wearable and portable neuroimaging systems: functional near infrared spectroscopy (fNIRS), electroencephalography (EEG) to asses both hemodynamic and electrophysiological responses. The methodology involves the simultaneous detection and comparison of various affective expressions by multi-modalities and classification of spatio-temporal data with neural signature traits. The experimental results show strong correlation between the spontaneous facial affective expressions and the affective states related brain activity. We propose a multimodal approach to jointly evaluate fNIRS signals and EEG signals for affective state detection. Results indicate that proposed method with fNIRS+EEG improves performance over fNIRS or EEG only approaches. These findings encourage further studies of the joint utilization of video and brain signals for face perception and brain-computer interface (BCI) applications.

Original languageEnglish (US)
Title of host publication2015 IEEE Global Conference on Signal and Information Processing, GlobalSIP 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages820-824
Number of pages5
ISBN (Electronic)9781479975914
DOIs
StatePublished - Feb 23 2016
EventIEEE Global Conference on Signal and Information Processing, GlobalSIP 2015 - Orlando, United States
Duration: Dec 13 2015Dec 16 2015

Publication series

Name2015 IEEE Global Conference on Signal and Information Processing, GlobalSIP 2015

Other

OtherIEEE Global Conference on Signal and Information Processing, GlobalSIP 2015
Country/TerritoryUnited States
CityOrlando
Period12/13/1512/16/15

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Signal Processing

Keywords

  • electroencephalography
  • facial emotion recognition
  • functional near infrared spectroscopy

Fingerprint

Dive into the research topics of 'Neural correlates of affective context in facial expression analysis: A simultaneous EEG-fNIRS study'. Together they form a unique fingerprint.

Cite this