Self-Attention Pooling-Based Long-Term Temporal Network for Action Recognition

Huifang Li, Jingwei Huang, Mengchu Zhou, Qisong Shi, Qing Fei

Research output: Contribution to journalArticlepeer-review

8 Scopus citations


With the development of Internet of Things (IoT), self-driving technology has been successful. Yet safe driving faces challenges due to such cases as pedestrians crossing roads. How to sense their movements and identify their behaviors from video data is important. Most of the existing methods fail to: 1) capture long-term temporal relationship well due to their limited temporal coverage and 2) aggregate discriminative representation effectively, such as caused by little or even no attention paid to differences among representations. To address such issues, this work presents a new architecture called a self-attention pooling-based long-term temporal network (SP-LTN), which can learn long-term temporal representations and aggregate those discriminative representations in an end-to-end manner, and on the other hand, effectively conduct long-term representation learning on a given video by capturing spatial information and mining temporal patterns. Next, it develops a self-attention pooling method to predict the importance scores of obtained representations for distinguishing them from each other and then weights them together to highlight the contributions of those discriminative representations in action recognition. Finally, it designs a new loss function that combines a standard cross-entropy loss function with a regularization term to further focus on the discriminative representations while restraining the impact of distractive ones on activity classification. Experimental results on two data sets show that our SP-LTN, fed by only red-green-blue (RGB) frames, outperforms the state-of-the-art methods.

Original languageEnglish (US)
Pages (from-to)65-77
Number of pages13
JournalIEEE Transactions on Cognitive and Developmental Systems
Issue number1
StatePublished - Mar 1 2023

All Science Journal Classification (ASJC) codes

  • Software
  • Artificial Intelligence


  • Action recognition
  • deep learning (DL)
  • long-term temporal networks (LTNs)
  • self-attention pooling


Dive into the research topics of 'Self-Attention Pooling-Based Long-Term Temporal Network for Action Recognition'. Together they form a unique fingerprint.

Cite this