Self-attention Pooling-based Long-term Temporal Network for Action Recognition

Huifang Li, Jingwei Huang, Mengchu Zhou, Qisong Shi, Qing Fei

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


With the development of Internet of Things (IoT), self-driving technology has been successful. Yet safe driving faces challenges due to such cases as pedestrians crossing roads. How to sense their movements and identify their behaviors from video data is important. Most of existing methods fail to: a) capture long-term temporal relationship well due to their limited temporal coverage, and b) aggregate discriminative representation effectively, such as caused by little or even no attention paid to differences among representations. To address such issues, this work presents a new architecture called a Self-attention Pooling-based Long-term Temporal Network (SPLTN), which can learn long-term temporal representations and aggregate those discriminative representations in an end-to-end manner, and on the other hand, effectively conduct long-term representation learning on a given video by capturing spatial information and mining temporal patterns. Next, it develops a self-attention pooling method to predict the importance scores of obtained representations for distinguishing them from each other, and then weights them together to highlight the contributions of those discriminative representations in action recognition. Finally, it gives a new loss function that combines a standard crossentropy loss function with a regularization term to further focus on the discriminative representations while restraining the impact of distractive ones on activity classification. Experimental results on two datasets show that our SP-LTN, fed by only Red-Green-Blue (RGB) frames outperforms the state-of-the-art methods.

Original languageEnglish (US)
JournalIEEE Transactions on Cognitive and Developmental Systems
StateAccepted/In press - 2022

All Science Journal Classification (ASJC) codes

  • Software
  • Artificial Intelligence


  • Action Recognition
  • Deep Learning
  • Feature extraction
  • Internet of Things
  • Long-term Temporal Networks
  • Predictive models
  • Representation learning
  • Roads
  • Self-attention Pooling.
  • Three-dimensional displays
  • Videos


Dive into the research topics of 'Self-attention Pooling-based Long-term Temporal Network for Action Recognition'. Together they form a unique fingerprint.

Cite this