An Adaptive Deep Belief Network with Sparse Restricted Boltzmann Machines

Gongming Wang, Junfei Qiao, Jing Bi, Qing Shan Jia, Meng Chu Zhou

Research output: Contribution to journalArticlepeer-review

15 Scopus citations

Abstract

Deep belief network (DBN) is an efficient learning model for unknown data representation, especially nonlinear systems. However, it is extremely hard to design a satisfactory DBN with a robust structure because of traditional dense representation. In addition, backpropagation algorithm-based fine-tuning tends to yield poor performance since its ease of being trapped into local optima. In this article, we propose a novel DBN model based on adaptive sparse restricted Boltzmann machines (AS-RBM) and partial least square (PLS) regression fine-tuning, abbreviated as ARP-DBN, to obtain a more robust and accurate model than the existing ones. First, the adaptive learning step size is designed to accelerate an RBM training process, and two regularization terms are introduced into such a process to realize sparse representation. Second, initial weight derived from AS-RBM is further optimized via layer-by-layer PLS modeling starting from the output layer to input one. Third, we present the convergence and stability analysis of the proposed method. Finally, our approach is tested on Mackey-Glass time-series prediction, 2-D function approximation, and unknown system identification. Simulation results demonstrate that it has higher learning accuracy and faster learning speed. It can be used to build a more robust model than the existing ones.

Original languageEnglish (US)
Article number8941264
Pages (from-to)4217-4228
Number of pages12
JournalIEEE Transactions on Neural Networks and Learning Systems
Volume31
Issue number10
DOIs
StatePublished - Oct 2020

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence

Keywords

  • Adaptive-sparse restricted Boltzmann machine (RBM)
  • convergence analysis
  • deep belief network (DBN)
  • partial least square (PLS)-based regression fine-tuning
  • robust structure

Fingerprint

Dive into the research topics of 'An Adaptive Deep Belief Network with Sparse Restricted Boltzmann Machines'. Together they form a unique fingerprint.

Cite this