TY - JOUR
T1 - An Adaptive Deep Belief Network with Sparse Restricted Boltzmann Machines
AU - Wang, Gongming
AU - Qiao, Junfei
AU - Bi, Jing
AU - Jia, Qing Shan
AU - Zhou, Meng Chu
N1 - Funding Information:
Manuscript received April 9, 2019; revised August 30, 2019; accepted November 2, 2019. Date of publication December 24, 2019; date of current version October 6, 2020. This work was supported in part by the Key Project of National Natural Science Foundation of China under Grant 61533002, in part by the National Natural Science Foundation of China under Grant 61703011 and Grant 61673229, in part by the Major Project for New Generation Artificial Intelligence under Grant 2018AAA0101600, and in part by the National Science and Technology Major Project under Grant 2018ZX07111005. (Corresponding author: Junfei Qiao.) G. Wang and Q.-S. Jia are with the Center for Intelligent and Networked Systems, Department of Automation, Tsinghua University, Beijing 100084, China (e-mail: wanggm@tsinghua.edu.cn; jiaqs@tsinghua.edu.cn).
Publisher Copyright:
© 2012 IEEE.
PY - 2020/10
Y1 - 2020/10
N2 - Deep belief network (DBN) is an efficient learning model for unknown data representation, especially nonlinear systems. However, it is extremely hard to design a satisfactory DBN with a robust structure because of traditional dense representation. In addition, backpropagation algorithm-based fine-tuning tends to yield poor performance since its ease of being trapped into local optima. In this article, we propose a novel DBN model based on adaptive sparse restricted Boltzmann machines (AS-RBM) and partial least square (PLS) regression fine-tuning, abbreviated as ARP-DBN, to obtain a more robust and accurate model than the existing ones. First, the adaptive learning step size is designed to accelerate an RBM training process, and two regularization terms are introduced into such a process to realize sparse representation. Second, initial weight derived from AS-RBM is further optimized via layer-by-layer PLS modeling starting from the output layer to input one. Third, we present the convergence and stability analysis of the proposed method. Finally, our approach is tested on Mackey-Glass time-series prediction, 2-D function approximation, and unknown system identification. Simulation results demonstrate that it has higher learning accuracy and faster learning speed. It can be used to build a more robust model than the existing ones.
AB - Deep belief network (DBN) is an efficient learning model for unknown data representation, especially nonlinear systems. However, it is extremely hard to design a satisfactory DBN with a robust structure because of traditional dense representation. In addition, backpropagation algorithm-based fine-tuning tends to yield poor performance since its ease of being trapped into local optima. In this article, we propose a novel DBN model based on adaptive sparse restricted Boltzmann machines (AS-RBM) and partial least square (PLS) regression fine-tuning, abbreviated as ARP-DBN, to obtain a more robust and accurate model than the existing ones. First, the adaptive learning step size is designed to accelerate an RBM training process, and two regularization terms are introduced into such a process to realize sparse representation. Second, initial weight derived from AS-RBM is further optimized via layer-by-layer PLS modeling starting from the output layer to input one. Third, we present the convergence and stability analysis of the proposed method. Finally, our approach is tested on Mackey-Glass time-series prediction, 2-D function approximation, and unknown system identification. Simulation results demonstrate that it has higher learning accuracy and faster learning speed. It can be used to build a more robust model than the existing ones.
KW - Adaptive-sparse restricted Boltzmann machine (RBM)
KW - convergence analysis
KW - deep belief network (DBN)
KW - partial least square (PLS)-based regression fine-tuning
KW - robust structure
UR - http://www.scopus.com/inward/record.url?scp=85077258713&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85077258713&partnerID=8YFLogxK
U2 - 10.1109/TNNLS.2019.2952864
DO - 10.1109/TNNLS.2019.2952864
M3 - Article
C2 - 31880561
AN - SCOPUS:85077258713
SN - 2162-237X
VL - 31
SP - 4217
EP - 4228
JO - IEEE Transactions on Neural Networks
JF - IEEE Transactions on Neural Networks
IS - 10
M1 - 8941264
ER -