TY - JOUR
T1 - TL-GDBN
T2 - Growing Deep Belief Network with Transfer Learning
AU - Wang, Gongming
AU - Qiao, Junfei
AU - Bi, Jing
AU - Li, Wenjing
AU - Zhou, Mengchu
N1 - Funding Information:
Manuscript received January 23, 2018; revised May 1, 2018; accepted July 7, 2018. Date of publication October 2, 2018; date of current version April 5, 2019. This work was supported in part by the Key Project of National Natural Science Foundation of China under Grant 61533002, in part by the National Natural Science Foundation of China under Grant 61802015, Grant 61703011 and Grant 61603009, and in part by the National Science and Technology Major Project under Grant 2018ZX07111005. This paper was recommended for publication by Associate Editor J. Mitsugi and Editor M. P. Fanti upon evaluation of the reviewers’ comments. (Corresponding author: Jing Bi.) G. Wang, J. Qiao, and W. Li are with the Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China, and also with the Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing 100124, China (e-mail: wanggm@emails.bjut.edu.cn; junfeiq@ bjut.edu.cn; wenjing.li@bjut.edu.cn).
PY - 2019/4
Y1 - 2019/4
N2 - A deep belief network (DBN) is effective to create a powerful generative model by using training data. However, it is difficult to fast determine its optimal structure given specific applications. In this paper, a growing DBN with transfer learning (TL-GDBN) is proposed to automatically decide its structure size, which can accelerate its learning process and improve model accuracy. First, a basic DBN structure with single hidden layer is initialized and then pretrained, and the learned weight parameters are frozen. Second, TL-GDBN uses TL to transfer the knowledge from the learned weight parameters to newly added neurons and hidden layers, which can achieve a growing structure until the stopping criterion for pretraining is satisfied. Third, the weight parameters derived from pretraining of TL-GDBN are further fine-tuned by using layer-by-layer partial least square regression from top to bottom, which can avoid many problems of traditional backpropagation algorithm-based fine-tuning. Moreover, the convergence analysis of the TL-GDBN is presented. Finally, TL-GDBN is tested on two benchmark data sets and a practical wastewater treatment system. The simulation results show that it has better modeling performance, faster learning speed, and more robust structure than existing models. Note to Practitioners - Transfer learning (TL) aims to improve training effectiveness by transferring knowledge from a source domain to target domain. This paper presents a growing deep belief network (DBN) with TL to improve the training effectiveness and determine the optimal model size. Facing a complex process and real-world workflow, DBN tends to require long time for its successful training. The proposed growing DBN with TL (TL-GDBN) accelerates the learning process by instantaneously transferring the knowledge from a source domain to each new deeper or wider substructure. The experimental results show that the proposed TL-GDBN model has a great potential to deal with complex system, especially the systems with high nonlinearity. As a result, it can be readily applicable to some industrial nonlinear systems.
AB - A deep belief network (DBN) is effective to create a powerful generative model by using training data. However, it is difficult to fast determine its optimal structure given specific applications. In this paper, a growing DBN with transfer learning (TL-GDBN) is proposed to automatically decide its structure size, which can accelerate its learning process and improve model accuracy. First, a basic DBN structure with single hidden layer is initialized and then pretrained, and the learned weight parameters are frozen. Second, TL-GDBN uses TL to transfer the knowledge from the learned weight parameters to newly added neurons and hidden layers, which can achieve a growing structure until the stopping criterion for pretraining is satisfied. Third, the weight parameters derived from pretraining of TL-GDBN are further fine-tuned by using layer-by-layer partial least square regression from top to bottom, which can avoid many problems of traditional backpropagation algorithm-based fine-tuning. Moreover, the convergence analysis of the TL-GDBN is presented. Finally, TL-GDBN is tested on two benchmark data sets and a practical wastewater treatment system. The simulation results show that it has better modeling performance, faster learning speed, and more robust structure than existing models. Note to Practitioners - Transfer learning (TL) aims to improve training effectiveness by transferring knowledge from a source domain to target domain. This paper presents a growing deep belief network (DBN) with TL to improve the training effectiveness and determine the optimal model size. Facing a complex process and real-world workflow, DBN tends to require long time for its successful training. The proposed growing DBN with TL (TL-GDBN) accelerates the learning process by instantaneously transferring the knowledge from a source domain to each new deeper or wider substructure. The experimental results show that the proposed TL-GDBN model has a great potential to deal with complex system, especially the systems with high nonlinearity. As a result, it can be readily applicable to some industrial nonlinear systems.
KW - Convergence analysis
KW - TL
KW - deep belief network (DBN)
KW - growing DBN with transfer learning (TL-GDBN)
KW - partial least square regression (PLSR)-based fine-tuning
UR - http://www.scopus.com/inward/record.url?scp=85054471522&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85054471522&partnerID=8YFLogxK
U2 - 10.1109/TASE.2018.2865663
DO - 10.1109/TASE.2018.2865663
M3 - Article
AN - SCOPUS:85054471522
SN - 1545-5955
VL - 16
SP - 874
EP - 885
JO - IEEE Transactions on Automation Science and Engineering
JF - IEEE Transactions on Automation Science and Engineering
IS - 2
M1 - 8478799
ER -