Improved Linear Convergence of Training CNNs with Generalizability Guarantees: A One-Hidden-Layer Case

Shuai Zhang, Meng Wang, Jinjun Xiong, Sijia Liu, Pin Yu Chen

Research output: Contribution to journalArticlepeer-review

8 Scopus citations

Abstract

We analyze the learning problem of one-hidden-layer nonoverlapping convolutional neural networks with the rectified linear unit (ReLU) activation function from the perspective of model estimation. The training outputs are assumed to be generated by the neural network with the unknown ground-truth parameters plus some additive noise, and the objective is to estimate the model parameters by minimizing a nonconvex squared loss function of the training data. Assuming that the training set contains a finite number of samples generated from the Gaussian distribution, we prove that the accelerated gradient descent (GD) algorithm with a proper initialization converges to the ground-truth parameters (up to the noise level) with a linear rate even though the learning problem is nonconvex. Moreover, the convergence rate is proved to be faster than the vanilla GD. The initialization can be achieved by the existing tensor initialization method. In contrast to the existing works that assume an infinite number of samples, we theoretically establish the sample complexity of the required number of training samples. Although the neural network considered here is not deep, this is the first work to show that accelerated GD algorithms can find the global optimizer of the nonconvex learning problem of neural networks. This is also the first work that characterizes the sample complexity of gradient-based methods in learning convolutional neural networks with the nonsmooth ReLU activation function. This work also provides the tightest bound so far of the estimation error with respect to the output noise.

Original languageEnglish (US)
Article number9152147
Pages (from-to)2622-2635
Number of pages14
JournalIEEE Transactions on Neural Networks and Learning Systems
Volume32
Issue number6
DOIs
StatePublished - Jun 2021
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence

Keywords

  • Accelerated gradient descent (GD)
  • convolutional neural networks
  • generalizability
  • global optimality
  • linear convergence

Fingerprint

Dive into the research topics of 'Improved Linear Convergence of Training CNNs with Generalizability Guarantees: A One-Hidden-Layer Case'. Together they form a unique fingerprint.

Cite this