HOW DOES UNLABELED DATA IMPROVE GENERALIZATION IN SELF-TRAINING? A ONE-HIDDEN-LAYER THEORETICAL ANALYSIS

Shuai Zhang, Meng Wang, Sijia Liu, Pin Yu Chen, Jinjun Xiong

Research output: Contribution to conferencePaperpeer-review

8 Scopus citations

Abstract

Self-training, a semi-supervised learning algorithm, leverages a large amount of unlabeled data to improve learning when the labeled data are limited. Despite empirical successes, its theoretical characterization remains elusive. To the best of our knowledge, this work establishes the first theoretical analysis for the known iterative self-training paradigm and proves the benefits of unlabeled data in both training convergence and generalization ability. To make our theoretical analysis feasible, we focus on the case of one-hidden-layer neural networks. However, theoretical understanding of iterative self-training is non-trivial even for a shallow neural network. One of the key challenges is that existing neural network landscape analysis built upon supervised learning no longer holds in the (semi-supervised) self-training paradigm. We address this challenge and prove that iterative self-training converges linearly with both convergence rate and generalization accuracy improved in the order of 1/M, where M is the number of unlabeled samples. Experiments from shallow neural networks to deep neural networks are also provided to justify the correctness of our established theoretical insights on self-training.

Original languageEnglish (US)
StatePublished - 2022
Externally publishedYes
Event10th International Conference on Learning Representations, ICLR 2022 - Virtual, Online
Duration: Apr 25 2022Apr 29 2022

Conference

Conference10th International Conference on Learning Representations, ICLR 2022
CityVirtual, Online
Period4/25/224/29/22

All Science Journal Classification (ASJC) codes

  • Language and Linguistics
  • Computer Science Applications
  • Education
  • Linguistics and Language

Fingerprint

Dive into the research topics of 'HOW DOES UNLABELED DATA IMPROVE GENERALIZATION IN SELF-TRAINING? A ONE-HIDDEN-LAYER THEORETICAL ANALYSIS'. Together they form a unique fingerprint.

Cite this