Information-Theoretic Bounds on Transfer Generalization Gap Based on Jensen-Shannon Divergence

Sharu Theresa Jose, Osvaldo Simeone

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In transfer learning, training and testing data sets are drawn from different data distributions. The transfer generalization gap is the difference between the population loss on the target data distribution and the training loss. The training data set generally includes data drawn from both source and target distributions. This work presents novel information-theoretic upper bounds on the average transfer generalization gap that capture (i) the domain shift between the target data distribution PZ0 and the source distribution PZ through a two-parameter family of generalized (α1, α2)-Jensen-Shannon (JS) divergences; and (ii) the sensitivity of the transfer learner output W to each individual sample of the data set Zi via the mutual information I(W; Zi). For α1 ∈ (0, 1), the (α1, α2)JS divergence can be bounded even when the support of PZ is not included in that of PZ0. This contrasts the Kullback-Leibler (KL) divergence DKL(PZ||PZ0)-based bounds of Wu et al. [1], which are vacuous under this assumption. Moreover, the obtained bounds hold for unbounded loss functions with bounded cumulant generating functions, unlike the φ-divergence based bound of Wu et al. [1]. We also obtain new upper bounds on the average transfer excess risk in terms of the (α1, α2)-JS divergence for empirical weighted risk minimization (EWRM), which minimizes the weighted average training losses over source and target data sets. Finally, we provide a numerical example to illustrate the merits of the introduced bounds.

Original languageEnglish (US)
Title of host publication29th European Signal Processing Conference, EUSIPCO 2021 - Proceedings
PublisherEuropean Signal Processing Conference, EUSIPCO
Pages1461-1465
Number of pages5
ISBN (Electronic)9789082797060
DOIs
StatePublished - 2021
Externally publishedYes
Event29th European Signal Processing Conference, EUSIPCO 2021 - Dublin, Ireland
Duration: Aug 23 2021Aug 27 2021

Publication series

NameEuropean Signal Processing Conference
Volume2021-August
ISSN (Print)2219-5491

Conference

Conference29th European Signal Processing Conference, EUSIPCO 2021
Country/TerritoryIreland
CityDublin
Period8/23/218/27/21

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Information-Theoretic Bounds on Transfer Generalization Gap Based on Jensen-Shannon Divergence'. Together they form a unique fingerprint.

Cite this