Robust PAC<inline-formula> <tex-math notation="LaTeX">$^m$</tex-math> </inline-formula>: Training Ensemble Models Under Misspecification and Outliers

Matteo Zecchin; Sangwoo Park; Osvaldo Simeone; Marios Kountouris; David Gesbert

doi:10.1109/TNNLS.2023.3295168

Robust PAC<inline-formula> <tex-math notation="LaTeX">$^m$</tex-math> </inline-formula> Training Ensemble Models Under Misspecification and Outliers

Matteo Zecchin, Sangwoo Park, Osvaldo Simeone, Marios Kountouris, David Gesbert

Electrical and Computer Engineering

Research output: Contribution to journal › Article › peer-review

2 Scopus citations

Abstract

Standard Bayesian learning is known to have suboptimal generalization capabilities under misspecification and in the presence of outliers. Probably approximately correct (PAC)-Bayes theory demonstrates that the free energy criterion minimized by Bayesian learning is a bound on the generalization error for Gibbs predictors (i.e., for single models drawn at random from the posterior) under the assumption of sampling distributions uncontaminated by outliers. This viewpoint provides a justification for the limitations of Bayesian learning when the model is misspecified, requiring ensembling, and when data are affected by outliers. In recent work, PAC-Bayes bounds—referred to as PAC<inline-formula> <tex-math notation="LaTeX">$^m$</tex-math> </inline-formula>—were derived to introduce free energy metrics that account for the performance of ensemble predictors, obtaining enhanced performance under misspecification. This work presents a novel robust free energy criterion that combines the generalized logarithm score function with PAC<inline-formula> <tex-math notation="LaTeX">$^m$</tex-math> </inline-formula> ensemble bounds. The proposed free energy training criterion produces predictive distributions that are able to concurrently counteract the detrimental effects of misspecification—with respect to both likelihood and prior distribution—and outliers.

Original language	English (US)
Pages (from-to)	1-15
Number of pages	15
Journal	IEEE Transactions on Neural Networks and Learning Systems
DOIs	https://doi.org/10.1109/TNNLS.2023.3295168
State	Accepted/In press - 2023

All Science Journal Classification (ASJC) codes

Software
Computer Science Applications
Computer Networks and Communications
Artificial Intelligence

Keywords

Bayes methods
Bayesian learning
Europe
Pollution measurement
Predictive models
Robustness
Standards
Training
ensemble models
machine learning
misspecification
outliers
robustness

Access to Document

10.1109/TNNLS.2023.3295168

Cite this

@article{d180df649e1a4143a1c3668dae398499,

title = "Robust PAC \$\textasciicircum{}m\$ : Training Ensemble Models Under Misspecification and Outliers",

abstract = "Standard Bayesian learning is known to have suboptimal generalization capabilities under misspecification and in the presence of outliers. Probably approximately correct (PAC)-Bayes theory demonstrates that the free energy criterion minimized by Bayesian learning is a bound on the generalization error for Gibbs predictors (i.e., for single models drawn at random from the posterior) under the assumption of sampling distributions uncontaminated by outliers. This viewpoint provides a justification for the limitations of Bayesian learning when the model is misspecified, requiring ensembling, and when data are affected by outliers. In recent work, PAC-Bayes bounds—referred to as PAC \$\textasciicircum{}m\$ —were derived to introduce free energy metrics that account for the performance of ensemble predictors, obtaining enhanced performance under misspecification. This work presents a novel robust free energy criterion that combines the generalized logarithm score function with PAC \$\textasciicircum{}m\$ ensemble bounds. The proposed free energy training criterion produces predictive distributions that are able to concurrently counteract the detrimental effects of misspecification—with respect to both likelihood and prior distribution—and outliers.",

keywords = "Bayes methods, Bayesian learning, Europe, Pollution measurement, Predictive models, Robustness, Standards, Training, ensemble models, machine learning, misspecification, outliers, robustness",

author = "Matteo Zecchin and Sangwoo Park and Osvaldo Simeone and Marios Kountouris and David Gesbert",

note = "Publisher Copyright: IEEE",

year = "2023",

doi = "10.1109/TNNLS.2023.3295168",

language = "English (US)",

pages = "1--15",

journal = "IEEE Transactions on Neural Networks and Learning Systems",

issn = "2162-237X",

publisher = "IEEE Computational Intelligence Society",

}

TY - JOUR

T1 - Robust PAC $^m$

T2 - Training Ensemble Models Under Misspecification and Outliers

AU - Zecchin, Matteo

AU - Park, Sangwoo

AU - Simeone, Osvaldo

AU - Kountouris, Marios

AU - Gesbert, David

N1 - Publisher Copyright: IEEE

PY - 2023

Y1 - 2023

N2 - Standard Bayesian learning is known to have suboptimal generalization capabilities under misspecification and in the presence of outliers. Probably approximately correct (PAC)-Bayes theory demonstrates that the free energy criterion minimized by Bayesian learning is a bound on the generalization error for Gibbs predictors (i.e., for single models drawn at random from the posterior) under the assumption of sampling distributions uncontaminated by outliers. This viewpoint provides a justification for the limitations of Bayesian learning when the model is misspecified, requiring ensembling, and when data are affected by outliers. In recent work, PAC-Bayes bounds—referred to as PAC $^m$ —were derived to introduce free energy metrics that account for the performance of ensemble predictors, obtaining enhanced performance under misspecification. This work presents a novel robust free energy criterion that combines the generalized logarithm score function with PAC $^m$ ensemble bounds. The proposed free energy training criterion produces predictive distributions that are able to concurrently counteract the detrimental effects of misspecification—with respect to both likelihood and prior distribution—and outliers.

AB - Standard Bayesian learning is known to have suboptimal generalization capabilities under misspecification and in the presence of outliers. Probably approximately correct (PAC)-Bayes theory demonstrates that the free energy criterion minimized by Bayesian learning is a bound on the generalization error for Gibbs predictors (i.e., for single models drawn at random from the posterior) under the assumption of sampling distributions uncontaminated by outliers. This viewpoint provides a justification for the limitations of Bayesian learning when the model is misspecified, requiring ensembling, and when data are affected by outliers. In recent work, PAC-Bayes bounds—referred to as PAC $^m$ —were derived to introduce free energy metrics that account for the performance of ensemble predictors, obtaining enhanced performance under misspecification. This work presents a novel robust free energy criterion that combines the generalized logarithm score function with PAC $^m$ ensemble bounds. The proposed free energy training criterion produces predictive distributions that are able to concurrently counteract the detrimental effects of misspecification—with respect to both likelihood and prior distribution—and outliers.

KW - Bayes methods

KW - Bayesian learning

KW - Europe

KW - Pollution measurement

KW - Predictive models

KW - Robustness

KW - Standards

KW - Training

KW - ensemble models

KW - machine learning

KW - misspecification

KW - outliers

KW - robustness

UR - https://www.scopus.com/pages/publications/85165930061

UR - https://www.scopus.com/inward/citedby.url?scp=85165930061&partnerID=8YFLogxK

U2 - 10.1109/TNNLS.2023.3295168

DO - 10.1109/TNNLS.2023.3295168

M3 - Article

AN - SCOPUS:85165930061

SN - 2162-237X

SP - 1

EP - 15

JO - IEEE Transactions on Neural Networks and Learning Systems

JF - IEEE Transactions on Neural Networks and Learning Systems

ER -