Information-theoretic analysis of generalization capability of learning
algorithms
A. Xu, and M. Raginsky. (2017)cite arxiv:1705.07809Comment: Final version, accepted to NIPS 2017.
Abstract
We derive upper bounds on the generalization error of a learning algorithm in
terms of the mutual information between its input and output. The bounds
provide an information-theoretic understanding of generalization in learning
problems, and give theoretical guidelines for striking the right balance
between data fit and generalization by controlling the input-output mutual
information. We propose a number of methods for this purpose, among which are
algorithms that regularize the ERM algorithm with relative entropy or with
random noise. Our work extends and leads to nontrivial improvements on the
recent results of Russo and Zou.
Description
[1705.07809] Information-theoretic analysis of generalization capability of learning algorithms
%0 Journal Article
%1 xu2017informationtheoretic
%A Xu, Aolin
%A Raginsky, Maxim
%D 2017
%K bounds generalization information theory
%T Information-theoretic analysis of generalization capability of learning
algorithms
%U http://arxiv.org/abs/1705.07809
%X We derive upper bounds on the generalization error of a learning algorithm in
terms of the mutual information between its input and output. The bounds
provide an information-theoretic understanding of generalization in learning
problems, and give theoretical guidelines for striking the right balance
between data fit and generalization by controlling the input-output mutual
information. We propose a number of methods for this purpose, among which are
algorithms that regularize the ERM algorithm with relative entropy or with
random noise. Our work extends and leads to nontrivial improvements on the
recent results of Russo and Zou.
@article{xu2017informationtheoretic,
abstract = {We derive upper bounds on the generalization error of a learning algorithm in
terms of the mutual information between its input and output. The bounds
provide an information-theoretic understanding of generalization in learning
problems, and give theoretical guidelines for striking the right balance
between data fit and generalization by controlling the input-output mutual
information. We propose a number of methods for this purpose, among which are
algorithms that regularize the ERM algorithm with relative entropy or with
random noise. Our work extends and leads to nontrivial improvements on the
recent results of Russo and Zou.},
added-at = {2019-09-19T13:38:51.000+0200},
author = {Xu, Aolin and Raginsky, Maxim},
biburl = {https://www.bibsonomy.org/bibtex/2586e90e3ad5f07e0f485cbd56a17d182/kirk86},
description = {[1705.07809] Information-theoretic analysis of generalization capability of learning algorithms},
interhash = {e3c98e9c1ea2e29fd7af1990be99c63f},
intrahash = {586e90e3ad5f07e0f485cbd56a17d182},
keywords = {bounds generalization information theory},
note = {cite arxiv:1705.07809Comment: Final version, accepted to NIPS 2017},
timestamp = {2019-09-19T13:38:51.000+0200},
title = {Information-theoretic analysis of generalization capability of learning
algorithms},
url = {http://arxiv.org/abs/1705.07809},
year = 2017
}