Conference,
A PAC-Bayesian Tutorial with A Dropout Bound
(2013)cite arxiv:1307.2118.
Abstract
This tutorial gives a concise overview of existing PAC-Bayesian theory
focusing on three generalization bounds. The first is an Occam bound which
handles rules with finite precision parameters and which states that
generalization loss is near training loss when the number of bits needed to
write the rule is small compared to the sample size. The second is a
PAC-Bayesian bound providing a generalization guarantee for posterior
distributions rather than for individual rules. The PAC-Bayesian bound
naturally handles infinite precision rule parameters, $L_2$ regularization,
provides a bound for dropout training, and defines a natural notion of a
single distinguished PAC-Bayesian posterior distribution. The third bound is a
training-variance bound --- a kind of bias-variance analysis but with bias
replaced by expected training loss. The training-variance bound dominates the
other bounds but is more difficult to interpret. It seems to suggest variance
reduction methods such as bagging and may ultimately provide a more meaningful
analysis of dropouts.
