Abstract
Leave-one-out cross-validation (LOO) and the widely applicable information
criterion (WAIC) are methods for estimating pointwise out-of-sample prediction
accuracy from a fitted Bayesian model using the log-likelihood evaluated at the
posterior simulations of the parameter values. LOO and WAIC have various
advantages over simpler estimates of predictive error such as AIC and DIC but
are less used in practice because they involve additional computational steps.
Here we lay out fast and stable computations for LOO and WAIC that can be
performed using existing simulation draws. We introduce an efficient
computation of LOO using Pareto-smoothed importance sampling (PSIS), a new
procedure for regularizing importance weights. Although WAIC is asymptotically
equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case
with weak priors or influential observations. As a byproduct of our
calculations, we also obtain approximate standard errors for estimated
predictive errors and for comparing of predictive errors between two models. We
implement the computations in an R package called 'loo' and demonstrate using
models fit with the Bayesian inference package Stan.
Users
Please
log in to take part in the discussion (add own reviews or comments).