The equivalence between Stein variational gradient descent and black-box
variational inference
C. Chu, K. Minami, and K. Fukumizu. (2020)cite arxiv:2004.01822Comment: ICLR 2020, Workshop on Integration of Deep Neural Models and Differential Equations.
Abstract
We formalize an equivalence between two popular methods for Bayesian
inference: Stein variational gradient descent (SVGD) and black-box variational
inference (BBVI). In particular, we show that BBVI corresponds precisely to
SVGD when the kernel is the neural tangent kernel. Furthermore, we interpret
SVGD and BBVI as kernel gradient flows; we do this by leveraging the recent
perspective that views SVGD as a gradient flow in the space of probability
distributions and showing that BBVI naturally motivates a Riemannian structure
on that space. We observe that kernel gradient flow also describes dynamics
found in the training of generative adversarial networks (GANs). This work
thereby unifies several existing techniques in variational inference and
generative modeling and identifies the kernel as a fundamental object governing
the behavior of these algorithms, motivating deeper analysis of its properties.
Description
[2004.01822] The equivalence between Stein variational gradient descent and black-box variational inference
%0 Journal Article
%1 chu2020equivalence
%A Chu, Casey
%A Minami, Kentaro
%A Fukumizu, Kenji
%D 2020
%K bayesian optimization readings variational
%T The equivalence between Stein variational gradient descent and black-box
variational inference
%U http://arxiv.org/abs/2004.01822
%X We formalize an equivalence between two popular methods for Bayesian
inference: Stein variational gradient descent (SVGD) and black-box variational
inference (BBVI). In particular, we show that BBVI corresponds precisely to
SVGD when the kernel is the neural tangent kernel. Furthermore, we interpret
SVGD and BBVI as kernel gradient flows; we do this by leveraging the recent
perspective that views SVGD as a gradient flow in the space of probability
distributions and showing that BBVI naturally motivates a Riemannian structure
on that space. We observe that kernel gradient flow also describes dynamics
found in the training of generative adversarial networks (GANs). This work
thereby unifies several existing techniques in variational inference and
generative modeling and identifies the kernel as a fundamental object governing
the behavior of these algorithms, motivating deeper analysis of its properties.
@article{chu2020equivalence,
abstract = {We formalize an equivalence between two popular methods for Bayesian
inference: Stein variational gradient descent (SVGD) and black-box variational
inference (BBVI). In particular, we show that BBVI corresponds precisely to
SVGD when the kernel is the neural tangent kernel. Furthermore, we interpret
SVGD and BBVI as kernel gradient flows; we do this by leveraging the recent
perspective that views SVGD as a gradient flow in the space of probability
distributions and showing that BBVI naturally motivates a Riemannian structure
on that space. We observe that kernel gradient flow also describes dynamics
found in the training of generative adversarial networks (GANs). This work
thereby unifies several existing techniques in variational inference and
generative modeling and identifies the kernel as a fundamental object governing
the behavior of these algorithms, motivating deeper analysis of its properties.},
added-at = {2020-04-07T12:33:10.000+0200},
author = {Chu, Casey and Minami, Kentaro and Fukumizu, Kenji},
biburl = {https://www.bibsonomy.org/bibtex/24e059dc4bc61836969192bcfbdf8345b/kirk86},
description = {[2004.01822] The equivalence between Stein variational gradient descent and black-box variational inference},
interhash = {c24b50dee82c51ff321bc5034c42538b},
intrahash = {4e059dc4bc61836969192bcfbdf8345b},
keywords = {bayesian optimization readings variational},
note = {cite arxiv:2004.01822Comment: ICLR 2020, Workshop on Integration of Deep Neural Models and Differential Equations},
timestamp = {2020-04-07T12:33:10.000+0200},
title = {The equivalence between Stein variational gradient descent and black-box
variational inference},
url = {http://arxiv.org/abs/2004.01822},
year = 2020
}