Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
(Nov 8, 2017)Abstract
Gradient-based optimization is the foundation of deep learning and
reinforcement learning. Even when the mechanism being optimized is unknown or
not differentiable, optimization using high-variance or biased gradient
estimates is still often the best strategy. We introduce a general framework
for learning low-variance, unbiased gradient estimators for black-box functions
of random variables. Our method uses gradients of a neural network trained
jointly with model parameters or policies, and is applicable in both discrete
and continuous settings. We demonstrate this framework for training discrete
latent-variable models. We also give an unbiased, action-conditional extension
of the advantage actor-critic reinforcement learning algorithm.
Links and resources
Tags
community
@jpvaldes's tags highlighted