Most stochastic optimization methods use gradients once before discarding
them. While variance reduction methods have shown that reusing past gradients
can be beneficial when there is a finite number of datapoints, they do not
easily extend to the online setting. One issue is the staleness due to using
past gradients. We propose to correct this staleness using the idea of implicit
gradient transport (IGT) which transforms gradients computed at previous
iterates into gradients evaluated at the current iterate without using the
Hessian explicitly. In addition to reducing the variance and bias of our
updates over time, IGT can be used as a drop-in replacement for the gradient
estimate in a number of well-understood methods such as heavy ball or Adam. We
show experimentally that it achieves state-of-the-art results on a wide range
of architectures and benchmarks. Additionally, the IGT gradient estimator
yields the optimal asymptotic convergence rate for online stochastic
optimization in the restricted setting where the Hessians of all component
functions are equal.
Description
[1906.03532] Reducing the variance in online optimization by transporting past gradients
%0 Journal Article
%1 arnold2019reducing
%A Arnold, Sébastien M. R.
%A Manzagol, Pierre-Antoine
%A Babanezhad, Reza
%A Mitliagkas, Ioannis
%A Roux, Nicolas Le
%D 2019
%K memory optimization robustness sparsity stable
%T Reducing the variance in online optimization by transporting past
gradients
%U http://arxiv.org/abs/1906.03532
%X Most stochastic optimization methods use gradients once before discarding
them. While variance reduction methods have shown that reusing past gradients
can be beneficial when there is a finite number of datapoints, they do not
easily extend to the online setting. One issue is the staleness due to using
past gradients. We propose to correct this staleness using the idea of implicit
gradient transport (IGT) which transforms gradients computed at previous
iterates into gradients evaluated at the current iterate without using the
Hessian explicitly. In addition to reducing the variance and bias of our
updates over time, IGT can be used as a drop-in replacement for the gradient
estimate in a number of well-understood methods such as heavy ball or Adam. We
show experimentally that it achieves state-of-the-art results on a wide range
of architectures and benchmarks. Additionally, the IGT gradient estimator
yields the optimal asymptotic convergence rate for online stochastic
optimization in the restricted setting where the Hessians of all component
functions are equal.
@article{arnold2019reducing,
abstract = {Most stochastic optimization methods use gradients once before discarding
them. While variance reduction methods have shown that reusing past gradients
can be beneficial when there is a finite number of datapoints, they do not
easily extend to the online setting. One issue is the staleness due to using
past gradients. We propose to correct this staleness using the idea of implicit
gradient transport (IGT) which transforms gradients computed at previous
iterates into gradients evaluated at the current iterate without using the
Hessian explicitly. In addition to reducing the variance and bias of our
updates over time, IGT can be used as a drop-in replacement for the gradient
estimate in a number of well-understood methods such as heavy ball or Adam. We
show experimentally that it achieves state-of-the-art results on a wide range
of architectures and benchmarks. Additionally, the IGT gradient estimator
yields the optimal asymptotic convergence rate for online stochastic
optimization in the restricted setting where the Hessians of all component
functions are equal.},
added-at = {2019-06-12T23:37:53.000+0200},
author = {Arnold, Sébastien M. R. and Manzagol, Pierre-Antoine and Babanezhad, Reza and Mitliagkas, Ioannis and Roux, Nicolas Le},
biburl = {https://www.bibsonomy.org/bibtex/2960fe0872f46d77c86a2c5699579b551/kirk86},
description = {[1906.03532] Reducing the variance in online optimization by transporting past gradients},
interhash = {be8ac6d7403b90d508ca881f12ec5fc4},
intrahash = {960fe0872f46d77c86a2c5699579b551},
keywords = {memory optimization robustness sparsity stable},
note = {cite arxiv:1906.03532Comment: Open-source implementation available at: https://github.com/seba-1511/igt.pth},
timestamp = {2019-06-12T23:37:53.000+0200},
title = {Reducing the variance in online optimization by transporting past
gradients},
url = {http://arxiv.org/abs/1906.03532},
year = 2019
}