New insights and perspectives on the natural gradient method.
J. Martens. (2014)cite arxiv:1412.1193Comment: Many small revisions/corrections throughout , Added a section on 2nd-order methods and future work.
Abstract
Natural gradient descent is an optimization method traditionally motivated
from the perspective of information geometry, and works well for many
applications as an alternative to stochastic gradient descent. In this paper we
critically analyze this method and its properties, and show how it can be
viewed as a type of approximate 2nd-order optimization method, where the Fisher
information matrix can be viewed as an approximation of the Hessian. This
perspective turns out to have significant implications for how to design a
practical and robust version of the method. Additionally, we make the following
contributions to the understanding of natural gradient and 2nd-order methods: a
thorough analysis of the convergence speed of stochastic natural gradient
descent (and more general stochastic 2nd-order methods) as applied to convex
quadratics, a critical examination of the oft-used "empirical" approximation of
the Fisher matrix, and an analysis of the (approximate) parameterization
invariance property possessed by natural gradient methods, which we show still
holds for certain choices of the curvature matrix other than the Fisher, but
notably not the Hessian.
Description
New insights and perspectives on the natural gradient method
%0 Generic
%1 martens2014insights
%A Martens, James
%D 2014
%K optimization
%T New insights and perspectives on the natural gradient method.
%U http://arxiv.org/abs/1412.1193
%X Natural gradient descent is an optimization method traditionally motivated
from the perspective of information geometry, and works well for many
applications as an alternative to stochastic gradient descent. In this paper we
critically analyze this method and its properties, and show how it can be
viewed as a type of approximate 2nd-order optimization method, where the Fisher
information matrix can be viewed as an approximation of the Hessian. This
perspective turns out to have significant implications for how to design a
practical and robust version of the method. Additionally, we make the following
contributions to the understanding of natural gradient and 2nd-order methods: a
thorough analysis of the convergence speed of stochastic natural gradient
descent (and more general stochastic 2nd-order methods) as applied to convex
quadratics, a critical examination of the oft-used "empirical" approximation of
the Fisher matrix, and an analysis of the (approximate) parameterization
invariance property possessed by natural gradient methods, which we show still
holds for certain choices of the curvature matrix other than the Fisher, but
notably not the Hessian.
@misc{martens2014insights,
abstract = {Natural gradient descent is an optimization method traditionally motivated
from the perspective of information geometry, and works well for many
applications as an alternative to stochastic gradient descent. In this paper we
critically analyze this method and its properties, and show how it can be
viewed as a type of approximate 2nd-order optimization method, where the Fisher
information matrix can be viewed as an approximation of the Hessian. This
perspective turns out to have significant implications for how to design a
practical and robust version of the method. Additionally, we make the following
contributions to the understanding of natural gradient and 2nd-order methods: a
thorough analysis of the convergence speed of stochastic natural gradient
descent (and more general stochastic 2nd-order methods) as applied to convex
quadratics, a critical examination of the oft-used "empirical" approximation of
the Fisher matrix, and an analysis of the (approximate) parameterization
invariance property possessed by natural gradient methods, which we show still
holds for certain choices of the curvature matrix other than the Fisher, but
notably not the Hessian.},
added-at = {2018-12-07T10:36:58.000+0100},
author = {Martens, James},
biburl = {https://www.bibsonomy.org/bibtex/2d6938cc850eb32009e99a4449315c8b4/jpvaldes},
description = {New insights and perspectives on the natural gradient method},
interhash = {5aa53b4d22911c7c08d404fe37a4b2b1},
intrahash = {d6938cc850eb32009e99a4449315c8b4},
keywords = {optimization},
note = {cite arxiv:1412.1193Comment: Many small revisions/corrections throughout , Added a section on 2nd-order methods and future work},
timestamp = {2018-12-07T10:36:58.000+0100},
title = {New insights and perspectives on the natural gradient method.},
url = {http://arxiv.org/abs/1412.1193},
year = 2014
}