Relative Natural Gradient for Learning Large Complex Models
K. Sun, and F. Nielsen. (2016)cite arxiv:1606.06069Comment: 24 pages, 5 figures.
Abstract
Fisher information and natural gradient provided deep insights and powerful
tools to artificial neural networks. However related analysis becomes more and
more difficult as the learner's structure turns large and complex. This paper
makes a preliminary step towards a new direction. We extract a local component
of a large neuron system, and defines its relative Fisher information metric
that describes accurately this small component, and is invariant to the other
parts of the system. This concept is important because the geometry structure
is much simplified and it can be easily applied to guide the learning of neural
networks. We provide an analysis on a list of commonly used components, and
demonstrate how to use this concept to further improve optimization.
Description
[1606.06069] Relative Natural Gradient for Learning Large Complex Models
%0 Journal Article
%1 sun2016relative
%A Sun, Ke
%A Nielsen, Frank
%D 2016
%K bayesian geometry information optimization
%T Relative Natural Gradient for Learning Large Complex Models
%U http://arxiv.org/abs/1606.06069
%X Fisher information and natural gradient provided deep insights and powerful
tools to artificial neural networks. However related analysis becomes more and
more difficult as the learner's structure turns large and complex. This paper
makes a preliminary step towards a new direction. We extract a local component
of a large neuron system, and defines its relative Fisher information metric
that describes accurately this small component, and is invariant to the other
parts of the system. This concept is important because the geometry structure
is much simplified and it can be easily applied to guide the learning of neural
networks. We provide an analysis on a list of commonly used components, and
demonstrate how to use this concept to further improve optimization.
@article{sun2016relative,
abstract = {Fisher information and natural gradient provided deep insights and powerful
tools to artificial neural networks. However related analysis becomes more and
more difficult as the learner's structure turns large and complex. This paper
makes a preliminary step towards a new direction. We extract a local component
of a large neuron system, and defines its relative Fisher information metric
that describes accurately this small component, and is invariant to the other
parts of the system. This concept is important because the geometry structure
is much simplified and it can be easily applied to guide the learning of neural
networks. We provide an analysis on a list of commonly used components, and
demonstrate how to use this concept to further improve optimization.},
added-at = {2019-12-11T14:36:42.000+0100},
author = {Sun, Ke and Nielsen, Frank},
biburl = {https://www.bibsonomy.org/bibtex/2add3fc3f42843d8a9bca0351f39c66a7/kirk86},
description = {[1606.06069] Relative Natural Gradient for Learning Large Complex Models},
interhash = {eed874eec71da0cee701d373cb1284fc},
intrahash = {add3fc3f42843d8a9bca0351f39c66a7},
keywords = {bayesian geometry information optimization},
note = {cite arxiv:1606.06069Comment: 24 pages, 5 figures},
timestamp = {2019-12-11T14:37:25.000+0100},
title = {Relative Natural Gradient for Learning Large Complex Models},
url = {http://arxiv.org/abs/1606.06069},
year = 2016
}