On optimization methods for deep learning.
ICML, page 265-272. Omnipress, (2011)Abstract
The predominant methodology in training
deep learning advocates the use of stochastic
gradient descent methods (SGDs). Despite
its ease of implementation, SGDs are diffi-
cult to tune and parallelize. These problems
make it challenging to develop, debug and
scale up deep learning algorithms with SGDs.
In this paper, we show that more sophisti-
cated off-the-shelf optimization methods such
as Limited memory BFGS (L-BFGS) and
Conjugate gradient (CG) with line search
can significantly simplify and speed up the
process of pretraining deep algorithms. In
our experiments, the difference between L-
BFGS/CG and SGDs are more pronounced
if we consider algorithmic extensions (e.g.,
sparsity regularization) and hardware ex-
tensions (e.g., GPUs or computer clusters).
Our experiments with distributed optimiza-
tion support the use of L-BFGS with locally
connected networks and convolutional neural
networks. Using L-BFGS, our convolutional
network model achieves 0.69% on the stan-
dard MNIST dataset. This is a state-of-the-
art result on MNIST among algorithms that
do not use distortions or pretraining.
