Training deep neural networks with low precision multiplications
M. Courbariaux, Y. Bengio, and J. David. (2014)cite arxiv:1412.7024v5.pdfComment: 10 pages, 5 figures, Accepted as a workshop contribution at ICLR 2015.
Abstract
Multipliers are the most space and power-hungry arithmetic operators of the
digital implementation of deep neural networks. We train a set of
state-of-the-art neural networks (Maxout networks) on three benchmark datasets:
MNIST, CIFAR-10 and SVHN. They are trained with three distinct formats:
floating point, fixed point and dynamic fixed point. For each of those datasets
and for each of those formats, we assess the impact of the precision of the
multiplications on the final error after training. We find that very low
precision is sufficient not just for running trained networks but also for
training them. For example, it is possible to train Maxout networks with 10
bits multiplications.
%0 Generic
%1 courbariaux2014training
%A Courbariaux, Matthieu
%A Bengio, Yoshua
%A David, Jean-Pierre
%D 2014
%K machinelearning
%T Training deep neural networks with low precision multiplications
%U http://arxiv.org/abs/1412.7024
%X Multipliers are the most space and power-hungry arithmetic operators of the
digital implementation of deep neural networks. We train a set of
state-of-the-art neural networks (Maxout networks) on three benchmark datasets:
MNIST, CIFAR-10 and SVHN. They are trained with three distinct formats:
floating point, fixed point and dynamic fixed point. For each of those datasets
and for each of those formats, we assess the impact of the precision of the
multiplications on the final error after training. We find that very low
precision is sufficient not just for running trained networks but also for
training them. For example, it is possible to train Maxout networks with 10
bits multiplications.
@misc{courbariaux2014training,
abstract = {Multipliers are the most space and power-hungry arithmetic operators of the
digital implementation of deep neural networks. We train a set of
state-of-the-art neural networks (Maxout networks) on three benchmark datasets:
MNIST, CIFAR-10 and SVHN. They are trained with three distinct formats:
floating point, fixed point and dynamic fixed point. For each of those datasets
and for each of those formats, we assess the impact of the precision of the
multiplications on the final error after training. We find that very low
precision is sufficient not just for running trained networks but also for
training them. For example, it is possible to train Maxout networks with 10
bits multiplications.},
added-at = {2016-02-24T07:37:27.000+0100},
author = {Courbariaux, Matthieu and Bengio, Yoshua and David, Jean-Pierre},
biburl = {https://www.bibsonomy.org/bibtex/2e2c4a880bfd4b2b84213296b07f57ada/macrobib},
interhash = {df2b6e0db438d5f956584d12b3900bfd},
intrahash = {e2c4a880bfd4b2b84213296b07f57ada},
keywords = {machinelearning},
note = {cite arxiv:1412.7024v5.pdfComment: 10 pages, 5 figures, Accepted as a workshop contribution at ICLR 2015},
timestamp = {2016-02-24T07:37:27.000+0100},
title = {Training deep neural networks with low precision multiplications},
url = {http://arxiv.org/abs/1412.7024},
year = 2014
}