The Kanerva Machine: A Generative Distributed Memory
Y. Wu, G. Wayne, A. Graves, and T. Lillicrap. (2018)cite arxiv:1804.01756Comment: Published as a conference paper at ICLR 2018 (corrected typos in revision).
Abstract
We present an end-to-end trained memory system that quickly adapts to new
data and generates samples like them. Inspired by Kanerva's sparse distributed
memory, it has a robust distributed reading and writing mechanism. The memory
is analytically tractable, which enables optimal on-line compression via a
Bayesian update-rule. We formulate it as a hierarchical conditional generative
model, where memory provides a rich data-dependent prior distribution.
Consequently, the top-down memory and bottom-up perception are combined to
produce the code representing an observation. Empirically, we demonstrate that
the adaptive memory significantly improves generative models trained on both
the Omniglot and CIFAR datasets. Compared with the Differentiable Neural
Computer (DNC) and its variants, our memory model has greater capacity and is
significantly easier to train.
Description
[1804.01756] The Kanerva Machine: A Generative Distributed Memory
%0 Journal Article
%1 wu2018kanerva
%A Wu, Yan
%A Wayne, Greg
%A Graves, Alex
%A Lillicrap, Timothy
%D 2018
%K generative-models memory
%T The Kanerva Machine: A Generative Distributed Memory
%U http://arxiv.org/abs/1804.01756
%X We present an end-to-end trained memory system that quickly adapts to new
data and generates samples like them. Inspired by Kanerva's sparse distributed
memory, it has a robust distributed reading and writing mechanism. The memory
is analytically tractable, which enables optimal on-line compression via a
Bayesian update-rule. We formulate it as a hierarchical conditional generative
model, where memory provides a rich data-dependent prior distribution.
Consequently, the top-down memory and bottom-up perception are combined to
produce the code representing an observation. Empirically, we demonstrate that
the adaptive memory significantly improves generative models trained on both
the Omniglot and CIFAR datasets. Compared with the Differentiable Neural
Computer (DNC) and its variants, our memory model has greater capacity and is
significantly easier to train.
@article{wu2018kanerva,
abstract = {We present an end-to-end trained memory system that quickly adapts to new
data and generates samples like them. Inspired by Kanerva's sparse distributed
memory, it has a robust distributed reading and writing mechanism. The memory
is analytically tractable, which enables optimal on-line compression via a
Bayesian update-rule. We formulate it as a hierarchical conditional generative
model, where memory provides a rich data-dependent prior distribution.
Consequently, the top-down memory and bottom-up perception are combined to
produce the code representing an observation. Empirically, we demonstrate that
the adaptive memory significantly improves generative models trained on both
the Omniglot and CIFAR datasets. Compared with the Differentiable Neural
Computer (DNC) and its variants, our memory model has greater capacity and is
significantly easier to train.},
added-at = {2019-12-26T21:58:40.000+0100},
author = {Wu, Yan and Wayne, Greg and Graves, Alex and Lillicrap, Timothy},
biburl = {https://www.bibsonomy.org/bibtex/26d7912dac636936f8a8d6efe35143d93/kirk86},
description = {[1804.01756] The Kanerva Machine: A Generative Distributed Memory},
interhash = {8040527a6adf656f08133acb727ebfdf},
intrahash = {6d7912dac636936f8a8d6efe35143d93},
keywords = {generative-models memory},
note = {cite arxiv:1804.01756Comment: Published as a conference paper at ICLR 2018 (corrected typos in revision)},
timestamp = {2019-12-26T21:58:40.000+0100},
title = {The Kanerva Machine: A Generative Distributed Memory},
url = {http://arxiv.org/abs/1804.01756},
year = 2018
}