We present an approach based on feed-forward neural networks for learning the
distribution of textual documents. This approach is inspired by the Neural
Autoregressive Distribution Estimator(NADE) model, which has been shown to be a
good estimator of the distribution of discrete-valued igh-dimensional vectors.
In this paper, we present how NADE can successfully be adapted to the case of
textual data, retaining from NADE the property that sampling or computing the
probability of observations can be done exactly and efficiently. The approach
can also be used to learn deep representations of documents that are
competitive to those learned by the alternative topic modeling approaches.
Finally, we describe how the approach can be combined with a regular neural
network N-gram model and substantially improve its performance, by making its
learned representation sensitive to the larger, document-specific context.
Description
Document Neural Autoregressive Distribution Estimation
%0 Generic
%1 lauly2016document
%A Lauly, Stanislas
%A Zheng, Yin
%A Allauzen, Alexandre
%A Larochelle, Hugo
%D 2016
%K LM NN TM discrete
%T Document Neural Autoregressive Distribution Estimation
%U http://arxiv.org/abs/1603.05962
%X We present an approach based on feed-forward neural networks for learning the
distribution of textual documents. This approach is inspired by the Neural
Autoregressive Distribution Estimator(NADE) model, which has been shown to be a
good estimator of the distribution of discrete-valued igh-dimensional vectors.
In this paper, we present how NADE can successfully be adapted to the case of
textual data, retaining from NADE the property that sampling or computing the
probability of observations can be done exactly and efficiently. The approach
can also be used to learn deep representations of documents that are
competitive to those learned by the alternative topic modeling approaches.
Finally, we describe how the approach can be combined with a regular neural
network N-gram model and substantially improve its performance, by making its
learned representation sensitive to the larger, document-specific context.
@misc{lauly2016document,
abstract = {We present an approach based on feed-forward neural networks for learning the
distribution of textual documents. This approach is inspired by the Neural
Autoregressive Distribution Estimator(NADE) model, which has been shown to be a
good estimator of the distribution of discrete-valued igh-dimensional vectors.
In this paper, we present how NADE can successfully be adapted to the case of
textual data, retaining from NADE the property that sampling or computing the
probability of observations can be done exactly and efficiently. The approach
can also be used to learn deep representations of documents that are
competitive to those learned by the alternative topic modeling approaches.
Finally, we describe how the approach can be combined with a regular neural
network N-gram model and substantially improve its performance, by making its
learned representation sensitive to the larger, document-specific context.},
added-at = {2017-10-04T16:33:53.000+0200},
author = {Lauly, Stanislas and Zheng, Yin and Allauzen, Alexandre and Larochelle, Hugo},
biburl = {https://www.bibsonomy.org/bibtex/2529f7fcf93ebea93992d4ba8773d16ee/daschloer},
description = {Document Neural Autoregressive Distribution Estimation},
interhash = {7dba4009edaad06bc04b0840c4428c52},
intrahash = {529f7fcf93ebea93992d4ba8773d16ee},
keywords = {LM NN TM discrete},
note = {cite arxiv:1603.05962},
timestamp = {2017-10-04T16:33:53.000+0200},
title = {Document Neural Autoregressive Distribution Estimation},
url = {http://arxiv.org/abs/1603.05962},
year = 2016
}