State-Regularized Recurrent Neural Networks
(2019)cite arxiv:1901.08817Comment: to appear at ICML2019, 20 pages.Abstract
Recurrent neural networks are a widely used class of neural architectures.
They have, however, two shortcomings. First, it is difficult to understand what
exactly they learn. Second, they tend to work poorly on sequences requiring
long-term memorization, despite having this capacity in principle. We aim to
address both shortcomings with a class of recurrent networks that use a
stochastic state transition mechanism between cell applications. This
mechanism, which we term state-regularization, makes RNNs transition between a
finite set of learnable states. We evaluate state-regularized RNNs on (1)
regular languages for the purpose of automata extraction; (2) nonregular
languages such as balanced parentheses, palindromes, and the copy task where
external memory is required; and (3) real-word sequence learning tasks for
sentiment analysis, visual object recognition, and language modeling. We show
that state-regularization (a) simplifies the extraction of finite state
automata modeling an RNN's state transition dynamics; (b) forces RNNs to
operate more like automata with external memory and less like finite state
machines; (c) makes RNNs have better interpretability and explainability.
Description
[1901.08817] State-Regularized Recurrent Neural Networks
