Misc,
Recurrent Independent Mechanisms
(2019)cite arxiv:1909.10893.
Abstract
Learning modular structures which reflect the dynamics of the environment can
lead to better generalization and robustness to changes which only affect a few
of the underlying causes. We propose Recurrent Independent Mechanisms (RIMs), a
new recurrent architecture in which multiple groups of recurrent cells operate
with nearly independent transition dynamics, communicate only sparingly through
the bottleneck of attention, and are only updated at time steps where they are
most relevant. We show that this leads to specialization amongst the RIMs,
which in turn allows for dramatically improved generalization on tasks where
some factors of variation differ systematically between training and
evaluation.
