Misc,
Hypersolvers: Toward Fast Continuous-Depth Models
(2020)cite arxiv:2007.09601.
Abstract
The infinite-depth paradigm pioneered by Neural ODEs has launched a
renaissance in the search for novel dynamical system-inspired deep learning
primitives; however, their utilization in problems of non-trivial size has
often proved impossible due to poor computational scalability. This work paves
the way for scalable Neural ODEs with time-to-prediction comparable to
traditional discrete networks. We introduce hypersolvers, neural networks
designed to solve ODEs with low overhead and theoretical guarantees on
accuracy. The synergistic combination of hypersolvers and Neural ODEs allows
for cheap inference and unlocks a new frontier for practical application of
continuous-depth models. Experimental evaluations on standard benchmarks, such
as sampling for continuous normalizing flows, reveal consistent pareto
efficiency over classical numerical methods.
