Article,
On Universal Approximation by Neural Networks with Uniform Guarantees on Approximation of Infinite Dimensional Maps
(2019)cite arxiv:1910.01545Comment: 12 pages.
Abstract
The study of universal approximation of arbitrary functions $f: X
Y$ by neural networks has a rich and thorough history dating back
to Kolmogorov (1957). In the case of learning finite dimensional maps, many
authors have shown various forms of the universality of both fixed depth and
fixed width neural networks. However, in many cases, these classical results
fail to extend to the recent use of approximations of neural networks with
infinitely many units for functional data analysis, dynamical systems
identification, and other applications where either $X$ or
$Y$ become infinite dimensional. Two questions naturally arise: which
infinite dimensional analogues of neural networks are sufficient to approximate
any map $f: X Y$, and when do the finite approximations
to these analogues used in practice approximate $f$ uniformly over its infinite
dimensional domain $X$?
In this paper, we answer the open question of universal approximation of
nonlinear operators when $X$ and $Y$ are both infinite
dimensional. We show that for a large class of different infinite analogues of
neural networks, any continuous map can be approximated arbitrarily closely
with some mild topological conditions on $X$. Additionally, we
provide the first lower-bound on the minimal number of input and output units
required by a finite approximation to an infinite neural network to guarantee
that it can uniformly approximate any nonlinear operator using samples from its
inputs and outputs.
