Abstract
We describe and develop a close relationship between two problems that have
customarily been regarded as distinct: that of maximizing entropy, and that of
minimizing worst-case expected loss. Using a formulation grounded in the
equilibrium theory of zero-sum games between Decision Maker and
Nature, these two problems are shown to be dual to each other, the solution
to each providing that to the other. Although Topsdescribed this connection
for the Shannon entropy over 20 years ago, it does not appear to be widely
known even in that important special case. We here generalize this theory to
apply to arbitrary decision problems and loss functions. We indicate how an
appropriate generalized definition of entropy can be associated with such a
problem, and we show that, subject to certain regularity conditions, the
above-mentioned duality continues to apply in this extended context.
This simultaneously provides a possible rationale for maximizing entropy and
a tool for finding robust Bayes acts. We also describe the essential identity
between the problem of maximizing entropy and that of minimizing a related
discrepancy or divergence between distributions. This leads to an extension, to
arbitrary discrepancies, of a well-known minimax theorem for the case of
Kullback-Leibler divergence (the ``redundancy-capacity theorem'' of information
theory). For the important case of families of distributions having certain
mean values specified, we develop simple sufficient conditions and methods for
identifying the desired solutions.
Users
Please
log in to take part in the discussion (add own reviews or comments).