Information-Theoretic Bounds on the Generalization Error and Privacy
Leakage in Federated Learning
(2020)cite arxiv:2005.02503Comment: Accepted for publication in Proceedings of 21st IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), 2020. arXiv version is 10pt font, 6 Pages. This is the same document as the SPAWC version, except that the conference version is written with 9pt font to meet the strict page margin requirements.Abstract
Machine learning algorithms operating on mobile networks can be characterized
into three different categories. First is the classical situation in which the
end-user devices send their data to a central server where this data is used to
train a model. Second is the distributed setting in which each device trains
its own model and send its model parameters to a central server where these
model parameters are aggregated to create one final model. Third is the
federated learning setting in which, at any given time $t$, a certain number of
active end users train with their own local data along with feedback provided
by the central server and then send their newly estimated model parameters to
the central server. The server, then, aggregates these new parameters, updates
its own model, and feeds the updated parameters back to all the end users,
continuing this process until it converges.
The main objective of this work is to provide an information-theoretic
framework for all of the aforementioned learning paradigms. Moreover, using the
provided framework, we develop upper and lower bounds on the generalization
error together with bounds on the privacy leakage in the classical, distributed
and federated learning settings.
Keywords: Federated Learning, Distributed Learning, Machine Learning, Model
Aggregation.
Description
[2005.02503] Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learning
