The Magic Barrier Revisited: Accessing Natural Limitations of Recommender Assessment
K. Jasberg, and S. Sizov. Proceedings of the Eleventh ACM Conference on Recommender Systems, page 56--64. New York, NY, USA, ACM, (2017)
DOI: 10.1145/3109859.3109898
Abstract
Recommender systems nowadays have many applications and are of great economic benefit. Hence, it is imperative for success-oriented companies to compare various of such systems and select the better one for their purposes. To this end, various metrics of predictive accuracy are commonly used, such as the Root Mean Square Error (RMSE), or precision and recall. All these metrics more or less measure how well a recommender system can predict human behaviour. Unfortunately, human behaviour is always associated with some degree of uncertainty, making the evaluation difficult, since it is not clear whether a deviation is system-induced or just originates from the natural variability of human decision making. At this point, some authors speculated that we may be reaching some Magic Barrier where this variability prevents us from getting much more accurate 12, 13, 24. In this article, we will extend the existing theory of the Magic Barrier 24 into a new probabilistic but a yet pragmatic model. In particular, we will use methods from metrology and physics to develop easy-to-handle quantities for computation to describe the Magic Barrier for different accuracy metrics and provide suggestions for common application. This discussion is substantiated by comprehensive experiments with real users and large-scale simulations on a high-performance cluster.
%0 Conference Paper
%1 citeulike:14420484
%A Jasberg, Kevin
%A Sizov, Sergej
%B Proceedings of the Eleventh ACM Conference on Recommender Systems
%C New York, NY, USA
%D 2017
%I ACM
%K noise rating recommender
%P 56--64
%R 10.1145/3109859.3109898
%T The Magic Barrier Revisited: Accessing Natural Limitations of Recommender Assessment
%U http://dx.doi.org/10.1145/3109859.3109898
%X Recommender systems nowadays have many applications and are of great economic benefit. Hence, it is imperative for success-oriented companies to compare various of such systems and select the better one for their purposes. To this end, various metrics of predictive accuracy are commonly used, such as the Root Mean Square Error (RMSE), or precision and recall. All these metrics more or less measure how well a recommender system can predict human behaviour. Unfortunately, human behaviour is always associated with some degree of uncertainty, making the evaluation difficult, since it is not clear whether a deviation is system-induced or just originates from the natural variability of human decision making. At this point, some authors speculated that we may be reaching some Magic Barrier where this variability prevents us from getting much more accurate 12, 13, 24. In this article, we will extend the existing theory of the Magic Barrier 24 into a new probabilistic but a yet pragmatic model. In particular, we will use methods from metrology and physics to develop easy-to-handle quantities for computation to describe the Magic Barrier for different accuracy metrics and provide suggestions for common application. This discussion is substantiated by comprehensive experiments with real users and large-scale simulations on a high-performance cluster.
%@ 978-1-4503-4652-8
@inproceedings{citeulike:14420484,
abstract = {{Recommender systems nowadays have many applications and are of great economic benefit. Hence, it is imperative for success-oriented companies to compare various of such systems and select the better one for their purposes. To this end, various metrics of predictive accuracy are commonly used, such as the Root Mean Square Error (RMSE), or precision and recall. All these metrics more or less measure how well a recommender system can predict human behaviour. Unfortunately, human behaviour is always associated with some degree of uncertainty, making the evaluation difficult, since it is not clear whether a deviation is system-induced or just originates from the natural variability of human decision making. At this point, some authors speculated that we may be reaching some Magic Barrier where this variability prevents us from getting much more accurate [12, 13, 24]. In this article, we will extend the existing theory of the Magic Barrier [24] into a new probabilistic but a yet pragmatic model. In particular, we will use methods from metrology and physics to develop easy-to-handle quantities for computation to describe the Magic Barrier for different accuracy metrics and provide suggestions for common application. This discussion is substantiated by comprehensive experiments with real users and large-scale simulations on a high-performance cluster.}},
added-at = {2018-03-19T12:24:51.000+0100},
address = {New York, NY, USA},
author = {Jasberg, Kevin and Sizov, Sergej},
biburl = {https://www.bibsonomy.org/bibtex/26d37e785e072eb27230870d08f77ee91/aho},
booktitle = {Proceedings of the Eleventh ACM Conference on Recommender Systems},
citeulike-article-id = {14420484},
citeulike-linkout-0 = {http://portal.acm.org/citation.cfm?id=3109898},
citeulike-linkout-1 = {http://dx.doi.org/10.1145/3109859.3109898},
doi = {10.1145/3109859.3109898},
interhash = {382fd98b8c41e5acea288268dec062df},
intrahash = {6d37e785e072eb27230870d08f77ee91},
isbn = {978-1-4503-4652-8},
keywords = {noise rating recommender},
location = {Como, Italy},
pages = {56--64},
posted-at = {2017-08-28 10:00:17},
priority = {3},
publisher = {ACM},
series = {RecSys '17},
timestamp = {2018-03-19T12:24:51.000+0100},
title = {{The Magic Barrier Revisited: Accessing Natural Limitations of Recommender Assessment}},
url = {http://dx.doi.org/10.1145/3109859.3109898},
year = 2017
}