Fleiss' kappa is a statistical measure for assessing the reliability of agreement between a fixed number of raters when assigning categorical ratings to a number of items or classifying items. This contrasts with other kappas such as Cohen's kappa, which only work when assessing the agreement between two raters. The measure calculates the degree of agreement in classification over that which would be expected by chance and is scored as a number between 0 and 1. There is no generally agreed on measure of significance, although guidelines have been given.
J. He, und X. Li. Proceedings of the 9th international semantic web conference on The semantic web - Volume Part II, Seite 339--354. Berlin, Heidelberg, Springer-Verlag, (2010)
G. Hua, M. Zhang, Y. Liu, S. Ma, und L. Ru. Proceedings of the 20th international conference companion on World wide web, Seite 59--60. New York, NY, USA, ACM, (2011)
U. Ozertem, R. Jones, und B. Dumoulin. Proceedings of the 20th international conference on World wide web, Seite 397--406. New York, NY, USA, ACM, (2011)
E. Voorhees. Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, Seite 315--323. New York, NY, USA, ACM, (1998)
F. Guo, C. Liu, und Y. Wang. Proceedings of the Second ACM International Conference on Web Search and Data Mining, Seite 124--131. New York, NY, USA, ACM, (2009)
R. Fagin, R. Kumar, und D. Sivakumar. SODA '03: Proceedings of the fourteenth annual ACM-SIAM symposium on Discrete algorithms, Seite 28--36. Philadelphia, PA, USA, Society for Industrial and Applied Mathematics, (2003)
T. Sakai, und C. Lin. the 3rd International Workshop on Evaluating Information Access (EVIA) --- A Satellite Workshop of NTCIR-8, Seite 25--33. Tokyo, Japan, National Institute of Informatics, (Juni 2010)
J. Kamps, S. Geva, A. Trotman, A. Woodley, und M. Koolen. Advances in Focused Retrieval: 7th International Workshop of the Initiative for the Evaluation of XML Retrieval (INEX 2008), Volume 5631 von LNCS, Seite 1--28. Springer Verlag, Berlin, Heidelberg, (2009)
K. Hofmann, B. Huurnink, M. Bron, und M. de Rijke. Proceeding of the 33rd international ACM SIGIR conference on Research and development in information retrieval, Seite 761--762. New York, NY, USA, ACM, (2010)