MACE (Multi-Annotator Competence Estimation) is an implementation of an item-response model that let's you evaluate redundant annotations of categorical data. It provides competence estimates of the individual annotators and the most likely answer to each item.
If we have 10 annotators answer a question, and five answer with 'yes' and five with 'no' (a surprisingly frequent event), we would normaly have to flip a coin to decide what the right answer is. If we knew, however, that one of the people who answered 'yes' is an expert on the question, while one of the others just alwas selects 'no', we would take this information into account to weight their answers. MACE does exactly that. It tries to find out which annotators are more trustworthy and upweighs their answers. All you need to provide is a CSV file with one item per line.
In tests, MACE's trust estimates correlated highly wth the annotators' true competence, and it achieved accuracies of over 0.9 on several test sets. MACE can take annotated items into account, if they are available. This helps to guide the training and improves accuracy.
Bilddateien so bearbeiten, dass künstliche Intelligenz sie auswerten kann - das ist eine der Aufgaben von sogenannten Klick-Arbeitern. Die meisten von ihnen stammen einer Studie zufolge aus dem krisengeplagten Venezuela.
R. Snow, B. O'Connor, D. Jurafsky, und A. Ng. EMNLP '08: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Seite 254--263. Morristown, NJ, USA, Association for Computational Linguistics, (2008)
M. Sabou, K. Bontcheva, L. Derczynski, und A. Scharl. Proceedings of the Ninth International Conference on Language Resources and Evaluation, ŁREC\ 2014, Reykjavik, Iceland, May 26-31, 2014, Seite 859--866. European Language Resources Association \(ELRA)\, (2014)
M. Becker, K. Borchert, M. Hirth, H. Mewes, A. Hotho, und P. Tran-Gia. Proceedings of the 15th International Conference on Knowledge Technologies and Data-driven Business, Seite 10:1--10:8. New York, NY, USA, ACM, (Oktober 2015)
A. Mladenow, C. Bauer, C. Strauss, und M. Gregus. 7th International Conference on Intelligent Networking and Collaborative Systems (INCoS 2015), Seite 1-6. IEEE, (2015)
A. Mladenow, C. Bauer, und C. Strauss. Blurring the Boundaries Through Digital Innovation: Individual, Organizational, and Societal Challenges, Volume 19 von Lecture Notes in Information Systems and Organisations (LNISO), Springer International Publishing Switzerland, Cham, Switzerland, (2016)
A. Mladenow, C. Bauer, und C. Strauss. 17th International Conference on Information Integration and Web-based Applications & Services (iiWAS2015), Seite 244-251. New York, NY, USA, ACM, (Dezember 2015)
A. Mladenow, C. Bauer, und C. Strauss. 12th International Conference on Cooperative Design, Visualization & Engineering (CDVE 2015), Volume 9320 von Lecture Notes in Computer Science (LNCS), Springer, (September 2015)
C. Bauer, A. Mladenow, und C. Strauss. 11th International Conference on Cooperative Design, Visualization & Engineering (CDVE 2014), Volume 9320 von Lecture Notes in Computer Science (LNCS), Springer, Seattle, WA, (September 2014)
A. Zhang, J. Chen, W. Chai, J. Xu, L. Hong, und E. CHI. ACM Trans. Interact. Intell. Syst., 8 (2):
14:1--14:28(Juni 2018)Interesting attempt to engineer a multi-step process for app search cluster evaluation using crowdsourcing. In some cases crowd performed better than experts..
M. Rokicki, C. Trattner, und E. Herder. Proceedings of the Twelfth International Conference on Web and Social Media, ICWSM 2018, Stanford, California, USA, June 25-28, 2018., Seite 310--319. (2018)
C. Hube, B. Fetahu, und U. Gadiraju. Proceedings of the 37th Annual ACM Conference on Human Factors in Computing Systems, CHI 2019, Glasgow, United Kingdom May 4-9, 2019, (2019)