Inproceedings,
TurKit: human computation algorithms on mechanical turk
Proceedings of the 23nd annual ACM symposium on User interface software and technology, page 57--66. New York, NY, USA, ACM, (2010)
DOI: 10.1145/1866029.1866040
Abstract
Mechanical Turk (<i>MTurk</i>) provides an on-demand source of human computation. This provides a tremendous opportunity to explore algorithms which incorporate human computation as a function call. However, various systems challenges make this difficult in practice, and most uses of MTurk post large numbers of independent tasks. TurKit is a toolkit for prototyping and exploring algorithmic human computation, while maintaining a straight-forward imperative programming style. We present the crash-and-rerun programming model that makes TurKit possible, along with a variety of applications for human computation algorithms. We also present case studies of TurKit used for real experiments across different fields.
Tags
Users
Comments and Reviewsshow / hide
- @jaeschke 12 years agoThe paper presents a system - including a complete IDE (!) - for implementing and debugging human computation programs. The "TurKit" language consists of the basic predicates "once", "fork", and "join". "once" allows the programmer to state that a computation shall be performed only once to save time and money during debugging the program. "fork" and "join" are for parallel execution of tasks. The core principle of "TurKit" is the "crash and rerun" idea: all processing steps are recorded, including intermediate data and in particular answers from the human workers. This allows the programmer to re-run the program as often as necessary without additional costs and to debug the code line-by-line. The model is relatively simple yet it allows to easily describe complex tasks. For example, the paper presents code snippets and results from an attempt to write and iteratively improve a text describing an image or to transcribe and improve blurred text.