CrowdTesting: A Novel Methodology for Subjective User Studies and QoE Evaluation
Technical Report, 486. Department of Computer Science, (February 2013)Abstract
The term "Crowdtesting" refers to subjective user studies which are
conducted via crowdsourcing. The anonymous test subjects are remotly
conducting the tests in their preferred environment. The advantages of
crowdtesting are reduced time and costs for tests, large and diverse
panel of international users, and realistic user settings. However,
conceptual and technical challenges emerge due to the remote test
settings. Key issues arising from crowdtesting include the reliability
of user (ratings), incentives and payment schemes, and unknown context
of user tests (e.g. environment, hardware, illumination, etc.).
Countermeasures have to be developed, included in the test design, and
implemented in the actual test campaign for which special attention is
drawn to reliablity in this article. In particular, we focus on QoE
assessment by means of crowdtesting. Furthermore, the test results have
to be analyzed with respect to these features. Best practices collected
from a large set of ownly conducted crowdtesting studies are presented
in this paper in which we use the example of QoE testing for video
streaming. The statistical analysis of experimental results guides how
to analyze crowdtesting results and shows that the recommended
two-stage crowdtesting design leads to reliable results.
Links and resources
Tags
community
@trcsuniwue's tags highlighted