A panel for lemons? Positivity bias, reputation systems and data quality on MTurk

The purpose of this paper is to investigate how the effectiveness of systems for ensuring cooperation in online transactions is impacted by a positivity bias in the evaluation of the work that is produced. The presence of this bias can reduce the informativeness of the reputation system and negatively impact its ability to ensure quality.

This research combines survey and experimental methods, collecting data from 1,875 Mechanical Turk (MTurk) workers in five studies designed to investigate the informativeness of the MTurk reputation system.

The findings demonstrate the presence of a positivity bias in evaluations of workers on MTurk, which leaves them undifferentiated, except at the extremity of the reputation system and by status markers.

Because MTurk workers self-select tasks, the findings are limited in that they may only be generalizable to those who are interested in research-related work. Further, the tasks used in this research are largely subjective in nature, which may decrease their sensitivity to differences in quality.

For researchers, the results suggest that requiring 99 per cent approval rates (rather than the previously advised 95 per cent) should be used to identify high-quality workers on MTurk.

The research provides insights into the design and use of reputation systems and demonstrates how design decisions can exacerbate the effect of naturally occurring biases in evaluations to reduce the utility of these systems.