Mechanical Turk

For our studies in color naming we have used two kind of experiments to elicit color terms: traditional tightly controlled psychophysics experiments in our laboratory, and crowdsourcing. There is a third kind of experiments, which does not rely neither on our acquaintances nor on our fans on the Internet: paid subjects.

The concern with paid subjects is that they are self-selected people willing to sell their time at a rather low price, so we assume they are not the best representatives of the population at large. However, the same applies for psychophysics experiments, where the subjects tend to have a Ph.D. and to live a sophisticated life.

A cheap and convenient way to recruit paid subjects is Amazon's Mechanical Turk, which has been used by others for research in color naming and color palette design. John Bohannon recently wrote a short review on MTurk for social science studies in Science (link).

Turkers make only $1.80 to $6.00 an hour, so quite a few do not understand the task or are spammers who skim through the jobs and and give random responses wherever possible to accelerate the process. Gabriel Lenz, a political scientist at the University of California, Berkeley had to reject about 20% of his American and 50% of his Indian Turkers for those reasons.

Gabriel Lenz and Adam Berinsky, a political scientist at the Massachusetts Institute of Technology in Cambridge have teamed up with Gregory Huber, a political scientist at Yale University, to study the Turker population. And of course, they are using MTurk to do so. They recently replicated two classic survey experiments and a political science experiment. In each case, the data obtained with MTurk were consistent with published studies that tested people in laboratories.