Seminar Details
2017-09-11 (14:00) : Achieving budget-optimality with adaptive schemes in crowdsourcing
At Euler building (room A.002)
Organized by Mathematical Engineering
Speaker :
Sewoong Oh (University of Illinois at Urbana-Champaign)
Abstract :
Crowdsourcing platforms provide marketplaces where task requesters can pay to get labels on their data. Such markets
have emerged recently as popular venues for collecting annotations that are crucial in training machine learning models
in various applications. However, as jobs are tedious and payments are low, errors are common in such crowdsourced labels.
A common strategy to overcome such noise in the answers is to add redundancy by getting multiple answers for each task
and aggregating them using some methods such as majority voting. For such a system, there is a fundamental question of
interest: how can we maximize the accuracy given a fixed budget on how many responses we can collect on the crowdsourcing
system.
We characterize this fundamental trade-off between the budget (how many answers the requester can collect in total) and
the accuracy in the estimated labels. In particular, we ask whether adaptive task assignment schemes lead to a more
efficient trade-off between the accuracy and the budget.
Adaptive schemes, where tasks are assigned adaptively based on the data collected thus far, are widely used in practical
crowdsourcing systems to efficiently use a given fixed budget. However, existing theoretical analyses of crowdsourcing
systems suggest that the gain of adaptive task assignments is minimal. To bridge this gap, we investigate this question
under a strictly more general probabilistic model, which has been recently introduced to model practical crowdsourced
annotations. Under this generalized Dawid-Skene model, we characterize the fundamental trade-off between budget and accuracy.
I will present a novel adaptive task assignment scheme that matches this fundamental limit. This allows us to quantify the
fundamental gap between adaptive and non-adaptive schemes, by comparing the trade-off with the one for non-adaptive schemes.