Post on 22-Feb-2016
description
Generative Models for Crowdsourced Data
Outline
• What is Crowdsourcing?• Modeling the labeling process• Example with real data• Extensions• Future Directions
What is Crowdsourcing?
• Human based computation.• Outsourcing certain steps of a computation to
humans.• ``Artificial artificial intelligence.’’• Data science:– Making an immediate decision.– Creating a labeled data set for learning.
Immediate Decision Workflow
Labeled Data Set Workflow
An Example HIT
An Example HIT
Funny enough …
• Not everybody agrees on the gender of a Twitter profile.
• Difficult Instances• Worker Ability / Motivation• Worker Bias• Adversarial Behaviour
Difficult Instance
Difficult Instance
Difficult Instance
Worker Ability
Worker Ability
Worker Ability
Worker Motivation
Worker Motivation
Worker Bias
Worker Bias
Worker Bias
Disagreements
• When some workers say “male” and some workers say “female”, what to do?
Majority Rules Heuristic
• Assign label l to item x if a majority of workers agree.
• Otherwise item x remains unlabeled.
Majority Rules Heuristic
• Assign label l to item x if a majority of workers agree.
• Otherwise item x remains unlabeled.• Ignores prior worker data.
Majority Rules Heuristic
• Assign label l to item x if a majority of workers agree.
• Otherwise item x remains unlabeled.• Ignores prior worker data.• Introduce bias in labeled data.
Train on all labels
• For labeled data set workflow.• Add all item-label pairs to the data set.• Equivalent to cost vector of:– P (l | { lw }) = 1/nw S 1{l = lw}
Train on all labels
• For labeled data set workflow.• Add all item-label pairs to the data set.• Equivalent to cost vector of:– P (l | { lw }) = 1/nw S 1{l = lw}
• Ignores prior worker data.
Train on all labels
• For labeled data set workflow.• Add all item-label pairs to the data set.• Equivalent to cost vector of:– P (l | { lw }) = 1/nw S 1{l = lw}
• Ignores prior worker data.• Models the crowd, not the “ground truth.”
What is ground truth
• Different theoretical approaches.– PAC learning with noisy labels.– Fully-adversarial active learning.
• Bayesians have been very active.– “Easy” to posit a functional form and quickly
develop inference algorithms.– Issue of model correctness is ultimately empirical.
Bayesian Literature
• (2009) Whitehill et. al. GLAD framework.– (1979) Dawid and Skene. Maximum Likelihood
Estimation of Observer Error-Rates Using the EM Algorithm.
• (2010) Welinder et. al. The Multidimensional Wisdom of Crowds.
• (2010) Raykar et. al. Learning from Crowds.
Bayesian Approach
• Define ground truth via a generative model which describes how “ground truth” is related to the observed output of crowdsource workers.
• Fit to observed data.• Extract posterior over ground truth.• Make decision or train classifier.
Generative Model
Example: Binary Classification
• Each worker has a matrix.
α = ( -1 α01 )
( α10 -1 )
• Each item has a scalar difficulty β > 0.• P (lw = j | z = i) = e-βαij / (Σk e-βαik)
• αij ~ N (μij, 1) ; μij ~ N (0, 1)• log β ~ N (ρ, 1) ; ρ ~ N (0, 1)
Other Problems
• Multiclass classification:– Same as binary with larger confusion matrix.
• Ordinal classification: (“Hot or not”)– Confusion matrix has special form.– O (L) parameters instead of O (L2).
• Multilabel classification:– Reduce to multiclass on power set.– Assume low-rank confusion matrix.
EM
EM
• Initially all workers are assumed moderately accurate and without bias.– Implies initial estimate of ground truth distribution
favors consensus.– Disagreeing with the majority is a likely error.
EM
• Initially all workers are assumed moderately accurate.
• Workers consistently in the minority have their confusion probabilities increase.
EM
• Initially all workers are assumed moderately accurate.
• Workers consistently in the minority have their confusion probabilities increase.
• Workers with higher confusion probabilities contribute less to the distribution of ground truth.
“Different” workers are marginalized
“Different” workers are marginalized
• Workers that are consistently in the minority will not contribute strongly to the posterior distribution over ground truth.– Even if they are actually more accurate.
• Can correct when an accurate worker(s) is paired with some inaccurate workers.
• Good for breaking ties.• Raykar et. al.
Example with real data
Online EM
• Given a set of worker-label pairs for a single item:
• (Inference) Using current α, find most likely β* and distribution q* over ground truth.
• (Training) Do SGD update of α with respect to EM auxiliary function evaluated at β* and q*.
Online EM
• Given a set of worker-label pairs for a single item:
• (Inference) Using current α, find most likely β* and distribution q* over ground truth.
• (Training) Do SGD update of α with respect to EM auxiliary function evaluated at β* and q*.
Things to do with q*
• Take an immediate cost-sensitive decision– d* = argmind Ez~q*[f (z, d)]
• Train a (importance-weighted) classifier– cost vector cd = Ez~q*[f (z, d)]– e.g. 0/1 loss: cd = (1 - q*d)– e.g. binary 0/1 loss: |c1 – c0| = |1 – 2 q*1|– No need to decide what the true label is!
• Raykar et. al.: why not jointly estimate classifier and worker confusion?
Raykar et. al. insight
• Cost vector is constructed by estimating worker confusion matrices.
• Subsequently, classifier is trained; it will sometimes disagree with workers.
• Would be nice to use that disagreement to inform the worker confusion matrices.
• Circular dependency suggests joint estimation.
Generative Model
Generative Model
Online Joint Estimation
Online Joint Estimation
• Initially the classifier will output an uninformative prior and therefore will be trained to follow consensus of workers.
• Eventually workers which disagree with the classifier will have their confusion probabilities increase.
• Workers consistently in the minority can contribute strongly to the posterior if they tend to agree with the classifier.
Additional Resources
• Software– http://code.google.com/p/nincompoop
• Blog– http://machinedlearnings.com/