Max-Margin Majority Voting for Learning from CrowdsTian Tian and Jun Zhu

from Department of Computer Science and Technology, Tsinghua University.

The Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS) Palais des Congrès de Montréal, Montréal CANADA

Apricot Peach Peach Peach

Apricot Apricot Apricot Peach

Objective

Aggregating noisy crowdsourcing labels to find Ground Truths.

Majority Voting (MV)Items: 𝑖 ∈ 𝑀 Workers: 𝑗 ∈ 𝑁 Worker labels: 𝑥𝑖𝑗∈ 𝐷

Each item have a ground truth: 𝑦𝑖 ∈ [𝐷] 𝒙𝑖：{𝑥𝑖𝑗，∀𝑗}

MV: find the most frequent labels

Limitations:1. Workers are equal, lack of discrimination ability

2. Do not consider worker confusability

Constraint Formulation of MVExpansion Expression

Def: 𝒈 𝒙𝑖 , 𝑑 ∈ {0,1}𝑁 , element 𝑗 is 𝕀(𝑥𝑖𝑗 = 𝑑)

Constraint Formulation

MV is equivalent to find 𝒚 satisfying the constraints:

(1 -1 -1 -1)𝒙𝑖:(1 0 0 0)

(0 1 1 1)

𝒈 𝒙𝑖 , 1 :

𝒈 𝒙𝑖 , −1 :

Dawid-Skene Model (DS)

Define and estimate worker confusion matrices.𝜙𝑗 is the confusion matrix of worker

𝜙𝑗𝑘𝑑 = 𝑝 𝑥𝑖𝑗 = 𝑑 𝑦𝑖 = 𝑘 , ∀𝑖

CrowdSVM

Consider Majority Voting and confusability in a single model.

Max Margin Majority Voting (M3V)

Incorporate max-margin principle to estimate 𝜼

Here we using soft-margin for robustness.

(3 1)𝒙𝑖:

(0 1)

(0 0)

𝒈 𝒙𝑖 , 1 :

𝒈 𝒙𝑖 , 2 :

(1 0)𝒈 𝒙𝑖 , 3 :

Solving by iteratively updating 𝜼 and 𝒚.

Solver for 𝜼:

𝝎 is the solution of the dual problem:

Solver for 𝒚:

Experimental Results

Convergence:

Generative vs. Discriminative:

Conclusion:

1.Max-margin principle can enhance majority voting.

2.Both generative and discriminative component benefits

from the other.

Contact Info

Tian Tian (田天) PhD Student @ THU

Email: tiant13@mails.Tsinghua.edu.cn

Welcome for discussing!

Weighted MVWe introduce worker weights 𝜼 ∈ ℝ𝑁, then the constraint

formulation changed into

The discriminative function for infer ground truth:

We test the aggregation error rate on several popular

datasets.