Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple...
Transcript of Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple...
![Page 1: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/1.jpg)
Proactive Learning withMultiple Class-Sensitive Labelers
Seungwhan (Shane) Moon, Jaime Carbonell
School of Computer Science, Carnegie Mellon University
DSAA 2014 Conference 10/30/2014
![Page 2: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/2.jpg)
Proactive Learning withMultiple Class-Sensitive Labelers
Seungwhan (Shane) Moon, Jaime Carbonell
School of Computer Science, Carnegie Mellon University
DSAA 2014 Conference 10/30/2014
![Page 3: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/3.jpg)
Unlabeled Data is Abundant
3
![Page 4: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/4.jpg)
Unlabeled Data is Abundant
• Imagine building a Vehicle classifier
4
Scarcity of labeled data
![Page 5: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/5.jpg)
Active Learning
5
![Page 6: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/6.jpg)
Active Learning
6
![Page 7: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/7.jpg)
Query Strategies
• Uncertainty Sampling
• Query by Committee
• Entropy Based Sampling
• Density Weighted Methods
• and more …
7
![Page 8: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/8.jpg)
Uncertainty Sampling
Label 1
Label 2Unlabeled
Current Decision Boundary
8
![Page 9: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/9.jpg)
Uncertainty Sampling
Label 1
Label 2Unlabeled
Current Decision Boundary
= most uncertain
9
![Page 10: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/10.jpg)
Assumptions in Traditional Active Learning
• Annotator(s) always give perfect answers (oracle)
• There is no difference in cost for querying different annotators
10
![Page 11: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/11.jpg)
Proactive Learning [Carbonell et. al]
• Relaxes the following assumptions:
• Only a single annotator gives labels
• Annotators always give perfect answers
• Annotators are insensitive to costs
—> utility optimization under budget constraint
11
![Page 12: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/12.jpg)
Proactive Learning [Carbonell et. al]
12
Multiple annotators
They have different labeling accuracy (expertise) incur different cost
![Page 13: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/13.jpg)
Proactive Learning [Carbonell et. al]
13
Key Component: Estimating Labeler Accuracy
Probability of getting a right answer for an unlabeled instance x, and an expert k
Limitation in previous literature on proactive learning
Labeler accuracy is independent of label in multi-class problems
![Page 14: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/14.jpg)
Proactive Learning with Multiple Domain Experts: Anology
Motivation
14
Diagnosis of a patient with unknown disease (uncertainty in data)
![Page 15: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/15.jpg)
Proactive Learning with Multiple Domain Experts: Anology
Motivation
15
Diagnosis of a patient with unknown disease (uncertainty in data)Given multiple physicians with different specialization (multiple class-sensitive experts)
If we know the patient has seemingly cancer symptoms (posterior class probability)
And that oncologist treats cancer issues (estimated labeler accuracy given a specific class)
Better delegate a task to its respective expert
![Page 16: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/16.jpg)
Proactive Learning with Multiple Domain Experts
Problem Formulation (Objective)
Greedy Approximation
:::
16
![Page 17: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/17.jpg)
Proactive Learning with Multiple Domain Experts
Utility Criteria for Greedy Approximation
17
Jointly optimize for an instance and expert pair which
- has high information value V(X) (instance)- has high probability of getting the right answer (both)- has low cost of annotation (expert)
![Page 18: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/18.jpg)
Expert EstimationEstimating Expertise of Labeling Sources
18
over set of categories
class posterior probability of label for sample x being c
the estimated probability of expert k answering for label c
![Page 19: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/19.jpg)
Expert EstimationEstimating Expertise of Labeling Sources
Per-class Reduced Estimation
19
![Page 20: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/20.jpg)
Density Based Sampling for Multi-classification Tasks
20
Label 1
Label 2
Unlabeled
Current Decision Boundary
20
Label 3
![Page 21: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/21.jpg)
Density Based Sampling for Multi-classification Tasks
21
Label 1
Label 2
Unlabeled
Current Decision Boundary
21
Label 3
![Page 22: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/22.jpg)
Density Based Sampling for Multi-classification Tasks
(2) Unknownness(1) Density
(3) Conflictivity
Def: Multi-class Information Density (MCID)
22
Final Value Function
![Page 23: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/23.jpg)
Density Based Sampling for Multi-classification Tasks
Induce Density using a Gaussian Mixture Model
Estimation via an EM Procedure
Each Mixture Sharing the Same Variance
23
![Page 24: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/24.jpg)
So far:New Proactive Learning Algorithmfor Multiple Domain Experts
Multi-class Information Density (MCID) as a query strategy
24
![Page 25: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/25.jpg)
ExperimentsDataset
Simulated Noisy Labelers (except for Diabetes dataset)Narrow Experts: Classifier trained over partially noised dataset (expertise in only a subset of classes)
Meta Expert: Classifier trained over the entire dataset25
![Page 26: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/26.jpg)
Baselines
Best Avg: learner always asks one of the narrow experts that has the highest average P(ans|x, k) Meta : learner always asks meta-oracle (expensive) BestAvg+Meta: joint optimization under uniform reliability assumption (Donmez et al., 2012)
*Narrow: joint optimization using our algorithm *Narrow+Meta: with the presence of an meta oracle as well
![Page 27: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/27.jpg)
Classification Performance Over Iterations
Cost Ratio of Narrow vs Meta: 1:627
![Page 28: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/28.jpg)
Classification Performance for Different Cost Ratios
28
![Page 29: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/29.jpg)
On other datasets
29
![Page 30: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/30.jpg)
Classification Performance vs. Budget Allocated for Expertise Estimation
- Works for both when there are ground truth samples available & via majority votes
- Is able to estimate expertise well enough with ~10% budget
30
![Page 31: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/31.jpg)
Conclusions
• A new proactive learning algorithm with multiple class sensitive labellers accounts better than baselines
• Efficient estimation of expert’s expertise via reduced per-class method
• Multi-class Information Density (MCID) as a new active learning criteria for noised multi-class active learning
31
![Page 32: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/32.jpg)
Future Work
• Theoretical min-max bounds of the proposed algorithm, under different reliabilities and costs of the experts
• Extend the framework to a crowdsourcing scenario with a larger pool of experts
32
![Page 33: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/33.jpg)
Proactive Learning withMultiple Class-Sensitive Labelers
Seungwhan Moon, Jaime Carbonell
Language Technology Institute School of Computer Science, Carnegie Mellon University
DSAA 2014 Conference 10/30/2014
33
![Page 34: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/34.jpg)
MCID Performance
34
![Page 35: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/35.jpg)
Performance when expertise was estimated via Majority Vote
![Page 36: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/36.jpg)
Proactive Learning Algorithm
36
![Page 37: Proactive Learning with Multiple Class-Sensitive …14c_DSAA...Proactive Learning with Multiple Class-Sensitive Labelers Seungwhan (Shane) Moon, Jaime Carbonell School of Computer](https://reader033.fdocuments.us/reader033/viewer/2022041821/5e5e5af5d50e272aba333230/html5/thumbnails/37.jpg)
Expertise Estimation
37