Aspect Guided Text Categorization with Unobserved Labels

Dan Roth, Yuancheng TuUniversity of Illinois at Urbana-Champaign

Text Categorization

An archetypical Multi-Class Classification (MCC) problem F : X → Y a document, d X , a collection of classes Y = {c∈ 1, c2, . . . , cN}

Sports

Health

Business…Science

C1C2C3…CN

2

Motivation: what are we missing?

Class labels (Y) contain information which can help classification

How can we explore the label space?

C1

C2

C3…CN

Sports

Health

Business…Science

3

Car Navigation Command Classification

4

XY

Aspect Variables

5

X

Y

find nearest restaurant

Show me where I can eat nearby

Find nearest restaurantAction Detail Modifier Topic Manner

z1 z2 z3 z4 z5

NullNull

Significance of the Aspect Variables

Predicting better aspects implies predicting better class labels

Adding constraints to the aspect space

Predicting previously unobserved labels

6

If Topic = “restaurant”, then Action ≠ “turn”

Observed Label

1. turn on the radio2. GPS navigation

Unobserved Label

turn on GPS

Outline

Car Command Text Categorization Task Data and aspects

Unobserved labels

Constrained Conditional Model (CCM) Aspects variables to introduce constraints

Objective function

Training and Inference

Experimental Results Standard multiclass classification setting

Predicting Unobserved Labels

Conclusion

7

xX

x1

x6

x2

x5

x4x3

x7

X

Yy1

Adding Constraints by Hidden Aspects

Intuition: introduce structure on hidden variables

Adding Constraints Through Hidden Aspects

xX

x1

x6

x2

x5

x4x3

x7

xX

YZ1

Z2

Z4

Z3

Z5

y1

Use constraints to capture the dependencies

X

Objective Function of CCM

10

Weight Vector for “local” learners

Aspect functions

Penalty for violatingthe constraint.

How far away is y from a “legal” assignment

11

Training and Inference

Learning + Inference (L+I) Ignore constraints during training

Inference Based Training (IBT) Consider constraints during training

References to CCM (aka ILP formulation) Roth&Yih04, Has been shown useful in the context of many NLP problems:

SRL, Summarization; Co-reference; Information Extraction; Transliteration

07; Punyakanok et.al 05,08; Chang et.al 07,08; Clarke&Lapata06,07;

Denise&Baldrige07;Goldwasser&Roth'08; Martin,Smith&Xing'09

12

Learning + Inference

x1

x6

x2

x5

x4x3

x7

Z1Z2

Z5

Z4

Z3

f1(x)

f2(x)

f3(x)f4(x)

f5(x)

X

Y

Learning + Inference (L+I)Learn models independently

13

-1 1 111Y’ Local Predictions

Inference Based TrainingExample: Perceptron-based Global Learning

x1

x6

x2

x5

x4x3

x7

f1(x)

f2(x)

f3(x)f4(x)

f5(x)

X

Y

-1 1 1-1-1YTrue Global Labeling

-1 1 11-1Y’ Apply Constraints:

Outline Car Command Text Categorization Task

Data and aspects

Unobserved labels

Constrained Conditional Model (CCM) Aspects variables to introduce constraints

Objective function

Training and Inference

Experimental Results Standard multiclass classification setting

Predicting Unobserved Labels

Conclusion

14

Car Navigation Command Classification

15

XY

Evaluation Metrics

Standard Accuracy The percentage of correctly labeled examples

Weighted Aspect-based Metric (WAM) A weighted Hamming distance computed at the aspect level

16

Experiments and Evaluation Standard MCC Setting

17

Algorithm Accuracy (%)

WAM(%)

Baseline 67.84 86.14

MCC(L+I) 71.18 89.65Error Reduction (%)

10.39 25.32

Accuracy Fless Kappa

Human Annotation 75% 0.764

Experiments and Evaluation

Standard MCC Setting

18

Aspects

CCM Baseline

Error Reduction(%)

Topic 86.14 81.55 24.88

Action 88.31 82.72 32.35

Manner 89.98 87.35 20.79

Modifier

91.15 89.51 15.64

Detail 92.68 89.59 29.68

Experiments and Evaluation

Predicting Unobserved Labels

19

Algorithm Accuracy (%)

WAM(%)

Baseline 0.00 58.43

MCC(L+I) 28.16 70.27Error Reduction (%)

28.16 28.48

Unobserved Label

turn on GPS

Observed Label

1. turn on the radio2. GPS navigation

Conclusion

Summary

Text Categorization with a meaningful,

structured label space

A model that exploits the structure by

adding hidden aspect variables

Adding constraints and reformulating

the task as a structure prediction

problem

Predicting unobserved new labels

20

Thank You!AND

Questions?

21