Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types...
Transcript of Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types...
![Page 1: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/1.jpg)
Argument Mining: Optimizing Search and Decision Processes by Means of Large-Scale Unstructured Text Data
Iryna GurevychUKP LabComputer Science
![Page 2: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/2.jpg)
12018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
What is Argument Mining?
Argument Mining§ Recognize arguments in text automatically § Assessing the quality of textual arguments§ Based on supervised machine learning
Discourse-level perspective§ Argument components§ Argumentative structures§ Single document of specific type
Information-seeking perspective (focus of this talk)§ Arguments relevant to a given topic§ Multiple documents
![Page 3: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/3.jpg)
22018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Goals of Information-Seeking Perspective
Given: a controversial topic (e.g. “autonomous cars” or “basic income”)
Extract pro and con arguments from different kinds of text
Unstructured text Extract evidence Summarize / Group
Pro Con
![Page 4: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/4.jpg)
32018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
ArgumenText:http://www.argumentext.de/showcases/
![Page 5: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/5.jpg)
42018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Research Challenges
Challenge 1: Annotating arguments in heterogeneous texts
Challenge 2: Creating large amounts of training data
Challenge 3: Training models robust enough for different topics
![Page 6: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/6.jpg)
52018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Challenge 1
Annotating arguments in heterogeneous texts
![Page 7: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/7.jpg)
62018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Annotation Model: Requirements and Solution
Requirements1. Applicable to information seeking perspective
2. General enough for heterogeneous texts
3. Simple enough for crowdsourcing
Our solution
§ Topic is some matter of controversy that can be expressed with keywords
§ Argument is a span of text with evidence supporting or opposing a given topic
§ Three classes sentence-wise: pro, con, no argument
![Page 8: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/8.jpg)
72018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Annotation Model: Examples
Topic Sentence Labelnuclear energy Nuclear fission is the process that is used in nuclear reactors to
produce energy using element called uranium.?
nuclear energy The amount of greenhouse gases have decreased by almost half because of the prevalence in the utilization of nuclear power.
minimum wage A 2014 study [. . . ] found that minimum wage workers are more likely to report poor health and suffer from chronic diseases.
minimum wage We should abolish all Federal wage standards and allow states and localities to set their own minimums.
Three classes sentence-wise: pro, con, no argument
![Page 9: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/9.jpg)
82018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Annotation Model: Examples
Topic Sentence Labelnuclear energy Nuclear fission is the process that is used in nuclear reactors to
produce energy using element called uranium.no argument
nuclear energy The amount of greenhouse gases have decreased by almost half because of the prevalence in the utilization of nuclear power.
?
minimum wage A 2014 study [. . . ] found that minimum wage workers are more likely to report poor health and suffer from chronic diseases.
minimum wage We should abolish all Federal wage standards and allow states and localities to set their own minimums.
Three classes sentence-wise: pro, con, no argument
![Page 10: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/10.jpg)
92018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Annotation Model: Examples
Topic Sentence Labelnuclear energy Nuclear fission is the process that is used in nuclear reactors to
produce energy using element called uranium.no argument
nuclear energy The amount of greenhouse gases have decreased by almost half because of the prevalence in the utilization of nuclear power.
pro argument
minimum wage A 2014 study [. . . ] found that minimum wage workers are more likely to report poor health and suffer from chronic diseases.
?
minimum wage We should abolish all Federal wage standards and allow states and localities to set their own minimums.
Three classes sentence-wise: pro, con, no argument
![Page 11: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/11.jpg)
102018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Annotation Model: Examples
Topic Sentence Labelnuclear energy Nuclear fission is the process that is used in nuclear reactors to
produce energy using element called uranium.no argument
nuclear energy The amount of greenhouse gases have decreased by almost half because of the prevalence in the utilization of nuclear power.
pro argument
minimum wage A 2014 study [. . . ] found that minimum wage workers are more likely to report poor health and suffer from chronic diseases.
con argument
minimum wage We should abolish all Federal wage standards and allow states and localities to set their own minimums.
?
Three classes sentence-wise: pro, con, no argument
![Page 12: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/12.jpg)
112018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Annotation Model: Examples
Topic Sentence Labelnuclear energy Nuclear fission is the process that is used in nuclear reactors to
produce energy using element called uranium.no argument
nuclear energy The amount of greenhouse gases have decreased by almost half because of the prevalence in the utilization of nuclear power.
pro argument
minimum wage A 2014 study [. . . ] found that minimum wage workers are more likely to report poor health and suffer from chronic diseases.
con argument
minimum wage We should abolish all Federal wage standards and allow states and localities to set their own minimums.
no argument
Three classes sentence-wise: pro, con, no argument
![Page 13: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/13.jpg)
122018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Annotation Model: Expert Annotations
Data§ Heterogeneous text types (news, online discussions, blogs, social media, etc.)§ Eight controversial topics, e.g. “school uniforms”, “gun control”, etc.§ Collected from web searches (query Google for topic)
Annotation Study§ Two expert annotators § Graduate-level language technology researchers § Independent annotation of 200 sentences for each topic (1.600 total)
Average agreement over topics§ kappa = 0.721§ Sufficient agreement: Annotation model is applicable to heterogeneous texts
by expert annotators
![Page 14: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/14.jpg)
132018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Challenge 2
Creating large amounts of training data
![Page 15: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/15.jpg)
152018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Comparing Experts to Crowdworkers
Results§ High quality annotations using crowdsourcing
§ Crowdworkers achieve kappa =.723 agreement with expert annotations
0,651
0,712
0,657
0,783
0,7290,779
0,686
0,767
0,660,704
0,576
0,638
0,749 0,745
0,825
0,889
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
1 2 3 4 5 6 7 8
kappa
topics
experts vs. experts experts vs. crowd
![Page 16: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/16.jpg)
162018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Statistics of Final Corpus
§ Annotation process is scalable: 25k+ instances in less than a week§ Costs: $2,774§ Corpus allows learning a classifier for argument mining across topics
![Page 17: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/17.jpg)
172018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Challenge 3
Training a classifier robust enough for different topics
![Page 18: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/18.jpg)
182018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Experimental Setup
Experiments1. Can we improve accuracy by leveraging the topic?
2. Does more training data improve the results?
Evaluation setup§ Task: classify a sentence as “argument” or “no argument” relevant to the topic
§ In-domain: train and test on the same topic
§ Cross-domain: train on n-1 topics and test on left-out-topic
![Page 19: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/19.jpg)
192018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Experiment 1: Models
Baselines§ majority: classifies each instance as “no argument”§ lr-uni: logistic regression with binary unigram features§ bilstm: bidirectional long short-term memory network 300d embeddings
Models with topic information§ bilstm+cos: bilstm model with topic similarity feature§ inner-att: learns weighting of input word with respect to the given topic§ inner-att+cos: combines bilstm+cos and inner-att models
![Page 20: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/20.jpg)
202018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Experiment 1: Evaluation
Mac
ro F
1
0,36 0,36
0,7
0,6
0,721
0,592
0,732
0,626
0,741
0,623
0,736
0,658
0,35
0,4
0,45
0,5
0,55
0,6
0,65
0,7
0,75
in-domain cross-domain
majority lr-uni bilstm bilstm+cos inner-att inner-att+cos
![Page 21: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/21.jpg)
212018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Experiment 1: Evaluation
Mac
ro F
1
0,36 0,36
0,7
0,6
0,721
0,592
0,732
0,626
0,741
0,623
0,736
0,658
0,35
0,4
0,45
0,5
0,55
0,6
0,65
0,7
0,75
in-domain cross-domain
majority lr-uni bilstm bilstm+cos inner-att inner-att+cos
Adding topic information improves baseline results
![Page 22: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/22.jpg)
222018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Experiment 1: Evaluation
Mac
ro F
1
0,36 0,36
0,7
0,6
0,721
0,592
0,732
0,626
0,741
0,623
0,736
0,658
0,35
0,4
0,45
0,5
0,55
0,6
0,65
0,7
0,75
in-domain cross-domain
majority lr-uni bilstm bilstm+cos inner-att inner-att+cos
Inner-att achieves best in-domain results
![Page 23: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/23.jpg)
232018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Experiment 1: Evaluation
Mac
ro F
1
0,36 0,36
0,7
0,6
0,721
0,592
0,732
0,626
0,741
0,623
0,736
0,658
0,35
0,4
0,45
0,5
0,55
0,6
0,65
0,7
0,75
in-domain cross-domain
majority lr-uni bilstm bilstm+cos inner-att inner-att+cos
Inner-att+cos generalizes best to unknown topics
![Page 24: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/24.jpg)
242018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Experiment 2:Corpus Extension
Does the model generalize better if more topics are in the training data?
Corpus Extension§ Add additional 41 topics to our training data
§ e.g. “autonomous driving”, “cryptocurrency”, “drones”, “biofuel”, etc.
§ Per topic ~600 additional annotated instances
Size of extended corpus§ 49 topics
§ 50k+ instances
![Page 25: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/25.jpg)
252018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Results Using Extended Corpus
0,36 0,36 0,36
0,7
0,6
0,68
0,74
0,66
0,73
0,35
0,4
0,45
0,5
0,55
0,6
0,65
0,7
0,75
majority lr-uni inner-att+cos
Mac
ro F
1
In-domain Cross-domain(eight topics)
Cross-domain(extended corpus)
![Page 26: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/26.jpg)
262018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Results Using Extended Corpus
0,36 0,36 0,36
0,7
0,6
0,68
0,74
0,66
0,73
0,35
0,4
0,45
0,5
0,55
0,6
0,65
0,7
0,75
majority lr-uni inner-att+cos
Mac
ro F
1
In-domain Cross-domain(eight topics)
Cross-domain(extended corpus)
Adding more topics to training data helps
![Page 27: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/27.jpg)
272018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Results Using Extended Corpus
0,36 0,36 0,36
0,7
0,6
0,68
0,74
0,66
0,73
0,35
0,4
0,45
0,5
0,55
0,6
0,65
0,7
0,75
majority lr-uni inner-att+cos
Mac
ro F
1
In-domain Cross-domain(eight topics)
Cross-domain(extended corpus)
Inner-att+cos achieves almost in-domain results without seeing the topic in test data
![Page 28: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/28.jpg)
282018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
What Does the Model Learn?
Visualization of attention weights
Topic relevant to the sentence
Topic not relevant to the sentence
![Page 29: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/29.jpg)
292018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Online Argument Search System
Data§ Web corpus (CommonCrawl)
§ 400 Mio. English webpages
Offline Processing§ Boilerplate removal
§ Sentence splitting
§ Indexing using ElasticSearch
Online Processing§ Retrieve topic relevant documents
§ Extract pro and con arguments
Web-Interface§ Pro/Con lists
§ Source filtering
§ Document ranking based on #arguments
![Page 30: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/30.jpg)
302018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
User Study
Compare system outcome with arguments from debate portals§ Three topics from ProCon.org (“cellphones”, “social networking”, and “animal testing”)§ 1,529 classified sentence from our system§ Three undergraduate students of computer science
For each sentence s from our system§ Can s be mapped to an expert-created argument (coverage)?§ Is s a completely new argument (novelty)?§ Is s not an argument / wrong stance / nonsensical (no argument)?
Results§ Coverage: 89% with arguments from ProCon.org (full coverage for two topics)§ Novelty: 12% are completely new arguments not mentioned on ProCon.org§ No argument: 47% are either an argument classified with a wrong stance, a non-
argument, or nonsensical
![Page 31: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/31.jpg)
312018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Summary
Sentential annotation model§ Reliably applicable to heterogeneous texts
§ Simple enough for crowdsourcing
New corpus for argument search§ Heterogeneous text types
§ Allows cross-topic experiments
Cross-topic experiments § Inner-att+cos generalizes best
§ Achieves almost in-domain results when trained with additional topics
Future Work§ Language adaptation to support German, argument clustering and structuring
![Page 32: Argument Mining: Optimizing Search and Decision Processes ... · Data §Heterogeneous text types (news, online discussions, blogs, social media, etc.) §Eight controversial topics,](https://reader030.fdocuments.us/reader030/viewer/2022041019/5ecd3fdcc59797419458500e/html5/thumbnails/32.jpg)
322018 | Computer Science Department | Ubiquitous Knowledge Processing (UKP) Lab | Prof. Iryna Gurevych |
Thank you for your attention.
Christian Stab
Johannes Daxenberger
TristanMiller
SteffenEger
BenjaminSchiller
ChristopherTauchmann
ChrisStahlhut
Researchers involved in this project (alphabetical order)