A Support Framework for ArgumentativeDiscussions Management in the Web

Elena Cabrio, Serena Villata, Fabien Gandon

Wimmics Team

INRIA, I3S - Sophia Antipolis, France

Supporting community managers usingNLP and argumentation

COMMUNITYMANAGER

GOALEfficient management of wiki pages by community managers and animations of communities

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 2

Supporting community managers usingNLP and argumentation

TEXTUALENTAILMENT

TEXTUALENTAILMENT

How to detect the arguments, And the relationships among them?

1

COMMUNITYMANAGER

GOALEfficient management of wiki pages by community managers and animations of communities

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 3

Supporting community managers usingNLP and argumentation

TEXTUALENTAILMENT

TEXTUALENTAILMENT

ARGUMENTATIONTHEORY

ARGUMENTATIONTHEORY

How to detect the arguments, And the relationships among them?

1

How to build the overall graph of the changes and discover the winning arguments?

2

COMMUNITYMANAGER

GOALEfficient management of wiki pages by community managers and animations of communities

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 4

Supporting community managers usingNLP and argumentation

TEXTUALENTAILMENT

TEXTUALENTAILMENT

ARGUMENTATIONTHEORY

ARGUMENTATIONTHEORY

How to detect the arguments, And the relationships among them?

1

How to build the overall graph of the changes and discover the winning arguments?

2

RDF/SPARQL

RDF/SPARQL

3COMMUNITYMANAGER

How to extract further insightful information?

GOALEfficient management of wiki pages by community managers and animations of communities

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 5

Outline

1 Related literature

2 Textual Entailment and Argumentation

3 Combined Framework

4 Experimental setting on Wikipedia revisions

5 Conclusions

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 6

Related literature

• Wikipedia revisions in NLP tasksZanzotto and Pennacchiotti (2010)Expanding textual entailment corpora from Wikipedia using co-trainingCabrio et al. (2012)Extracting context-rich entailment rules from wikipedia revision historyNelken and Yamangil (2008)Mining wikipedia revision histories for improving sentence compressionMax and Wisniewski (2010)Mining naturally-occurring corrections and paraphrases from wikipedia’s revisionhistoryDutrey et al. (2011)Local modifications and paraphrases in wikipedia’s revision history

• Argumentation and NLPMoens et al. (2007)Automatic detection of arguments in legal textsCarenini and Moore (2006)Generating and evaluating evaluative argumentsWyner and van Engers (2010)A framework for enriched, controlled online discussion forums for e-governmentpolicy-makingHeras et al. (2010)How argumentation can enhance dialogues in social networks

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 7

Related literature

• Wikipedia revisions in NLP tasksZanzotto and Pennacchiotti (2010)Expanding textual entailment corpora from Wikipedia using co-trainingCabrio et al. (2012)Extracting context-rich entailment rules from wikipedia revision historyNelken and Yamangil (2008)Mining wikipedia revision histories for improving sentence compressionMax and Wisniewski (2010)Mining naturally-occurring corrections and paraphrases from wikipedia’s revisionhistoryDutrey et al. (2011)Local modifications and paraphrases in wikipedia’s revision history

• Argumentation and NLPMoens et al. (2007)Automatic detection of arguments in legal textsCarenini and Moore (2006)Generating and evaluating evaluative argumentsWyner and van Engers (2010)A framework for enriched, controlled online discussion forums for e-governmentpolicy-makingHeras et al. (2010)How argumentation can enhance dialogues in social networks

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 8

Textual Entailment

• Generic framework for capturing major semantic inferenceneeds in NLP applications (Dagan and Glickman, 2004).

• Relation between two textual fragments T and H:

T ⇒ H: meaning of H can be inferred from meaning of T , asinterpreted by a typical language user.

T (Wiki11): The land area of the contiguous United States is approximately1,800 million acres (7,300,000 km2)

H (Wiki10): The land area of the contiguous United States is approximately1.9 billion acres (770 million hectares)

T (Wiki10): The land area of the contiguous United States is approximately1.9 billion acres (770 million hectares)

H (Wiki09): The total land area of the contiguous United States is approxima-tely 1.9 billion acres.

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 9

Textual Entailment

• Generic framework for capturing major semantic inferenceneeds in NLP applications (Dagan and Glickman, 2004).

• Relation between two textual fragments T and H:

T ⇒ H: meaning of H can be inferred from meaning of T , asinterpreted by a typical language user.

T (Wiki11): The land area of the contiguous United States is approximately1,800 million acres (7,300,000 km2)

H (Wiki10): The land area of the contiguous United States is approximately1.9 billion acres (770 million hectares)

T (Wiki10): The land area of the contiguous United States is approximately1.9 billion acres (770 million hectares)

H (Wiki09): The total land area of the contiguous United States is approxima-tely 1.9 billion acres.

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 10

Abstract Argumentation Theory

• Directed graph (Dung, 1995)

Nodes: abstract argumentsEdges: attack relation

argumentA

argumentB

argumentC

argumentA

argumentB

IN OUT IN OUT IN

ATTACK ATTACK ATTACK

• Bipolar argumentation(Cayrol & Lagasquie-Schiex, 2005),(Boella et al., 2010)

b ca a bc b ca

Supported attack Secondary attack Mediated attack

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 11

Combined FrameworkWikipedia revisions for the article “United States”

T (Wiki12): The land area of the contiguous United States is 2,959,064 square miles (7,663,941 km2).

H (Wiki11): The land area of the contiguous United States is approximately 1,800 million acres

(7,300,000 km2)

T (Wiki11): The land area of the contiguous United States is approximately 1,800 million acres

(7,300,000 km2)

H (Wiki10): The land area of the contiguous United States is approximately 1.9 billion acres )

(770 million hectares)

T (Wiki10): The land area of the contiguous United States is approximately 1.9 billion acres

(770 million hectares)

H (Wiki09): The total land area of the contiguous United States is approximately 1.9 billion acres.

A2Wiki10

A3Wiki11

A1Wiki09

A4Wiki12

(a)

A2Wiki10

A3Wiki11

A1Wiki09

A4Wiki12

(b)

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 12

Revisions in RDF using

SIOC-Argumentation extended vocabulary

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 13

Revisions in RDF using

SIOC-Argumentation extended vocabulary

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 14

Extracting further informationfrom revisions in RDF

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 15

Experimental setting:preprocessing Wikipedia dumps

• 4 dumps of English Wikipedia (2009, 2010, 2011, 2012)• 5 most revised pages: United States, World War II, George

Bush, Michael Jackson, Britney Spears

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 16

Experimental setting:extraction of entailment pairs

• Documents are sentence splitted, and sentences are aligned

• To measure the similarity between the sentences: PositionIndependent Word Error Rate (PER) [Tillman et al., 1997]

• Different thresholds are set to cluster pairs into different sets

• Sentences with major editing are selected (0.2<PER<0.6)

• TE pair: revised sentence as T, original sentence as H

Entailment No EntailmentTraining Set 114 pairs 114 pairs

Test Set 101 pairs 123 pairs

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 17

Experimental setting: evaluation

• EDITS system (Edit Distance Textual Entailment Suite)(Kouylekov and Negri, 2010), off-the-shelf system

Basic configuration: word overlap and cosine similarityalgorithms; distance calculated on lemmas; stopword list

• FIRST STEP: TEXTUAL ENTAILMENTTrain Test

EDITS configurations rel Precision Recall Accuracy Precision Recall Accuracy

WordOverlapyes 0.83 0.82

0.830.83 0.82

0.78no 0.76 0.73 0.79 0.82

CosineSimilarityyes 0.58 0.89

0.630.52 0.87

0.58no 0.77 0.37 0.76 0.34

• SECOND STEP: TE+ARGUMENTATION THEORYTest

Configuration Precision Recall F-measureWordOverlap + AT 0.90 0.92 0.91

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 18

Experimental setting: evaluation

• EDITS system (Edit Distance Textual Entailment Suite)(Kouylekov and Negri, 2010), off-the-shelf system

Basic configuration: word overlap and cosine similarityalgorithms; distance calculated on lemmas; stopword list

• FIRST STEP: TEXTUAL ENTAILMENTTrain Test

EDITS configurations rel Precision Recall Accuracy Precision Recall Accuracy

WordOverlapyes 0.83 0.82

0.830.83 0.82

0.78no 0.76 0.73 0.79 0.82

CosineSimilarityyes 0.58 0.89

0.630.52 0.87

0.58no 0.77 0.37 0.76 0.34

• SECOND STEP: TE+ARGUMENTATION THEORYTest

Configuration Precision Recall F-measureWordOverlap + AT 0.90 0.92 0.91

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 19

Experimental setting: evaluation

• EDITS system (Edit Distance Textual Entailment Suite)(Kouylekov and Negri, 2010), off-the-shelf system

Basic configuration: word overlap and cosine similarityalgorithms; distance calculated on lemmas; stopword list

• FIRST STEP: TEXTUAL ENTAILMENTTrain Test

EDITS configurations rel Precision Recall Accuracy Precision Recall Accuracy

WordOverlapyes 0.83 0.82

0.830.83 0.82

0.78no 0.76 0.73 0.79 0.82

CosineSimilarityyes 0.58 0.89

0.630.52 0.87

0.58no 0.77 0.37 0.76 0.34

• SECOND STEP: TE+ARGUMENTATION THEORYTest

Configuration Precision Recall F-measureWordOverlap + AT 0.90 0.92 0.91

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 20

1 Connect users to their arguments in online communities

2 Arguments’ evaluation depending on sources’ expertise

3 TE three-way judgement task:entailment, contradiction, unknown

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 21

Thanks for your attention!

http://bit.ly/WikipediaDatasetXMLhttp://bit.ly/WikipediaDatasetRDF

http://bit.ly/SIOC_Argumentation

E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 22