amw_2013

7/30/2019 amw_2013

1/21

Semantic Answer Validation inQuestion Answering Systems for

Reading Comprehension Tests

7th Alberto Mendelzon International Workshop on

Foundations of Data ManagementMAY 21 - 23, 2013

Authors:

Helena Gmez Adorno

David Pinto Avendao

Darnes Vilario Ayala

Benemrita Universidad Autnoma de Puebla

Facultad de Ciencias de la Computacin

7/30/2019 amw_2013

2/21

Outline

Introduction

Proposed System Architecture

Document Processing

Information Retrieval Answer Validation

Test Corpus

Obtained Results

Conclusions

Future Work

2

7/30/2019 amw_2013

3/21

The problem3

Annie Lennox Why I am an HIVAIDS

activist. I'm going to share with you

the story as to how I have become

an HIV/AIDS campaigner. And thisis the name of my campaign, SING

Campaign. In November of 2003 I

was invited to take part in the

launch of Nelson Mandela's 46664

Foundation. That is his HIV/AIDS

foundation. And 46664 is thenumber that Mandela had when he

was imprisoned in Robben Island.

Who is the founder of the SING campaign?1) Nelson Mandela

2) Youssou N'Dour

3) Michel Sidibe

4) Zackie Achmat

5) Annie Lennox

7/30/2019 amw_2013

4/21

System Architecture4

7/30/2019 amw_2013

5/21

Document Processing5

Perform anaphora resolution for the documents using the

JavaRAP system.

Step 1

Identify the author ofthe document.

Which is usually the

first name in thedocument (NNP tag)

The Stanford POStagger was used

Step 2

Each personalpronoun in the firstperson of the set

PRP={ "I", "me","my", "myself }

Generally refers tothe author.

Step 3

Replace each termof the document

that is in the PRP, by

the documentauthor name

identified in step 1

7/30/2019 amw_2013

6/21

Anafora Resolution Example6

Given the following text:

Emily Oster flips our thinking on AIDS in Africa. So I want to talk to you today about

AIDS in sub-Saharan Africa. I imagine you all know something about AIDS

Step 1: Emily_NNP Oster_NNP flips_VBZ our_PRP$ thinking_NN on_IN AIDS_NNP

in_IN Africa.In this case, the 2 first terms that have the NNP label are selected to identify the

author.Autor = Emily Oster

Step 2: So_NNP I_PRP want_VBP to_TO talk_VB to_TO you_PRP today_NN about_IN

AIDS_NNP in_IN sub-Saharan_NNP Africa_NNP ._. I_PRP imagine_VBP you_PRP

all_DT know_VBP something_NN about_IN AIDS_NNP ._.Here are identified 2 labels that belong to the PRP set.

Step 3: So Emily Oster want to talk to you today about AIDS in sub-Saharan Africa.

Emily Oster imagine you all know something about AIDS.

The words of the PRP set are replaced by the author of the document.

7/30/2019 amw_2013

7/21


7/30/2019 amw_2013

8/21

Hypothesis Generation8

Question: Who is the founder of the SING campaign?

Answer 1: Nelson Mandela

Answer 5: Annie Lennox

From the previous question and their possible answers, the following

hypotheses are obtained:

Hipothesis 1: Nelson Mandela is the founder of the SING campaign

Hipothesis 5: Annie Lennox is the founder of the SING campaign

.

.

.

.

.

.

A Part-Of-Speech (POS) tagger is applied in order to identify

the questionkeywords (what, where, when, who, etc.).

Afterwards those words are replaced by each of the five

possible answers, thereby obtaining five hypotheses for each

question

7/30/2019 amw_2013

9/21

Information Retrieval9

Was built using the Lucene IR library.

Responsible for indexing the document collection and for the

further passage retrieval, given an hypothesis as a query.

Returns a relevant passage for each hypothesis. This passages

are later used as a support text to decide if hypothesis may be

the right answer.

H1 Nelson Mandela founder SING campaign Everyone reveres Nelson Mandela .

H2 Youssou N'Dour founder SING campaign So this is Annie Lennox SING Campaign .

H3 Michel Sidibe founder SING campaignAnnie Lennox 'm sitting here in New Yorkwith Michel Sidibe .

H4 Zackie Achmat founder SING campaign

Annie Lennox met Zackie Achmat , the

founder of Treatment Action Campaign , an

incredible campaigner and activist at a

46664 event .

H5 Annie Lennox founder SING campaign So this is Annie Lennox SING Campaign .

7/30/2019 amw_2013

10/21


7/30/2019 amw_2013

11/21

Semantic Similarity11

Given the pair (H, T), a semantic similarity score is calculated.

The similarity measure proposed by Rada Mihalcea gives a

weight to each word of the sentence in terms of the degree of

specificity of the word.

Hipothesis (H) : Then perhaps we couldhave avoided a catastrophe.

Support Text (T) : Perhaps we should have been able to prevent a

disaster.

Similarity: 4.500

The similarity between two sentences (H, T) is given by:

, =1

2

,

+

,

7/30/2019 amw_2013

12/21

maxSim(w1,w2) Similarity12

PMI-IR: It is based on statistical data collected by an

information retrieval engine over a very large corpus (i.e. the

web). Given two words w1 y w2, its PMI-IR is calculated:

Path_similarity : Return a score denoting how similar two

word senses are, based on the shortest path that connects thesenses in the is-a (hypernym/hypnoym) taxonomy of Wordnet.

, = &

7/30/2019 amw_2013

13/21

Answer Selection13

Question : Whois the founder of the SING campaign?

Hypothesis Retrieved Passage Score Pmi Path Sum

Nelson Mandela founder SING campaign Everyone reveres Nelson Mandela . 0.668 2.77 1.77 5.84

Youssou N'Dour founder SING campaign

So this is Annie Lennox SING

Campaign . 0.432 1.15 1.15 3.416

Michel Sidibe founder SING campaign

Annie Lennox 'm sitting here in New

York with Michel Sidibe . 0.569 2.16 1.86 5.296

Zackie Achmat founder SING campaign

Annie Lennox met Zackie Achmat ,

the founder of Treatment Action

Campaign , an incredible campaigner

and activist at a 46664 event . 1.361 2.48 2.08 6.633

Annie Lennox founder SING campaign

So this is Annie Lennox SING

Campaign . 1.530 2.54 2.54 7.359

7/30/2019 amw_2013

14/21

Test Corpus14

3 Topics 4 Reading Tests

AIDS

Climate Change

Music & Society

12 Reading tests

doc

10 Questions +

5 Candidate

Answers

120 Questions

600 Answers

Documents and questions are available in English German, Italian, Romanian and Spanish.

Provided by the QA4MRE task of the CLEF 2011

7/30/2019 amw_2013

15/21

Background Corpus15

7/30/2019 amw_2013

16/21

Obtained Results16

Similarity Score NCA NoA NIA Precision

PATH (wordnet) 32 0 88 27 %

PMI-IR 36 0 84 30 %

Lucene Score 39 0 81 33 %

PATH + PMI 33 0 87 28 %

PMI + Lucene Score 41 0 79 34 %

13%

20%

14%

53%

Lucene

Both

Lucene + PMI

None34 %

33 %

No. Of Correct AnswerApproach Quantity

None 64

Lucene Score 15

Both Aproaches 24

PMI + Lucene Score 17

7/30/2019 amw_2013

17/21

Conclusions17

I have presented a system using two semantic similarity

measures: PMI and PATH similarity.

We have perform an experimental comparison of themethodology using semantic and lexical similarity measures.

We have observed that the semantic similarity measures are

able to discover answers that with the lexical similarity measurecould not be discovered.

7/30/2019 amw_2013

18/21

Future Work18

As future work we would like to determine which question is

more suitable to be validated by a semantic measure, and

which one is better to be validated with a lexical measure.

Making this process automatic will improve the overall

precision of the methodology.

New methodologies are being implemented without the use of

Information retrieval systems, based on graph representation of

the knowledge (syntactic trees, semantic networks).

7/30/2019 amw_2013

19/21

Thank you!

7th Alberto Mendelzon

International Workshop on

Foundations of Data

Management

Contacts:

[email protected]

[email protected]

[email protected]

Benemrita Universidad Autnoma de Puebla

Facultad de Ciencias de la Computacin

7/30/2019 amw_2013

20/21

Obtained Results20

System 20111. jucs1106enen 0.57

2. jucs1107enen 0.473. in1102enen 0.37

5. SemAnsVal 0.34

Random baseline 0.20

42. swai1103enen 0.02

Se utiliza la siguiente medida para la evaluacin del sistema:

@1 =

( +

)

Donde:

= Nmero de preguntas

= Nmero de Preguntas contestadas correctamente

= Nmero de Preguntas sin contestar

7/30/2019 amw_2013

21/21

Similarity Score of Lucene21

Lucene combina el modelo booleano along with the vector

space model.

The vector space model uses the cosine similarity to rank the

documents.

Lucene refines this score with the equation:

amw_2013

Documents

Transcript of amw_2013