Techniques Used in Modern Question-Answering Systems
description
Transcript of Techniques Used in Modern Question-Answering Systems
Techniques Used in Modern Techniques Used in Modern
Question-Answering SystemsQuestion-Answering Systems
Candidacy Exam
Elena FilatovaDecember 11, 2002
CommitteeLuis Gravano Columbia UniversityVasileios Hatzivassiloglou Department of Computer ScienceRebecca J. Passonneau
Present Present vsvs Past Research on QAPast Research on QA
Current systems– Mainly systems written for TREC conference
• factoid questions• short answers• huge text collections
Related systems– IR
• queries vs questions• return documents vs short answers
– Systems based on semantic representations (Lehnert): • questions about one text vs text collections • inference from semantic structure of a text vs searching for
an answer in the text– One type of output (NP) from a closed collection (Kupiec)
answer inference vs answer extraction
Lehner’t systemLehner’t system
John loved Mary but she didn’t want to marry him. One day, adragon stole Mary from the castle. John got on top of hishorse and killed the dragon. Mary agreed to marry him. Theylived happily ever after.
Q: Why did Mary agree to marry John? A: Because she was indebted to him
Problems stated:– right classification– dependency of answer inference procedure on the type of the
question
Current QA SystemsCurrent QA Systems
questionanalysis
question queryextracteddocuments
rules foranswer
list of
answers
InformationInformation ExtractionExtraction
•right query
•long text
•domain dependency
•predefined types of answers
Information Information RetrievalRetrieval
PlanPlan
• Classification
• Information (document) retrieval– Query formation
• Information extraction– Passage extraction
– Answer extraction
• Usage of answer redundancy on Web in QA
• QA for restricted domain
• Evaluation procedure for current QA systems and analysis of the performance
Classification and QAClassification and QA
questionanalysis
question queryextracteddocuments
rules foranswer
list of
answers
Theory of ClassificationTheory of Classification
Rosch et al: classification of basic objects
World is structured: real-world attributes do not occur independently of each other:
object_has(wings) => P(object_has(feathers)) > P(object_has(fur))
Each category (class) – set of attributes that are common for all the objects in the category
Types of categories• Superordinate – small amount of common attributes (furniture)• Subordinate – a lot of common attributes (floor lamp, desk lamp)• Basic – optimal amount of common attributes (lamp): basic
objects are the most inclusive categories which delineate the correlation structure of the environment
Though classification is a converging problem for objects, it is not possible to compile a list of all possible basic categories.
QA classification. QA classification. • Hierarchical/nonhierarchical classification
– Even if there exist hierarchy in the classification it can be represented as flat: detailed classes + other class
• Amount of types(MULDER – 3 types vs Webclopedia – over 140 types)
• Trade off between– Detailed classes for better answer extraction and– High precision in defining the classes
• Usage of semantics
• Usage of syntax– Most of syntactic parsers are built on corpora which do no
contain a lot of questions (WSJ) => need of additional corpus
• Attempts to automate this process– Maximum Entropy (Ittycheriah)– Classifiers (Li&Roth)
Why QA classification is important?Why QA classification is important?
Usage of question type for1. query construction
• question keywords + filtering mechanism (Harabagiu)
• synonyms and syn.sets from WordNet (Webclopedia)
in both cases there is no connection with possible answer space
• information retrieval (Agichtein, Berger)there is connection between question and answer spaces but these types do not give the type of the answer
2. searching for a correct answer in the passage extracted from a text
Logical FormsLogical Forms
•Syntactic analysis plus semantic => logical form•Mapping of question and potential answer LFs to find the best match (Harabagiu, Webclopedia)
Query formationQuery formation
• WordNet: synonyms, hyponyms, etc.• Morphology: verbal forms, plural/single nouns,
etc.• Knowledge of the domain (IBM’s system)• Statistical methods for connecting question and
answer spaces:– Agichtein: automatic acquisition of patterns that might be
good candidates for query expansion4 ‘types ‘ of question
– Berger: to facilitate query modification (expansion) each question term gets a set of answer terms
FQA: closed set of question-answer pairs
Information retrievalInformation retrieval
• Classical IR is the first step of QA• Vector-space model (calculation of similarity between terms
in the query and terms in the document)• IR techniques used in current QA systems are usually for
one database (either web or TREC collection)• Is it possible to apply Distributed IR techniques?
– domain restricted QA with extra knowledge about the text collection IBM system
– “splitting” one big collection of documents into smaller collections about specific topics
– it might require change in classification: type of the question might cause the changes in query formulation, document extraction process, answer extraction process
questionanalysis
question queryextracteddocuments
rules foranswer
list of
answers
InformationInformation ExtractionExtraction
Information Information RetrievalRetrieval
Passage extractionPassage extraction
• Passages of particular length (Cardie) + Vector representation for each passage
• Paragraphs or sentences
• Classical text excerpting– Each sentence is assigned a score– Retrieved passages are formed by taking the sentences with
the highest score
• Global-Local Processing (Salton)
• McCallum: passage extraction based not only on words but also on other features (e.g. syntactic constructions)
Information ExtractionInformation Extraction
• Domain dependency (Grishman)predefined set of attributes for the search specific for eachtopic, e.g. terrorism: victims, locations, perpetrators
• usually a lot of manually tagged data for training or
• texts divided into two groups: one topic – all other texts (Riloff)
in both cases division into topics is anecessary step which is not applicable to open domain
QA systems
What information can be extracted (IE)What information can be extracted (IE)
• Named entities (NE-tagging)– Numbers (incl. dates, ZIP codes, etc.)– Proper names (locations, people, etc.)– Other depending on the systemTREC8 – 80% questions asked for NEs
NEs might also support
• Correlated entity: mini-CV (Srihari)Who is Julian Hill?name; age; gender; position; affiliation; education
• General events (Srihari)Who did what to whom when
More complicated IE techniques lead QA back to AI approach
Answer Extraction
Three main techniques for answer extraction are based on:
1. syntactic-semantic tree dependencies: (Harabagiu, Webclopedia) LF of the question is mapped to LF of possible answers
2. surface patterns (Webclopedia) – <Name> (<Answer> -)– <Name> was born on <Answer>Good patterns require detailed classification: NUMBER vs DOB
3. text window – Cardie: query-dependant text summarization of text passages
with/without syntactic and semantic information
LF mapping classical MTsurface patterns example-based MTtext window statistical MT
Usage of WebUsage of Web (Answer redundancy)(Answer redundancy)
Multiple formulation of answer can useful for:1. IR stage: increased chances to find an answer that
matches query (Clarke, Brill)no need in searching for an exact formulation of the answer
2. IE stage: facilitation of answer extraction (Agichtein, Ravichandran, Brill)
create a list of patterns which might contain the answereither completely automatic (Agichtein) or using handwritten
filters based on question types and domain (Brill)
3. Answer validation (Magnini) correct answer redundancy
)(*)(
),(),(
AspPQspP
AspQspPAspQspPMI
Domain restricted applicationsDomain restricted applications
• FAQ (different from IR or QA)– match the input question with a list of already existing
questions– predefined output (according to the above question
matching)
• Rillof– 5 types of questions– answer extraction from a given text => no IR stage– always there is an answer (unique answer)
• IBM system– based on good knowledge of inner structure of IBM web-
site– Use of FAQ techniques
results are better than for open-domain QA systems
restricted-domain MTvs
open-domain MT
EvaluationEvaluationIR and IE have different evaluation measures
– IR: each document is marked either relevant/non-relevant recall + precision
– IE: gold standard answer key enumerates all acceptable responses recall + precision
– QA: mean reciprocal rank (MRR) • For each question:
receive score equal to reciprocal of rank of first correct response, or 0 if no correct response found.
• Overall system score is mean of individual question scores.
N
KiMRR
N
i 1
/1
N – amount of questions asked;Ki = rank of the correct answer or 0; RAR =1/ Ki
Future of QAFuture of QA
FROM TO
Questions: Simple facts
Questions: Complex: Uses Judgments Terms; Knowledge of User Context Needed
Answers:Simple Factoid
Answers found in Single Document
Answers:Search Mult. Sources;
Fusion of Info; Resolution of Conflicting Data;
Interpretations, Conclusions