Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken...
-
Upload
sherman-mills -
Category
Documents
-
view
215 -
download
0
Transcript of Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken...
![Page 1: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/1.jpg)
Co-funded by the European Union
The QALL-ME Benchmark:The QALL-ME Benchmark:a Multilingual Resource of a Multilingual Resource of
Annotated Spoken Annotated Spoken Requests for Question Requests for Question
AnsweringAnsweringE. Cabrio, M. Kouylekov, B. Magnini, M. Negri (FBK-Irst)
L. Hasler, C. Orasan, (University of Wolverhampton)
D. Tomas, J.L. Vicedo (University of Alicante)
G. Neumann, C. Weber (DFKI)
![Page 2: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/2.jpg)
Outline:Outline:o Motivations and goalso QALL-ME Projecto QALL-ME Benchmark
oData collectiono Translation into Englisho Speech Acts AnnotationoQuestion Answering Annotationo Annotation of relations
o Conclusion and Future Work
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 3: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/3.jpg)
Context: the Qall-me Context: the Qall-me projectproject
QALL-ME (Question Answering Learning QALL-ME (Question Answering Learning technologies in a multiLingual and multiModal technologies in a multiLingual and multiModal Environment): Environment): an EU-funded project aiming at the realization of a shared and distributed infrastructure for Question Answering systems on mobile devices (e.g. mobile phones).
LREC - 28-30 May 2008 - Marrakech (Morocco)
SMSSMS
INPUT OUTPUT
SMSSMS
MMSMMS
VOICE
TEXT
TEXT
VOICE
VIDEO
DIGITAL
ASSISTANT
![Page 4: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/4.jpg)
QALL-ME detailsQALL-ME detailso Reference: FP6 IST-033860o Contract Type: STREP o Start date: October 1st, 2006o Duration: 36 monthso Project Funding: 2.82 M euros
http://qallme.fbk.eu
LREC - 28-30 May 2008 - Marrakech (Morocco)
FBK- Irst, Italy Comdata S,p.A., Italy
DFKI, Germany Ubiest S.p.A., Italy
University of Alicante, Spain Waycom S.r.l., Italy
University of Wolverhampton, UK
![Page 5: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/5.jpg)
MotivationsMotivations
o Providing a dataset of requests dataset of requests beyond factoid questions (e.g. verification, procedural)
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 6: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/6.jpg)
Motivation: beyond Motivation: beyond factoid…factoid…
has Venezia hotel a restaurant is there a toll free number for the INAIL office in via Gazzoletti in Trento
VERIFICATIONVERIFICATION
where is the INAIL office and how can I get there
how can I get to the pharmacy De Gerloni of Trento
PROCEDURALPROCEDURAL
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 7: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/7.jpg)
MotivationsMotivations
o Providing a dataset of requests dataset of requests beyond factoid questions (e.g. verification, procedural)
o Investigating domain dependent domain dependent vs domain independent annotation schema (Qall-me project domain: cultural events in a town).
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 8: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/8.jpg)
ChallengesChallengeso Context aware QA
o What can I see tonight at cinema o Where is the nearest pharmacy
o Persistent vs dynamic information
o Multiple sources (database, newspaper, web)
LREC - 28-30 May 2008 - Marrakech (Morocco) 8
![Page 9: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/9.jpg)
Challenges related to Challenges related to eventsevents
o Context aware QA
o What can I see tonight at cinema (in Trento)
o Where is the nearest pharmacy (to piazza Duomo)
o Persistent vs dynamic information
o Multiple sources (database, newspaper, web)
LREC - 28-30 May 2008 - Marrakech (Morocco) 9
![Page 10: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/10.jpg)
MotivationsMotivations
o Providing a dataset of requests dataset of requests beyond factoid questions (e.g. verification, procedural)
o Investigating domain dependent domain dependent vs domain independent annotation schema (Qall-me project domain: cultural events in a town).
o Experimenting the impact of QA annotations QA annotations (e.g. EAT) on spoken requests (speech vs QAQA).
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 11: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/11.jpg)
QA annotationQA annotation
may I know where the ice stadium of Trento is located and at what time it opens
LREC - 28-30 May 2008 - Marrakech (Morocco)
LOCATION
DATE
Expected Answer Type :
![Page 12: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/12.jpg)
MotivationsMotivations
o Providing a dataset of requests dataset of requests beyond factoid questions (e.g. verification, procedural)
o Investigating domain dependent domain dependent vs domain independent annotation schema (Qall-me project domain: cultural events in a town).
o Experimenting the impact of QA annotations QA annotations (e.g. EAT) on spoken requests (speech vs QAQA).
o Investigating of the portability of semantic annotation portability of semantic annotation through languages.
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 13: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/13.jpg)
Portability of annotationsPortability of annotations
may I know where the ice stadium of Trento is located
LREC - 28-30 May 2008 - Marrakech (Morocco)
Expected Answer Type: LOCATION
potrei sapere dov’è lo stadiodel ghiaccio di Trento
puedo saber donde esta el estadio de hielo de Trento
ich möchte wissen wo das Eisstadium von Trento ist
![Page 14: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/14.jpg)
Data collectionData collection
o 14645 questions in four different languages:
ITALIAN, ENGLISH, GERMAN, SPANISHITALIAN, ENGLISH, GERMAN, SPANISHo Domain: cultural events in a town
Acquisition:Acquisition:Every speaker performs 30 questions, based on 15 scenarios :o Using a graphical interface, for each scenario is first generated a
spontaneous request and then a written one (previously predefined)o A telephone was used to acquire questions.
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 15: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/15.jpg)
Data collectionData collection# words # utterances avg. len
(words)
ITALIANread utterances 25715 2290 11.2
spontaneous utterances 33492 2374 14.1
total utterances 59207 4664 12.7
SPANISHread utterances 25919 2250 11.52
spontaneous utterances 26327 2250 11.70
total utterances 52246 4500 11.61
ENGLISHread utterances 26626 2215 12
spontaneous utterances 36000 2286 15.8
total utterances 62626 4501 13.9
GERMANread utterances 10990 903 12.17
spontaneous utterances 985 77 12.79
total utterances 11975 980 12.22
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 16: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/16.jpg)
Data acquisition featuresData acquisition features
# speakers males females non-native
tot. speech duration
avg. utt. dur
IT 161 68 93 12 9h20’ 7”
SP 150 109 41 8 16h4’ 5.14”
EN 113 46 63 21 7h35’ 6.1”
GER 9 4 5 2 1h21’ 4.9”
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 17: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/17.jpg)
TranscriptionTranscription
All the audio files acquired from a speaker were joined together and orthographically transcribed using the tool Transcriber. (http://trans.souceforge.net)
Being domain-restricted, our scenarios led sometimes to the same utterance (matching word sequence). However, the number of repetitions is actually small.
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 18: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/18.jpg)
Translation into EnglishTranslation into EnglishTranslation made by simulating the real situation of an
English speaker visiting a foreign city.
E.g.
o what is the address of museo dell'aeronautica Gianni Caproni
Future work: Future work: all data collected for one language translated into the other three languages
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 19: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/19.jpg)
Annotation of speech actsAnnotation of speech acts
o As a starting point for further analyses, it is important to separate within an utterance (each speaker’s turn) what has to be interpreted as the actual request from what does not need an answer.
hallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks
LREC - 28-30 May 2008 - Marrakech (Morocco)
from the QALL-ME benchmark
![Page 20: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/20.jpg)
Annotation of speech actsAnnotation of speech acts
o As a starting point for further analyses, it is important to separate within an utterance (each speaker’s turn) what has to be interpreted as the actual request from what does not need an answer.
to greet hallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks
LREC - 28-30 May 2008 - Marrakech (Morocco)
from the QALL-ME benchmark
![Page 21: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/21.jpg)
Annotation of speech actsAnnotation of speech acts
o As a starting point for further analyses, it is important to separate within an utterance (each speaker’s turn) what has to be interpreted as the actual request from what does not need an answer.
to contextualisehallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks
LREC - 28-30 May 2008 - Marrakech (Morocco)
from the QALL-ME benchmark
![Page 22: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/22.jpg)
Annotation of speech actsAnnotation of speech acts
o As a starting point for further analyses, it is important to separate within an utterance (each speaker’s turn) what has to be interpreted as the actual request from what does not need an answer.
hallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks
to ask
LREC - 28-30 May 2008 - Marrakech (Morocco)
from the QALL-ME benchmark
![Page 23: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/23.jpg)
Annotation of speech actsAnnotation of speech acts
o As a starting point for further analyses, it is important to separate within an utterance (each speaker’s turn) what has to be interpreted as the actual request from what does not need an answer.
hallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks
to thank
LREC - 28-30 May 2008 - Marrakech (Morocco)
from the QALL-ME benchmark
![Page 24: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/24.jpg)
Annotation of speech actsAnnotation of speech acts
REQUESTS
DIRECT:
• wh-questions
• Introduced by:
Could you tell me…
May I know…
• pronounced with ascendant intonation
INDIRECT:
•requests formulated in indirect or implicit ways
NON REQUESTSAll the utterances used by the speaker to introduce himself, to contextualize himself or his request in time and space, to thank, to greet.
ASSERT
GREETINGS
THANKS
OTHER
For our purposes, we used CLaRK, an XML Based System for Corpora Development (http://www.bultreebank.org/clark/index.html).
LREC - 28-30 May 2008 - Marrakech (Morocco)
UTTERANCE
![Page 25: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/25.jpg)
Agreement (speech acts)Agreement (speech acts)
Inter-annotator agreement (calculated on 1000 randomly picked sentences) for ITALIAN:
Dice coefficient = 2C/(A+B)C=number of common annotations
A , B =number of annotations provided by the first and the second annotator
Overall agreement 96.1%
ASSERT 85.5%
DIRECT 97.88%
INDIRECT 97.33%
OTHER 76.47%
THANKS 98.51%
GREETINGS 99.49%
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 26: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/26.jpg)
Expected Answer Type Expected Answer Type
For EAT annotation we propose the following scheme:
EAT
PROCEDURAL VERIFICATION FACTOID DEFINITION/DESCRIPTION
• DOMAIN-INDEPENDENT (SEKINE’S ENE HIERARCHY)• DOMAIN-SPECIFIC (QALL-ME ONTOLOGY)
LREC - 28-30 May 2008 - Marrakech (Morocco)
Extracted from Graesser’s (1988) taxonomy
![Page 27: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/27.jpg)
Sekine’s ENE vs Qall-me Sekine’s ENE vs Qall-me ont.ont.
LREC - Marrakech (Morocco ) , 28-30 May 2008 27
what is the restaurant in via Brennero in Trento
EAT
Sekine’s ENE hierarchy
Qall-me ontology
![Page 28: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/28.jpg)
Sekine’s ENE vs Qall-me Sekine’s ENE vs Qall-me ont.ont.
LREC - Marrakech (Morocco ) , 28-30 May 2008 28
can you give me the name of the pharmacy in piazza Pasi 20 in Trento
EAT
Sekine’s ENE hierarchy
Qall-me ontology
![Page 29: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/29.jpg)
Annotation of RelationsAnnotation of Relationso Relations among entities: convey and complete the context
in which a specific request has to be interpreted
LREC - 28-30 May 2008 - Marrakech (Morocco)
At what time is the movie il grande capo beginning tomorrow afternoon at Vittoria cinema
o Rel1 (MOVIE, DATE)o Rel2 (MOVIE, STARTINGHOUR)o Rel3 (MOVIE, CINEMA)
o 10% of the Italian questions (referring to Cinema/Movie domain) have been annotated with the 12 relations holding in such domain (Qall-me ontology).
![Page 30: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/30.jpg)
Status of the benchmarkStatus of the benchmark
audio transcr. translat. speech acts
EAT Sekine
EAT ontology
ITALIAN X X X X X X
SPANISH X X X X X in progress
ENGLISH X X --- in progress in progress in progress
GERMAN in progress in progress in progress in progress in progress in progress
LREC - 28-30 May 2008 - Marrakech (Morocco)
Present situation and tentative scheduling:
The QALL-ME benchmark is being made incrementally available at the project website (http://qallme.fbk.eu)
![Page 31: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/31.jpg)
Future workFuture work
Additional annotation layers will be Additional annotation layers will be considered:considered:
o Focus of the questiono Multiwordso Named Entitieso Normalized Temporal Expressionso …
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 32: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/32.jpg)
ConclusionsConclusions
o QALL-ME benchmarkQALL-ME benchmark: multilingual resource (for Italian, Spanish, English and German) of annotated spoken requests in the tourism domain.
o Beyond factoido Context aware QA and dynamic changeso QA annotation on spoken requestso Portability of semantic annotation
o Reference resource, useful to train and test ML based QA systems
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 33: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/33.jpg)
Thank youThank you{cabrio, kouylekov, magnini, negri}@fbk.eu
{L.Hasler, c.orasan}@wlv.ac.uk
{tomas, vicedo}@disi.ua.es
{neumann, cowe01}@dfki.de
Project website: http://qallme.fbk.eu
LREC - 28-30 May 2008 - Marrakech (Morocco)
![Page 34: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/34.jpg)
Acquisition scenariosAcquisition scenarios
34
SubDomainDesiredOutput
MandatoryItems
OptionalItems
![Page 35: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/35.jpg)
Example from the corpus Example from the corpus
<question id="3118">
<text>buongiorno chiamo da Trento avrei bisogno dell'indirizzo del teatro Auditorium per un concerto di Salvatore Accardo del 17 gennaio 2007</text>
<analysis>
<greetings>buongiorno</greetings>
<assert>chiamo da Trento</assert>
<indirect>avrei bisogno dell'indirizzo del teatro Auditorium per un concerto di Salvatore Accardo del 17 gennaio 2007</indirect></analysis>
<reference>
<ref>
<speaker>spk075_27mar07comd_it_sid023</speaker>
<turn>6</turn>
<originalString>
buongiorno chiamo da Trento ho [mmm] avrei bisogno dell'indirizzo del teatro Auditorium per un [eh] concerto di Salvatore Accardo del 17 gennaio 2007 [b]
</originalString>
</ref>
</reference>
<translation>good morning I am calling from Trento I would like to know the address of Auditorium theatre for Salvatore Accardo's concert on 17th January 2007</translation>
</question>
35
![Page 36: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/36.jpg)
Expected Answer Type (1)Expected Answer Type (1)
The semantic category associated to the desired answer, chosen out of a predefined set of labels (e.g. PERSON, LOCATION, DATE).
o How many colors are in the Italian flag
QUANTITY
o Where is the Uffizi museum
LOCATION
Most QA systems described in literature heavily rely on EAT information, at least in the Answer Extraction phase, to narrow the potential answer candidate search space.
36
![Page 37: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/37.jpg)
Example from the corpusExample from the corpus
37
What are the address and the telephone number of Venezia hotel in Trento
<eats><EAT type= “FACTOID” sekine=“ADDRESS_OTHER” qallme=“PostalAddress” eaq= “one”/>
<EAT type= “FACTOID” sekine=“ADDRESS_OTHER” qallme=“Contact” eaq= “one”/></eats>
![Page 38: Co-funded by the European Union The QALL-ME Benchmark: a Multilingual Resource of Annotated Spoken Requests for Question Answering E. Cabrio, M. Kouylekov,](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649f435503460f94c64332/html5/thumbnails/38.jpg)
Expected Answer Expected Answer QuantifierQuantifier
Attribute of the EAT that specifies the number of expected items in the answer.
o I would like to know the three colors of the Italian flag
o which movies are on tonight at Multisala Modena
all
The possible values are: one, at least one, all, n.
38