Evaluating Semantic Search Query Approaches with Expert and Casual Users
-
Upload
kelbedweihy -
Category
Technology
-
view
350 -
download
0
description
Transcript of Evaluating Semantic Search Query Approaches with Expert and Casual Users
![Page 1: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/1.jpg)
Evaluating Semantic Search Query Approaches with Expert and Casual
Users
Khadija Elbedweihy, Stuart N. Wrigley and Fabio CiravegnaOAK Research Group,
Department of Computer Science, University of Sheffield, UK
![Page 2: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/2.jpg)
• Motivation• Research Question• Evaluation Design• Evaluation Setup• Findings• Conclusions
Outline
![Page 3: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/3.jpg)
Motivation – Semantic Search
• Wikipedia states that Semantic Search “seeks to improve search accuracy by understanding searcher intent and the contextual meaning of terms as they appear in the searchable dataspace, whether on the Web or within a closed system, to generate more relevant results”.
• Covers broad category of applications in Semantic Web:– Search engines (e.g., Swoogle, FalconS, Sindice)– Closed-domain query interfaces (e.g., AquaLog, Querix)– Open-domain query interfaces (e.g., PowerAqua)
![Page 4: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/4.jpg)
Motivation - Evaluations
• Evaluation of software is critical.
• Large-scale evaluations foster research and development.
• Semantic search evaluations (SemSearch, TREC ELC, QALD) focused on assessing retrieval performance.
Assessing usability of tools and user satisfaction is important in Semantic Search.
![Page 5: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/5.jpg)
Research Question
How do different types of users perceive the usability of different query approaches?
• Method- Assess usability and user satisfaction of:
* Free-NL, Controlled-NL, Form-based, Graph-based
- from the perspective of * expert users and casual users
![Page 6: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/6.jpg)
Query Approaches
Free-NLNatural language queries
What is the capital of Alabama? Submit
capital Alabama Submit
Controlled-NLSpecific vocabulary
Which state has Submitrivercapitallakemountainaany
Form-based Visualize the search space
Graph-based Visualize the search space
![Page 7: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/7.jpg)
Evaluation Design: Dataset
• Mooney Natural Language Learning Data- simple and well-known domain (geography)- used by other studies within the search community- questions already available (877 NL questions)
• Geography Dataset:– Concepts: State, City, Lake, Mountain, Capital, River, etc– Properties: population of state, length of river, etc– Relations linking concepts: State ‘hasCity’ City
![Page 8: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/8.jpg)
Evaluation Design: Data Collected
• Objective data:1) Input time 2) Number of attempts3) Success rate
• Subjective data, collected using: 1) Questionnaires (e.g., System Usability Scale ‘SUS’)2) Ranking of the tools (w.r.t: system, query approach,
results content, results presentation)3) Observations
![Page 9: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/9.jpg)
Evaluation Setup
• 20 subjects – 10 casual users, 10 expert users– 12 females, 8 males
• Within-subjects: allows direct comparison.
• Randomising tool order: normalize learning or tiredness effects.
• Randomising question order: normalize learning effects.
![Page 10: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/10.jpg)
Results
• Evaluated tools:
– Free-NL: NLP-Reduce
– Controlled-NL: Ginseng
– Form-based: K-Search
– Graph- based: • Semantic-Crystal (Graph-based 1)• Affective Graphs (Graph-based 2)
![Page 11: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/11.jpg)
QUERY APPROACH
![Page 12: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/12.jpg)
Results for expert users
• Expert users prefer graph- and form- based approach.• View-based allow more complex queries than NL-based.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Graph-based1Graph-based2Form-basedControlled-NLFree-NL
Que
ry L
angu
age
Ran
k
Best
Worst
![Page 13: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/13.jpg)
Results for casual users
• Casual users prefer form-based query approach.• Required less input time than graph-based approach.
Que
ry L
angu
age
Ran
k
Best
Worst0
102030405060708090
100
Inpu
t Tim
e (S
ec)
00.10.20.30.40.50.60.70.80.9
1
Graph-based1Graph-based2Form-basedControlled-NLFree-NL
![Page 14: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/14.jpg)
ONTOLOGY VISUALIZATION
![Page 15: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/15.jpg)
Results for expert users• Visualizing the entire ontology supports query formulation
– Semantic Crystal: shows the entire ontology.– Affective Graphs: shows selected concepts & relations.
![Page 16: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/16.jpg)
Results for casual users
• Not showing ontology more complex for casual users: – Semantic Crystal receiving higher scores.– Affective Graphs perceived as complex and difficult to use
• 50% of the users found it to increase complexity and difficulty
![Page 17: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/17.jpg)
CONTROLLED-NL APPROACH
![Page 18: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/18.jpg)
Results for expert users
• Controlled-NL very restrictive for expert users (least-liked) • Highest query input time
00.10.20.30.40.50.60.70.80.9
1
Graph-based1Graph-based2Form-basedControlled-NLFree-NL
Que
ry L
angu
age
Ran
k
Best
Worst0
20
40
60
80
100
120
Inpu
t Tim
e (S
ec)
![Page 19: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/19.jpg)
Results for casual users
• Controlled-NL provided most support for casual users.
• Users’ positive feedback for controlled-NL:– allow only correct queries (50%)– suggestions and guidance to formulate queries (40%)
Example: Although Ginseng is limited to specific vocabulary, I knew that I will get answers once I can do the query because it only allows the correct ones and thus I didn't keep trying a lot of queries that I wasn't sure about.
![Page 20: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/20.jpg)
RESULTS INDEPENDENT OF USER TYPE
![Page 21: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/21.jpg)
Free-NL approach
+ simplest and most natural- suffer from habitability problem.
• Feedback: “I have to guess the right words”– Example: `run through’ with `river’ but not `traverse’.
• NLP-Reduce:– lowest success rate: 20%– highest number of attempts: 4.2
![Page 22: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/22.jpg)
Negation
• Tell me which rivers do not traverse the state with the capital Nashville?
Expert Users Casual Users0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Graph-based1Graph-based2Form-basedControlled-NLFree-NL
Ans
wer
Fou
nd R
ate
![Page 23: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/23.jpg)
Negation
Tell me which states does the river Mississippi does not traverse.
• “Closed world assumption (CWA): presumption that what is not currently known to be true is false”.
<Mississippi, traverse, Louisiana>
• “Open world assumption (OWA): assumption that the truth-value of a statement is independent of whether or not it is known by any single observer or agent to be true”.<Mississippi, not_traverse, Alabama>
![Page 24: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/24.jpg)
Formal Query
• Formal Query (e.g., SPARQL)
![Page 25: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/25.jpg)
Formal Query
• Benefit of showing formal query depends on user type.
• Formal query perceived by:
– Casual users: not understandable and confusing
– Expert users: increased confidence
Also, performing direct changes to the formal queryincreased the expressiveness of the query language.
![Page 26: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/26.jpg)
Results presentation
• Results presentation and format affected usability and user satisfaction.– Unless users are very familiar with the data, presenting URIs
alone is not very helpful.
– Example: A query for rivers returns one of the answers:http://www.mooney.net/geo#tennesse2
![Page 27: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/27.jpg)
Results Content
• Results should be augmented with associated information to provide a `richer’ user experience.
• Users feedback:– Maybe a `mouse over' function to show more
information.– Perhaps related information with the results.– Results very limited, would be good to have more
context.
![Page 28: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/28.jpg)
CONCLUSIONS
![Page 29: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/29.jpg)
Research Question & Approach
How do different types of users perceive the usability of different query approaches?
- Assess usability and user satisfaction of:* Free-NL, Controlled-NL, Form-based, Graph-
based
- from the perspective of * expert users and casual users
![Page 30: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/30.jpg)
Conclusions
Expert Users• Graph-based most preferred - Intuitive - Support complex queries
• Controlled-NL least preferred - Very restrictive. - Limited expressiveness
• Prefer flexibility of free-NL• Formal query provides confidence - Ability to change query increases
expressiveness.
Casual Users• Form-based mid-point - Allow more complex queries than
NL. - Easier than graph-based - Faster than graph-based• Controlled-NL most supportive - Only valid queries: Confidence - Vocabulary suggestions: guidance
• Formal Query not understandable and confusing.
• Users want search results to be augmented with more information to have a better understanding of the answers.
![Page 31: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/31.jpg)
Recommendations
Cater to both expert and casual users:
• Hybridized query approach: Combine a view-based approach (visualize search space) with a NL-input feature (balance difficulty and speed) while including optional suggestions for the NL input (provide guidance).
• Results Content: Augment results with ‘extra’ and ‘related’ information.– extra information: for ‘State’: capital, area, population.– related information: for ‘State’: rivers, lakes, mountains.
![Page 32: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/32.jpg)
Limitations & Future work
• Limitation: Small size of the dataset.
• Assess learnability of different query approaches.
• Assess how interaction with the search tools affect the information seeking process: usefulness.
– Use questions with an overall goal and compare users' knowledge before and after the search task.
![Page 33: Evaluating Semantic Search Query Approaches with Expert and Casual Users](https://reader035.fdocuments.us/reader035/viewer/2022081512/5564a7d7d8b42afd4f8b5572/html5/thumbnails/33.jpg)
Questions
Questions?