Semantic Search

1

Semantic SearchAndisheh Keikha

Ryerson University

Ebrahim BagheriRyerson University

May 7th 2014

2

Outline Search Process Query Processing Document Ranking Search Result Clustering and

Diversification What is the Goal Contributions

3

Search Process Simple search

Query: keywords Find documents which have those keywords Rank them based on query Result: ranked documents

4



5

Query Processing Query length

Correlated with performance in the search task Query is small collection of keywords Hard to find relevant documents only

based on 2,3 words Solution

Query reformulation Query expansion

6

Query Processing Query Expansion

Selection of new terms

Relevant documentsWordNet (Synonym, hyponym, …)

…Disambiguation

7

Query Processing Query Expansion

Selection of new terms Weighting those terms

8



9

Document Ranking Probabilistic Methods

What is the probability that this document is relevant to this query?

𝑃 (𝐿|𝐷 )=𝑃 (𝐷|𝐿 )𝑃 (𝐿)

𝑃 (𝐷)

The event that the document is

judged as relevant to query

The document description

10

Document Ranking Language Models

What is the probability of generating query Q, given document d, with language model Md.

𝑃 (𝑄|𝑀𝑑 )=∏𝑡∈𝑄

�̂�𝑚𝑙 (𝑡 ,𝑑)

Maximum likelihood

estimate of the probability

11



12

Search Result Clustering and Diversification

13



14

What is the Goal Searching on google

15

What is the Goal Searching on google

I want all of these searches show the same results, since they have same meaning, and

it is the intent of the user to know all of them, when searching for one.

16



Query Expansion Query Expansion(Tasks to Decide) Document Ranking

17

Contributions How?

New Semantic Query Expansion Method New Semantic Document Ranking Method

18




19

Query Expansion Example: “Gain Weight” Desirable keywords in expanded query:

“Gain, weight, muscle, mass, fat”

Gain weight

Muscle

Mass

Fat

What are these relations?

20

Query Expansion Digging in dbpedia and wikipedia

http://en.wikipedia.org/wiki/Weight_gain

http://dbpedia.org/page/Muscle http://dbpedia.org/page/Adipose_tissue

21




22

Query Expansion(Tasks to Decide)

How to map query phrases into Wikipedia components?

Which properties and their related entitles should be selected?

Can those properties be selected automatically for each phrase? Or should it be fixed for the whole algorithm?

If it’s automatic, what is the process?

23

Query Expansion(Tasks to Decide)

Is dbpedia and Wikipedia enough to decide, or should we use other ontologies?

How should we weight the extracted entities (terms, senses) in order to select the expanded query among them.

24




25

Document Ranking Are the documents annotated?

Yes• Rank documents using the extracted entitles from the

query expansion phase. No

• Rank the documents based on the semantics of the expanded query other than the terms or phrases.

• Define probabilities over senses other than terms in the query and documents.

26

Document Ranking Are the documents annotated?

Yes• Rank documents using the extracted entitles from the

query expansion phase. No

• Rank the documents based on the semantics of the expanded query other than the terms or phrases.

• Define probabilities over senses other than terms in the query and documents.

Documents are not annotated, so how?

27

Document Ranking Semantic Similarity between two non-

annotated documents ( the expanded query and the document) There are papers on using WordNet ontology,

with “topic specific PageRank algorithm”, for similarity of two sentences (phrase or word).

The application on information retrieval has not been seen yet.

28





Find the aspects of different algorithms which are more

beneficial in the information retrieval domain (two large

documents)

29





More reasonable is to apply the algorithm on dbpedia (instead of WordNet) in the entity domain

(instead of sense domain)

30

Document Ranking Applying a search result clustering and

diversification, based on the different semantics of the query.

31

Reference 1. B. Selvaretnam, M. B. (2011). Natural language technology and query expansion: issues,

state-of-the-art and perspectives. Journal of Intelligent Information Systems, 38(3), 709-740. 2. C. Carpineto, G. R. (2012). A Survey of Automatic Query Expansion in Information

Retrieval. ACM Computing Surveys, 44(1), 1-50. 3. Hiemstra, Djoerd. "A linguistically motivated probabilistic model of information retrieval."

In Research and advanced technology for digital libraries, pp. 569-584. Springer Berlin Heidelberg, 1998.

4. S. W. S. R. K. Sparck Jones, "A probabilistic model of information retrieval : development and comparative experiments Part 1," Information Processing & Management, vol. 36, no. 6, pp. 779-808, 2000.

5. Sparck Jones, Karen, Steve Walker, and Stephen E. Robertson. "A probabilistic model of information retrieval: development and comparative experiments: Part 2." Information Processing & Management 36.6 (2000): 809-840.

6. a. R. N. A. Di Marco, "Clustering and Diversifying Web Search Results with Graph-Based Word Sense Induction," Computational Linguistics, vol. 39, no. 3, pp. 709-754, 2013.

7. Di Marco, Antonio, and Roberto Navigli. "Clustering and diversifying web search results with graph-based word sense induction." Computational Linguistics 39, no. 3 (2013): 709-754.

8. Pilehvar, Mohammad Taher, David Jurgens, and Roberto Navigli. "Align, disambiguate and walk: A unified approach for measuring semantic similarity." InProceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013). 2013.

Semantic Search

Documents

Transcript of Semantic Search