Overview of the TREC 2016 Open Search track: Academic Search Edition
-
Upload
krisztianbalog -
Category
Science
-
view
250 -
download
1
Transcript of Overview of the TREC 2016 Open Search track: Academic Search Edition
Overview of the TREC 2016 Open Search track
Academic search edi>on
Krisz>an Balog University of Stavanger
@krisz'anbalog
25th Text REtrieval Conference (TREC 2016) | Gaithersburg, 2016
Anne Schuth Blendle
@anneschuth
USERS
TREC assessors "unsuspec>ng users"
VS
THE DATA DIVIDE
INDUSTRY ACADEMIA
WHAT IS OPEN SEARCH?
Open Search is a new evalua1on paradigm for IR. The experimenta1on pla=orm is an exis1ng search engine. Researchers have the opportunity to replace components of this search engine and evaluate these components using interac1ons with real, "unsuspec1ng" users of this search engine.
WHY OPEN SEARCH?
• Because it opens up the possibility for people outside search organiza>ons to do meaningful IR research
• Meaningful includes • Real users of an actual search system
• Access to the same data
RESEARCH QUESTIONS
• How does online evalua>on compare to offline, Cranfield style, evalua>on?
• Would systems be ranked differently?
• How stable are such system rankings?
• How much interac>on volume is required to be able to reach reliable conclusions about system behavior?
• How many queries are needed?
• How many query impressions are needed? • To which degree does it maSer how query impressions are
distributed over queries?
RESEARCH QUESTIONS (2)
• Should systems be trained or op>mized differently when the objec>ve is online performance?
• What are ques>ons that cannot be answered about a specific task (e.g., scien>fic literature search) using offline evalua>on?
• How much risk do search engines that serve as experimental plaYorm take?
• How can this risk be controlled while s>ll be able to experiment?
LIVING LABS METHODOLOGY
KEY IDEAS
• An API orchestrates all the data exchange between sites (live search engines) and par>cipants
• Focus on frequent (head) queries • Enough traffic on them for experimenta>on
• Par>cipants generate rankings offline and upload these to the API
• Eliminates real->me requirement
• Freedom in choice of tools and environment
K. Balog, L. Kelly, and A. Schuth. Head First: Living Labs for Ad-hoc Search Evalua=on. CIKM'14
OVERVIEW
experimental systems
users live site
API
K. Balog, L. Kelly, and A. Schuth. Head First: Living Labs for Ad-hoc Search Evalua=on. CIKM'14
METHODOLOGY (1)
experimental systemusers live site
API
• Sites make queries, candidate documents (items), historical search and click data available through the API
METHODOLOGY (2)
experimental systemusers live site
API
• Rankings are generated (offline) for each query and uploaded to the API
METHODOLOGY (3)
experimental system
API
• When any of the test queries is fired on the live site, it requests an experimental ranking from the API and interleaves it with that of the produc>on system
query
interleaved ranking
query
experimental ranking
INTERLEAVING
doc 1
doc 2
doc 3
doc 4
doc 5
doc 2
doc 4
doc 7
doc 1
doc 3
system A system Bdoc 1
doc 2
doc 4
doc 3
doc 7
interleaved list
A>BInference:
• Experimental ranking is interleaved with the produc>on ranking
• Needs 1-2 order of magnitudes data than A/B tes>ng (also, it is within subject as opposed to between subject design)
INTERLEAVING
doc 1
doc 2
doc 3
doc 4
doc 5
doc 1
doc 2
doc 3
doc 7
doc 4
system A system Bdoc 1
doc 2
doc 3
doc 4
doc 7
interleaved list
Inference: tie
• Team Drac Interleaving • No preferences are inferred from common prefix of A and B
METHODOLOGY (4)
• Par>cipants get detailed feedback on user interac>ons (clicks)
experimental systemusers live site
API
METHODOLOGY (5)
• Evalua>on measure:
• where the number of “wins” and “losses” is against the produc>on system, aggregated over a period of >me
• An Outcome of > 0.5 means bea>ng the produc>on system
Outcome =#Wins
#Wins + #Losses
WHAT IS IN IT FOR PARTICIPANTS?
• Access to privileged (search and click-through) data • Opportunity to test IR systems with real,
unsuspec>ng users in a live seing • Not the same as crowdsourcing!
• Con>nuous evalua>on is possible, not limited to yearly evalua>on cycle
KNOWN ISSUES
• Head queries only • Considerable por>on of traffic, but only popular info needs
• Lack of context • No knowledge of the searcher’s loca>on, previous searches, etc.
• No real->me feedback • API provides detailed feedback, but it’s not immediate
• Limited control • Experimenta>on is limited to single searches, where results are interleaved
with those of the produc>on system; no control over the en>re result list
• Ul>mate measure of success • Search is only a means to an end, it is not the ul>mate goal
KNOWN ISSUES
• Head queries only • Considerable por>on of traffic, but only popular info needs
• Lack of context • No knowledge of the searcher’s loca>on, previous searches, etc.
• No real->me feedback • API provides detailed feedback, but it’s not immediate
• Limited control • Experimenta>on is limited to single searches, where results are interleaved
with those of the produc>on system; no control over the en>re result list
• Ul>mate measure of success • Search is only a means to an end, it is not the ul>mate goal
Come to the planning session tomorrow!
OPEN SEARCH 2016:ACADEMIC SEARCH
ACADEMIC SEARCH
• Interes>ng domain • Need seman>c matching to overcome vocabulary mismatch
• Different en>ty types (papers, authors, orgs, conferences, etc.)
• Beyond document ranking: ranking en>>es, recommending related literature, etc.
• This year • Single task: ad hoc scien>fic literature search
• Three academic search engines
TRACK ORGANIZATION
• Mul>ple evalua>on rounds • Round #1: Jun 1 - Jul 15
• Round #2: Aug 1 - Sep 15
• Round #3: Oct 1 - Nov 15 (official TREC round)
• Train/test queries • For train queries feedback is available individual impressions
• For test queries only aggregated feedback is available (and only acer the end of each evalua>on period)
• Single submission per team
EXAMPLE RANKING
Ranking in TREC format
Ranking to be uploaded to the API
R-q2 Q0 R-d70 1 0.9 MyRunID R-q2 Q0 R-d72 2 0.8 MyRunID R-q2 Q0 R-d74 3 0.7 MyRunID R-q2 Q0 R-d75 4 0.6 MyRunID R-q2 Q0 R-d1270 5 0.5 MyRunID R-q2 Q0 R-d73 6 0.4 MyRunID R-q2 Q0 R-d1271 7 0.3 MyRunID R-q2 Q0 R-d71 8 0.2 MyRunID ...
{ 'doclist': [ {'docid': 'R-d70'}, {'docid': 'R-d72'}, {'docid': 'R-d74'}, {'docid': 'R-d75'}, {'docid': 'R-d1270'}, {'docid': 'R-d73'}, {'docid': 'R-d1271'}, {'docid': 'R-d71'} ], 'qid': 'R-q2', 'runid': 'MyRunID' }
SITES AND RESULTS
CITESEERX
• Main focus is on Computer and Informa>on Sci. • hSp://citeseerx.ist.psu.edu/
• Queries • 107 test + 100 training for Rounds #1 and #2
• 700 addi>onal test queries for Round #3
• Documents • Title
• Full document text (extracted from PDF)
CITESEERX RESULTSROUNDS #1 & #2
TeamRound #1 Round #2
Outcome #Wins #Losses #Ties #Impr. Outcome #Wins #Losses #Ties #Impr.
UDel-IRL 0.86 6 1 2 9
webis 0.75 3 1 1 5
UWM 0.67 2 1 3 6
IAPLab 0.73 8 3 1 12 0.60 3 2 1 6
BJUT 0.33 3 6 1 10 0.60 6 4 1 11
QU 0.50 3 3 3 9 0.50 3 3 1 7
Gesis 0.67 4 2 3 9 0.50 2 2 1 5
OpnSearch_404 0.00 0 0 1 1 0.50 4 4 1 9
KarMat 0.60 3 2 2 7 0.44 4 5 0 9
CITESEERX RESULTSROUND #3 (=OFFICIAL RANKING)
TeamRound #3
Outcome #Wins #Losses #Ties #Impr.
Gesis 0.71 5 2 0 7
OpnSearch_404 0.71 5 2 2 9
KarMat 0.67 4 2 0 6
UWM 0.67 2 1 0 3
IAPLab 0.63 5 3 2 10
BJUT 0.55 44 36 15 95
UDel-IRL 0.54 33 28 14 75
webis 0.50 20 20 11 51
DaiictIr2 0.38 6 10 5 21
QU 0.25 2 6 2 10
SSOAR
• Social Science Open Access Repository • hSp://www.ssoar.info/
• Queries • 74 test + 57 training for Rounds #1 and #2
• 988 addi>onal test queries for Round #3
• Documents • Title, abstract, author(s), various metadata field (subject, type,
year, etc.)
SSOAR RESULTSROUNDS #1 & #2
TeamRound #1 Round #2
Outcome #Wins #Losses #Ties #Impr. Outcome #Wins #Losses #Ties #Impr.
Gesis 1.00 1 0 461 462 1.00 1 0 96 97
UWM 0.60 3 2 473 478 1.00 1 0 94 95
QU 0.33 1 2 472 475 0.50 1 1 112 114
webis 0.50 1 1 88 90
KarMat 0.80 4 1 504 509 0.00 0 2 84 86
IAPLab 0.00 0 0 148 148 0.00 0 0 24 24
UDel-IRL 0.00 0 0 11 11 0.00 0 1 84 85
OpnSearch_404 0.00 0 0 2 2 0.00 0 0 2 2
SSOAR RESULTSROUND #3 (=OFFICIAL RANKING)
TeamRound #3
Outcome #Wins #Losses #Ties #Impr.
IAPLab 1.00 1 0 185 186
Gesis 0.61 11 7 5136 5154
webis 0.50 2 2 1640 1644
UDel-IRL 0.11 2 17 4723 4742
UWM 0.00 0 1 176 177
QU 0.00 0 0 179 179
KarMat 0.00 0 0 185 185
OpnSearch_404 0.00 0 0 6 6
MICROSOFT ACADEMIC SEARCH
• Research service developed by MSR • hSp://academic.research.microsoc.com/
• Queries • 480 test queries
• Documents • Title, abstract, URL
• En>ty ID in the Microsoc Academic Search Knowledge Graph
MICROSOFT ACADEMIC SEARCHEVALUATION METHODOLOGY
• Offline evalua>on, performed by Microsoc • Head queries (139)
• Binary relevance, inferred from historical click data
• Tradi>onal rank-based evalua>on (MAP)
• Tail queries (235) • Side-by-side evalua>on against a baseline produc>on system
• Top 10 results decorated with Bing cap>ons
• Rela>ve ranking of systems w.r.t. the baseline
MICROSOFT ACADEMIC SEARCHRESULTS
Team MAP
UDEL-IRL 0.60
BJUT 0.56
webis 0.52*
Team Rank
webis #1
UDEL-IRL #2
BJUT #3
* Significantly different from UDEL-IRL and BJUT
Head queries(click-based evalua>on)
Tail queries(side-by-side evalua>on)
SUMMARY
• Ad hoc scien>fic literature search • 3 academic search engines, 10 par>cipants
• TREC OS 2017 • Academic search domain
• Addi>onal sites
• One more subtask (recommending literature; ranking people, conferences, etc.)
• Mul>ple runs per team
• Consider a second use-case • Product search, contextual adver>sing, news recommenda>on, ...
CONTRIBUTORS
• API development and maintenance • Peter Dekker
• CiteSeerX • Po-Yu Chuang, Jian Wu, C. Lee Giles
• SSOAR • Narges Tavakolpoursaleh, Philipp Schaer
• MS Academic Search • Kuansan Wang, Tobias Hassmann, Artem Churkin, Ioana
Varsandan, Roland DiSel