Post on 08-Jan-2017
News REcommendation Evaluation Lab (NewsREEL)
Results
Frank Hopfgartner, Benjamin Kille, Andreas Lommatzsch, Martha Larson, Torben Brodt, Jonas Seiler
Registrations
2
Task 1: Online Evaluation
• Provide recommendations for visitors of the news portals of plista’s customers
• Ten portals (local news, sports, business, technology)
• Communication via Open Recommendation Platform (ORP)
Dat
a
• Benchmark own performance with other participants and baseline algorithms during three pre-defined evaluation windows
• Best algorithms determined in final evaluation period
• Standard evaluation metricsEva
luat
ion
Recommend news articles in real-time
Task 2: Offline Evaluation
• Traffic and content updates of nine German-language news content provider websites
• Traffic: Reading article, clicking on recommendations
• Updates: adding and updating news articlesD
ata
• Simulation of data stream using Idomaar framework
• Participants have to predict interactions with data stream
• Quality measured by the ratio of successful predictions by the total number of predictionsE
valu
atio
n
Predict interactions in a simulated data stream
Evaluation Schedule (Task 1)
First evaluation window
Second evaluation window
Third evaluation window
CTR (Task 1)
9
Response rate vs number of requests (Task 1)
10
Switched off throughout the day
Error rate (Task 1)
11
Click-through rate
12
Congratulations to „xyz-2.0“ for achieving highest
CTR with 1.17%
Availability/response rate
13
Congratulations to „flumingsparkteam“ for
achieving highest response rate with
99.64%
Presentations