Recommending the World’s KnowledgeApplication of Recommender Systems at Quora

Lei Yang (leiyang@quora.com)

Quora: Share and grow the world’s knowledge

Millions of questions & answers, millions of users, thousands of topics

Data and Scale

Topic Question

User

Answer

Actions

Lots of data relations

Complex network propagation effects

Complex topics network & semantics

RecommendationProblems

Recommending the digest email

Recommend 10 most interesting answers in an

email to get users to re-engage on Quora

● Requirements

○ Relevance

○ Freshness

○ Quality

● Challenges

○ Many candidates

○ Personalized frequency and send times

● Personalized LTR model

○ Predict probability of click on answers

shown

Recommending the digest email

● Candidates

○ User/topic follows

○ Globally popular/viral content

○ Collaborative filtering suggestions

■ Relevant writers

■ Relevant topics

■ Relevant answers

● Features

○ Text features

○ User follow graph

○ Historic data

○ ...

Recommending writers for a question

Recommend experts who are best qualified

to answer a question

● Ask to Answer (A2A)

● Given a questions and a viewer, predict

how “well-suited” a candidate user is for an

A2A suggestion

● Well-suited

○ viewer send a request &

○ candidate write a good answer

Recommending writers for a question

● Candidates

○ Users who have written at least an

answer on one of the topics of the

question

○ A number of filters

● Features

○ Candidate history features

○ Viewer history features

○ Candidate-viewer relevance

○ ...

Recommending the home feed

Present most interesting stories for a user at

a given time

● Stories: questions + answers

● Relevance-ordered vs time-ordered = big

gains in engagement

● Challenges

○ Many candidate stories

○ Real-time ranking

○ Objective function

Recommending the home feed

● Candidates

○ Different temporal windows

○ Multi-stage solution with different

“streams”

● Features

○ Quality of question/answer

○ Topics the user is interested in

or knows about

○ Users the user is following

○ What is trending/popular

○ ...

Recommending topics and users

Recommend new topics and users for

the user to follow, based on

● Topics you already follow

● Users you already follow

● Interactions with

questions/answers/users

● Topic-related features

● Social graph

● …..

Many more problems…

Related Questions Sessions

Trending

Topic feed

What We’ve Learned

Learning to recommend

● ML can often solve recommendation

problems

○ Select candidates

○ ML model to score each candidate

○ Top scored candidates are returned

● Workflow

○ Build data pipeline

○ Define training label

○ Choose your model and metric

○ Create features

ML models used at Quora

● Logistic Regression

● Elastic Nets

● Gradient Boosted Decision Trees

● Random Forests

● (Deep) Neural Networks

● LambdaMART

● Matrix Factorization

● LDA

● ...

Open source project -- QMF

Quora Matrix Factorization

https://github.com/quora/qmf

● Currently BPR and WALS

● Multithreaded implementation in

C++14

Challenges

● Personalization

● Explicit v.s. implicit user feedback

● Explore v.s. exploit and the filter bubble

● Business logic in your system

● Trade-offs between different optimization goals

● Explain the recommendations

● ...

Conclusions

Conclusions

● At Quora we have not only big, but also “rich” data

● Many interesting recommendation problems and unsolved challenges

● We use ML to solve many of our recommendation problems

● Our algorithms need to understand and optimize complex aspects such

as quality, relevance, or user expertise

We are hiring! www.quora.com/careers