Browsemap: Collaborative Filtering at LinkedIn
-
Upload
lili-wu -
Category
Technology
-
view
266 -
download
1
description
Transcript of Browsemap: Collaborative Filtering at LinkedIn
![Page 1: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/1.jpg)
Recruiting Solutions Recruiting Solutions Recruiting Solutions 1
Browsemap: Collaborative Filtering At LinkedIn
Lili Wu, Sam Shah, Sean Choi, Mitul Tiwari, Christian Posse
RSWeb 2014 with RecSys
![Page 2: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/2.jpg)
2
Agenda § Motivation § Architecture § Applications § Lessons Learned
![Page 3: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/3.jpg)
3
Profile Browsemap: People who viewed this profile also viewed… Count co-views
Collaborative filtering for member profile
![Page 4: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/4.jpg)
4
Collaborative filtering for job page
Job Browsemap: People who viewed this job also viewed… Count co-views
![Page 5: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/5.jpg)
5
company group portfolio
… many CF based recommenders
![Page 6: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/6.jpg)
6
• Many different entities
• Similar problems with different requirement • Fast product development cycle
• Hybrid recommender systems
• Handle LinkedIn data volume and traffic
Challenges
![Page 7: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/7.jpg)
7
Challenges
è Horizontal Platform
• Many different entities
• Similar problems with different requirement • Fast product development cycle
• Hybrid recommender systems
• Handle LinkedIn data volume and traffic
![Page 8: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/8.jpg)
8
Browsemap
Collaborative Filtering Platform at LinkedIn
![Page 9: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/9.jpg)
9
Browsemap Platform
• Scalability Ø Online/offline architecture Ø Hundreds of millions of entities, billions of
monthly page views • Browsemap Domain Specific Language (DSL)
Ø Code reuse through modular components Ø Flexible computation workflow construction
• Data are used by hybrid recommenders
![Page 10: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/10.jpg)
10 10 10
Browsemap Architecture
HDFS
User Activity
Data
Frontend Services
Results Queries
Hadoop
Browsemap Engine
Browsemap DSL Online
Query API
Key-value storage
Voldemort
![Page 11: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/11.jpg)
11 11 11
Browsemap Architecture
HDFS
Frontend Services
Results Queries
Hadoop
Browsemap Engine
Browsemap DSL Online
Query API
Key-value storage
Voldemort
User Activity
Data High Throughput
![Page 12: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/12.jpg)
12 12 12
Browsemap Architecture
HDFS
Frontend Services
Results Queries
Hadoop
Browsemap Engine
Browsemap DSL Online
Query API
Key-value storage
Voldemort
User Activity
Data Low Latency
![Page 13: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/13.jpg)
13
Browsemap Domain Specific Language (DSL)
Module Collection
Co-view counting
Spam User Filtering
Expired Job Filtering
Expired Job Filtering
Cold-start techniques
Co-view counting
…
Cold-start techniques
… Job browsemap
���
Job ��� Company
…
Spam User Filtering
Co-view counting
…
Cold-start techniques
…
Spam User Filtering
Company browsemap
![Page 14: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/14.jpg)
14
• Support all entity types • Adjust to each product requirement
• Scale
Recap
Voldemort
![Page 15: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/15.jpg)
15
Agenda ü Motivation ü Architecture § Applications § Lessons Learned
![Page 16: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/16.jpg)
16 16 16
Applications – CF based recommenders Profile Browsemap
Portfolio Browsemap
Job Browsemap Group Browsemap
Hiring Browsemap
Company Browsemap
Influencer Browsemap
![Page 17: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/17.jpg)
17 17 17
Applications – Hybrid Recommender Systems
Suggested Profile Update
Swee Lim
![Page 18: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/18.jpg)
18 18 18
Applications – Hybrid Recommender Systems
Suggested Profile Update
Goal: for each member,
find companies he may want to follow
![Page 19: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/19.jpg)
19 19 19
Applications – Hybrid Recommender Systems
Google Cisco Member followed companies
Linkedin, Facebook
Juniper, Arista Companies user may
be interested in
…
…
Member info: • Content-based features
title, industry, location, … • Collaborative filtering feature
Co-follow Browsemaps: People who follow this company also follow these companies
![Page 20: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/20.jpg)
20 20 20
Applications – Hybrid Recommender Systems
Question: For a company C, will member M like it?
Approach: Logistic regression Features:
member location company location 1 if yes, 0 if no
company is in the list of the co-follow browsemaps ? 1 if yes, 0 if no
…
![Page 21: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/21.jpg)
21 21 21
Applications – Hybrid Recommender Systems
Collaborative Filtering is important: • Surface implicit connection between companies • Based on Member’s preference
![Page 22: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/22.jpg)
22
Agenda ü Motivation ü Architecture ü Applications § Lessons Learned
![Page 23: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/23.jpg)
Lesson 1: Tall oaks grow from little acorns
23
![Page 24: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/24.jpg)
Lesson 1: Tall oaks grow from little acorns
24
![Page 25: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/25.jpg)
Lesson 1: Tall oaks grow from little acorns
25
![Page 26: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/26.jpg)
Lesson 1: Tall oaks grow from little acorns
26
A generic horizontal platform is essential
![Page 27: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/27.jpg)
Lesson 2: One hand washes the other
27
Job Browsemap
Similar Jobs
Collaborative filtering: “Follower audience”
Content based: “Leader audience”
![Page 28: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/28.jpg)
Lesson 3: You can’t get blood out of a stone
28
Job 1 Job 2 Job 3 (new)
Need to handle cold start problem
(view time)
merge
Leverage Browsing History Personalized Backfill
![Page 29: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/29.jpg)
Lesson 4: A chain is only as strong as its weakest link
29
CF: Relies solely on user activities Good data is crucial
§ Mistakes can be hard to detect / debug
§ Simple mistakes can have big impact e.g. “jobid” à “id”
§ Need prevention mechanism Ø Improve tracking Ø Unit test Ø Browsemap platform data-check :
input volume, coverage/metrics analysis
![Page 30: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/30.jpg)
Lesson 5: User experience matters
50% CTR
30
500% more applications
ª Put recommendations in user’s flow
![Page 31: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/31.jpg)
31
§ Collaborative filtering is important for LinkedIn
§ Browsemap is in production for 3+ years § Horizontal platform is crucial
Conclusion
![Page 32: Browsemap: Collaborative Filtering at LinkedIn](https://reader034.fdocuments.us/reader034/viewer/2022042518/557d819fd8b42a58788b4da0/html5/thumbnails/32.jpg)
32
§ Questions?
Thank you !