Recommending for the World
-
Upload
moustaki -
Category
Engineering
-
view
825 -
download
0
Transcript of Recommending for the World
![Page 1: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/1.jpg)
![Page 2: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/2.jpg)
Recommending for the World
Yves Raimond (@moustaki)
03/16
Research/Engineering Manager
Search & Recommendations Algorithm Engineering
Netflix
![Page 3: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/3.jpg)
Some background
![Page 4: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/4.jpg)
![Page 5: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/5.jpg)
![Page 6: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/6.jpg)
![Page 7: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/7.jpg)
● > 75M members
● > 190 countries
● > 3.7B hours of content streamed every
month
● > 1000 device types
● 36% of peak US downstream traffic
Netflix scale
![Page 8: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/8.jpg)
Recommendations @ Netflix
![Page 9: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/9.jpg)
Goal
Help members find content to watch and enjoy
to maximize satisfaction and retention
![Page 10: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/10.jpg)
![Page 11: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/11.jpg)
![Page 12: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/12.jpg)
![Page 13: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/13.jpg)
![Page 14: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/14.jpg)
![Page 15: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/15.jpg)
![Page 16: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/16.jpg)
![Page 17: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/17.jpg)
![Page 18: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/18.jpg)
![Page 19: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/19.jpg)
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪ …
Models & Algorithms
![Page 21: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/21.jpg)
Going global● How do we make sure all these
algorithms are ready to work on a
global scale?
● Led us to investigate many
challenges, leading to many rollouts
of new algorithms, over the last year
○ Tech blog post
○ Company blog post
![Page 22: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/22.jpg)
Challenge 1: Uneven Video Availability
![Page 23: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/23.jpg)
US
FR
![Page 24: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/24.jpg)
US
FR
1,000 users
100 users
![Page 25: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/25.jpg)
0
...
...Co-occurrences
![Page 26: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/26.jpg)
! =
?????
![Page 27: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/27.jpg)
R ≈ UM
![Page 28: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/28.jpg)
! =
![Page 29: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/29.jpg)
What would have happened if the two videos were available to the same
members?
![Page 30: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/30.jpg)
US
FR
1,000 users
100 users
100,000 users
10 users
![Page 31: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/31.jpg)
≈
![Page 32: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/32.jpg)
US
FR
2016-01-01 2016-01-02
Newly available
![Page 33: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/33.jpg)
What would have happened if the two videos were available to the same
members for the same amount of time?
![Page 34: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/34.jpg)
Challenge 2: Cultural Awareness
![Page 35: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/35.jpg)
Two questions
![Page 36: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/36.jpg)
1) Similar users, in two different countries. Should they get similar
recommendations?
![Page 37: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/37.jpg)
2) Overall, should recommendations be different for users in Japan vs users
in Argentina? What about new users?
![Page 38: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/38.jpg)
Regional modelsGroup countries into regions, and train
individual models on each region.
Pros
● Easy!
● Catalog can be constrained to be
relatively uniform
● Solves question 2
Cons
● Doesn’t solve question 1
● How to define groupings?
● Algorithms x A/B model variants x
regions
● Biggest country in the region will
dominate
● Sparsity
![Page 39: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/39.jpg)
Sparsity and global models
Only a small fraction of users from all countries
would be interested in these titles. Models trained
locally perform poorly -- lack of data.
Pooling data from all countries discovers a
worldwide community of interest, making
recommendations better for these users.
![Page 40: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/40.jpg)
Global communities - Anime
![Page 41: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/41.jpg)
Global communities - Bollywood
![Page 42: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/42.jpg)
Local taste vs personal taste● Personal taste benefits from global algorithms
○ Taste patterns travel globally
● Local taste still needs to be taken into account in order to solve 2)
● Incorporate signals and priors capturing local taste patterns (e.g. country and
language)
![Page 43: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/43.jpg)
![Page 44: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/44.jpg)
Challenge 3: Language
![Page 45: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/45.jpg)
Instant search● Ranking entities for partial queries
● Optimizing for the minimum number of interactions needed to find something
● Different languages involve very different interaction patterns
● How to automatically detect and adapt to such patterns in newly introduced
languages?
![Page 46: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/46.jpg)
Hangul alphabet, 3 syllables but requires 7 (2 + 3 + 2) interactions
![Page 47: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/47.jpg)
One interaction
![Page 48: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/48.jpg)
Language & Recommendations
≈ +
US US/AU FR
?
![Page 49: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/49.jpg)
Challenge 4: Does it even work?
![Page 50: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/50.jpg)
Tracking quality● Objective: build algorithms that work equally well for all our members
● Looking at global metrics might hide issues with small subsets of members
● How to identify sub-optimality for a subset of our members?
○ Language, country, device, …
○ Slicing on all dimensions lead to sparsity and noisiness
○ Automatically grouping observations for the purpose of automatically detecting outliers
● Metrics, instrumentation and monitoring
○ Detect problems
○ Highlight areas of improvement
![Page 51: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/51.jpg)
Conclusion
![Page 52: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/52.jpg)
● Catalog differences, cultural awareness, language and metrics
● Worldwide communities of interest for better recommendations
○ Thinking about global actually led us to test and release better algorithms
○ But also need to capture signals and priors related to cultural preferences
● Quickly finding entities in any language
● Detecting issues at a finer grain
● … Still a lot of work to do!
○ Better global algorithms… (Now that we have data)
○ Better cultural/language awareness
○ Better user and item cold start
○ Reactiveness
○ Better algorithms for anomaly detection
Conclusion
![Page 53: Recommending for the World](https://reader030.fdocuments.us/reader030/viewer/2022032711/58f9adaf760da3da068b9add/html5/thumbnails/53.jpg)
Questions?