Recommendations and Discoveryat StumbleUpon

Sumanth Kolar,

Director, Engineering

@_5K

StumbleUpon’s Mission

Help users find content they did not expect to find

Be the best way to discover new and interesting things from across

the Web.

How StumbleUpon works

1. Register 2. Tell us your interests 3. Start Stumbling and rating web pages

We use your interests and behavior to recommend new content for you!

There is a ongoing shift from search to discovery

Discovery is very different from search

Discovery at StumbleUpon Search

Serendipitous Intent driven

One at a time List of articles

Never repeats Always repeats

Constantly adapting Fixed results

Tailored for you Impersonal

StumbleUpon

StumbleUpon Overview

Discovery Crawled

Ingestion Pipeline

Sampling Pass?

Rec Engine

Users Automated

URL Index

Yes

1

2

3

What are the key challenges to good recommendations?

Pillars of good recommendations

Understand who the user is and what he is interested in.

Separate good content from the bad.

Learn from your recommendations.

Explore various techniques for matching users to content.

Pillars of good recommendations

Understand who the user is and what he is interested in.

Separate good content from the bad.

Learn from your recommendations.

Explore various techniques for matching users to content.

User self reports topics of interest

Part of the sign up flow…

User’s Interest Graph

Food/Cooking User

Cars

VintageCars

Italian Recipes

Continually Enhance a User’s Interest Graph

Analyze user’s StumbleUpon history to expand on interest preferences:

• Add/remove topics• Follow/block particular domains

Continually Enhance a User’s Interest Graph

Leverage social network data:

• Find friends & people to follow

• Find content trending in your social circles

• Find additional interests

Continually Enhance a User’s Interest Graph

Mine internal StumbleUpon rating and sharing data to suggest other stumblers, topics.

Enhanced Interest Graph

Food/Cooking

User

Cars

VintageCars

Italian Recipes

nasa.gov

1x.com

News

Friends

Trending

Pillars of good recommendations

Understand who the user is and what he is interested in.

Separate good content from the bad.

Learn from your recommendations.

Explore various techniques for matching users to content.

On average hundreds of URLs are ingested into the

StumbleUpon pipeline every minute.

• Sampling key goals:

1. Determine which URLs to sample and which to skip completely

2. Examine sampling results to identify good URLs

• URL features used when sampling:

• Known domain performance(ratings, timespent)• Content related features (#images, #ads, url length etc)• User features of the discoverer (spammer vs trusted user)

Sampling

Vote

Yes

Yes

NoYes

Recommend

Classifier based on User Feedback (Timespent, Ratings)

Yes

Random Forest

Webpage

Recommendations at StumbleUpon: Sampling

Rating Timespent

Good 35sec

Good 22sec

Bad 15sec

Good 45sec

Good 14sec

Good 28sec

• Users who thumb-up good content and thumb-down bad content

• For example– Joe DiMaggio – Baseball– Julia Child- Food/Cooking– Da Vinci- Art and Architecture

• Ratings from Experts are more trustworthy and earn more weight.

Leveraging In-Network Experts

Expe

rtRecommendations at StumbleUpon: Experts

Non

Exp

ert

Pillars of good recommendations

Understand who the user is and what he is interested in.

Separate good content from the bad.

Learn from your recommendations.

Explore various techniques for matching users to content.

Challenge: User expectations are different

“I LOVE cars!”-Anonymous Stumbler

“Me too!”-Another Stumbler

• Find users who like content similar to the content you do

• Signals can be ratings, time spent, interests, etc.

• Use the content they’ve liked

Like-Minded Users

NeuroscienceAstronomySpace ExplorationComedy Movies

Astronomy Space ExplorationPhysics Classic Movies

Vintage CarsAction moviesAstronomyRobotics

Science

Space

Movies

Cars

PLSI based like-minded

Total Pairwise Similarity Calculations

= 50K users * 5 million users * 1K features

= 250 Trillion Probabilistic Latent Semantic Index (PLSI)

based similarity over 500 trillion calculations PLSI based similarity framework computes in

less than an hour

Like-Minded Users: Challenges Scaling

Food/Cooking

User

Cars

VintageCars

Italian Recipes

nasa.gov

1x.com

News

Experts Friends

Trending

Grow User’s Interest Graph: Implicit + Explicit

LikemindedUsers

Different methods perform differently for different users at different times

User 1 User 2 User 3 User 4 User 50%

25%

50%

75%

100%

TrendingFollowBias domainsExpertsNewsLike-minded

Recommendation context

Pillars of good recommendations

Understand who the user is and what he is interested in.

Separate good content from the bad.

Learn from your recommendations.

Explore various techniques for matching users to content.

Two Main Signals from Recommendation

Rating Time Spent

Both present numerous challenges . . .

# Ra

tings

Time

Users rate more during their initial experience

Why is this happening?

Ratings: volume decay

Images

T5 sec

?

Text

T4 sec

• Ratings are sparse• < 10% of recommendations have explicit ratings.

• Using time spent decide whether the stumble was skipped• Timespent on videos is longer than images. • Solution: Estimate p(Like | Timespent)

• Model based on user, content patterns

Video

T3 sec

?

Images

T2 sec

Time Spent

Video

T1 sec

Installed plugin

Stumble Bar

Mobile / Tablets

Challenges: Time spent on different devices

5th percentile time spent per stumble

Med

ian

time

spen

t per

stum

ble

Pillars of good recommendations

Understand who the user is and what he is interested in.

Separate good content from the bad.

Learn from your recommendations.

Explore various techniques for matching users to content.

How do we know we are doing a good job?

Extensive A/B Testing

AB Tests on metrics such as session length, retention, rating behavior etc

Measurable Improvements In Rec Quality

12/1/0

8 0:00

2/1/0

9 0:00

4/1/0

9 0:00

6/1/0

9 0:00

8/1/0

9 0:00

10/1/0

9 0:00

12/1/0

9 0:00

2/1/1

0 0:00

4/1/1

0 0:00

6/1/1

0 0:00

8/1/1

0 0:00

10/1/1

0 0:00

12/1/1

0 0:00

2/1/1

1 0:00

4/1/1

1 0:00

6/1/1

1 0:00

8/1/1

1 0:00

10/1/1

1 0:00

12/1/1

1 0:00

2/1/1

2 0:00

4/1/1

2 0:00

6/1/1

2 0:00

8/1/1

2 0:000

2

4

6

8

10

12

14

16

R² = 0.737311794306772

Normalized Likes vs Dislikes

Recent Months

+111% improvement!

• Dupe detection• Anti-spam• News• Topic classification• Metrics, quality analysis• Trending• Search• User biases, mood• Many more…

Many other interesting problems…

We are HIRING !!!