Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service...
Transcript of Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service...
![Page 1: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/1.jpg)
Ad Serving at Spotify ScaleA journey of incremental full stack overhaul
Kinshuk Mishra, Director of [email protected]@_kinshukmishra
![Page 2: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/2.jpg)
A lucky mistake
![Page 3: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/3.jpg)
Expected consequences
![Page 4: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/4.jpg)
Sarcastic empathy
![Page 5: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/5.jpg)
Some valuable feedback
![Page 6: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/6.jpg)
The unintended consequence
Artist engagement for exposed users went up
![Page 7: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/7.jpg)
The unintended consequence
Promising insights about content promotion use-case
![Page 8: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/8.jpg)
The unintended consequence
Confirmation that the ad server is a powerful
messaging platform
![Page 9: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/9.jpg)
Why should you care?
![Page 10: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/10.jpg)
Introduction
Ad technology
stack
Architecture Evolution
![Page 11: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/11.jpg)
Introduction
Ad technology
stack
Architecture Evolution
![Page 12: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/12.jpg)
What I do● Founded ads engineering team at Spotify in 2011
● Build all things ads engineering - team & software
● Major focus areas :
○ Ad delivery (Backend and Web)
○ Multi-platform native ads (Client Platform)
○ Ad performance (ML and Data)
![Page 13: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/13.jpg)
3 noteworthy things
![Page 14: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/14.jpg)
Full stack refactor
Evolution at scale
Pragmatic choices
![Page 15: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/15.jpg)
100,000,000+ MAU
![Page 16: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/16.jpg)
50,000,000+Subscribers
![Page 17: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/17.jpg)
30,000,000+ Songs
![Page 18: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/18.jpg)
2,000,000,000+ Playlists
![Page 19: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/19.jpg)
$5,000,000,000+ Revenue paid to rightsholders
![Page 20: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/20.jpg)
60 Markets
![Page 21: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/21.jpg)
Platform Ubiquity
![Page 22: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/22.jpg)
Freemium business model
![Page 23: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/23.jpg)
Ad
![Page 24: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/24.jpg)
![Page 25: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/25.jpg)
![Page 26: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/26.jpg)
![Page 27: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/27.jpg)
Introduction
Ad technology
stack
Architecture Evolution
![Page 28: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/28.jpg)
Beauty of Ad Server
Relevancy Pacing Unique View Sequence Optimization
![Page 29: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/29.jpg)
Complexity of Ad tech ecosystem
![Page 30: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/30.jpg)
In essence it is pretty simple
Client
User Profile database
Ad Server
Campaign Management Portal
Billing/Reporting
Ad campaign database
Data CollectionSystem
![Page 31: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/31.jpg)
Spotify Ads infrastructure in 2011
EdgeServiceDesktop
LogDelivery HDFS
User Profile
Batch
Basic Ad Server
CampaignManagement
Billing/Reporting
![Page 32: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/32.jpg)
Spotify Ads infrastructure in 2017
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
![Page 33: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/33.jpg)
Multi-platform clients
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
![Page 34: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/34.jpg)
Data collection
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
![Page 35: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/35.jpg)
Intelligence
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
![Page 36: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/36.jpg)
Ad Delivery
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
![Page 37: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/37.jpg)
Demand fulfillment
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
![Page 38: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/38.jpg)
Now you know too
Ad server is a powerful messaging platform
![Page 39: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/39.jpg)
Introduction
Ad technology
stack
Architecture Evolution
![Page 40: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/40.jpg)
Architecture overhaul is hard
● While keeping the business running
● While innovating on new products
● When you should have done it yesterday
![Page 41: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/41.jpg)
Why did Spotify evolve Ads architecture?
![Page 42: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/42.jpg)
Future needs●
● Growth in scale
● Emergence of new client platforms
● Cheap cloud computing
● New products to meet business objectives
● Technical debt
![Page 43: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/43.jpg)
The 3 stories
![Page 44: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/44.jpg)
Fixing the legacy mess
Story 1
![Page 45: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/45.jpg)
Original ad server designEdge Service
Routerhash(userid)
Ad server ring with partitions
Ad server instance
Memcache
Memcache
Memcache
Memcache
Campaign DB
User DBDesktop
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
![Page 46: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/46.jpg)
Problems
![Page 47: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/47.jpg)
Stateful service with faulty persistence
![Page 48: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/48.jpg)
Cache as a data store
![Page 49: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/49.jpg)
Service cluster as a hashed ring
![Page 50: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/50.jpg)
Ad decisioning in Client
![Page 51: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/51.jpg)
Batch Client-Server Calls
![Page 52: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/52.jpg)
Fix strategy
![Page 53: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/53.jpg)
Fix strategy tactic
![Page 54: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/54.jpg)
Isolate refactor to one system at a time
![Page 55: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/55.jpg)
The ad server transition
EdgeService
LogDelivery HDFS
User Profile
Batch
Smart Ad Server
CampaignManagement
Billing/Reporting
Ad Server Proxy
(routing) Basic Ad Server
Gradual transition from basic to smart ad serving
Desktop
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
![Page 56: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/56.jpg)
After the ad server transition
Proxy Service
LogDelivery HDFS
User Profile
Batch
CampaignManagement
Billing/Reporting
Smart Ad Server
Desktop
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
![Page 57: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/57.jpg)
Lean, mean and fast
Story 2
![Page 58: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/58.jpg)
Division of responsibilities
Desktop iOS
Android
Ads SDK
Desktop
Web
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
Ad decisioning
Ad fetch orchestration
Client context
Ad Trigger & Render
Before After
![Page 59: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/59.jpg)
Problems
![Page 60: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/60.jpg)
Thick Clients
![Page 61: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/61.jpg)
Logic duplication
![Page 62: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/62.jpg)
Tightly coupled monolith
![Page 63: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/63.jpg)
Fix strategy
![Page 64: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/64.jpg)
Reduce State Management
![Page 65: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/65.jpg)
Break monolith into services
![Page 66: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/66.jpg)
Isolate platform independent logic into a lib
![Page 67: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/67.jpg)
Fix tactic
![Page 68: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/68.jpg)
Design your systems to be master of one thing
![Page 69: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/69.jpg)
Remember division of responsibilities?
Desktop iOS
Android
Ads SDK
Desktop
Web
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
Ad decisioning
Ad fetch orchestration
Client context
Ad Trigger & Render
BAD GOOD
![Page 70: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/70.jpg)
Multiplatform Client design
iOS
Proxy Service
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
ModelingSelf-Serve Service
Creative Generation Payments
Billing/Reporting
![Page 71: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/71.jpg)
Knowledge is power, Unreliable data is your enemy
Story 3
![Page 72: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/72.jpg)
Event Stream Historical
ETL1 ETL2 ETL3
UserEntity1(attribute1, attribute2) UserEntity1(attribute1, attribute3) UserEntity1(attribute1, attribute3’)
![Page 73: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/73.jpg)
![Page 74: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/74.jpg)
Problems
![Page 75: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/75.jpg)
Duplicate, undiscoverable and fragmented datasets
![Page 76: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/76.jpg)
Metric inaccuracy
![Page 77: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/77.jpg)
Overloaded Data Infra
![Page 78: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/78.jpg)
Fix strategy
![Page 79: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/79.jpg)
Focus on reliable and timely log delivery
![Page 80: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/80.jpg)
![Page 81: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/81.jpg)
Data engineering with SLA
![Page 82: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/82.jpg)
Dataset canonicalization
![Page 83: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/83.jpg)
Some useful lessons learnt from architectural overhaul
![Page 84: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/84.jpg)
Test with minimal impact radius
![Page 85: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/85.jpg)
Mistakes are inevitable
![Page 86: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/86.jpg)
Speed up build decisions
![Page 87: Ad Serving at Spotify Scale - qconlondon.com · Spotify Ads infrastructure in 2017 iOS Edge Service Android Ads SDK Desktop Web Chromecast/ Playstation/ FireTV Ad Aggregation Service](https://reader033.fdocuments.us/reader033/viewer/2022042407/5f22559e80ef6b3eb66323a8/html5/thumbnails/87.jpg)
Think for tomorrow, Solve for today