Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale
-
Upload
kinshuk-mishra -
Category
Technology
-
view
133 -
download
0
Transcript of Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale
![Page 1: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/1.jpg)
Ad Serving at Spotify ScaleA journey of incremental full stack overhaul
Kinshuk Mishra, Director of [email protected]@_kinshukmishra
![Page 2: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/2.jpg)
A lucky mistake
![Page 3: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/3.jpg)
Expected consequences
![Page 4: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/4.jpg)
Sarcastic empathy
![Page 5: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/5.jpg)
Some valuable feedback
![Page 6: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/6.jpg)
The unintended consequence
Artist engagement for exposed users went up
![Page 7: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/7.jpg)
The unintended consequence
Promising insights about content promotion use-case
![Page 8: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/8.jpg)
The unintended consequence
Confirmation that the ad server is a powerful
messaging platform
![Page 9: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/9.jpg)
Why should you care?
![Page 10: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/10.jpg)
Introduction
Ad technology
stack
Architecture Evolution
![Page 11: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/11.jpg)
Introduction
Ad technology
stack
Architecture Evolution
![Page 12: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/12.jpg)
What I do● Founded ads engineering team at Spotify in 2011
● Build all things ads engineering - team & software
● Major focus areas :
○ Ad delivery (Backend and Web)
○ Multi-platform native ads (Client Platform)
○ Ad performance (ML and Data)
![Page 13: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/13.jpg)
3 noteworthy things
![Page 14: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/14.jpg)
Full stack refactor
Evolution at scale
Pragmatic choices
![Page 15: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/15.jpg)
100,000,000+ MAU
![Page 16: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/16.jpg)
50,000,000+Subscribers
![Page 17: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/17.jpg)
30,000,000+ Songs
![Page 18: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/18.jpg)
2,000,000,000+ Playlists
![Page 19: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/19.jpg)
$5,000,000,000+ Revenue paid to rightsholders
![Page 20: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/20.jpg)
60 Markets
![Page 21: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/21.jpg)
Platform Ubiquity
![Page 22: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/22.jpg)
Freemium business model
![Page 23: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/23.jpg)
Ad
![Page 24: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/24.jpg)
![Page 25: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/25.jpg)
![Page 26: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/26.jpg)
![Page 27: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/27.jpg)
Introduction
Ad technology
stack
Architecture Evolution
![Page 28: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/28.jpg)
Beauty of Ad Server
Relevancy Pacing Unique View Sequence Optimization
![Page 29: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/29.jpg)
Complexity of Ad tech ecosystem
![Page 30: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/30.jpg)
In essence it is pretty simple
Client
User Profile database
Ad Server
Campaign Management Portal
Billing/Reporting
Ad campaign database
Data CollectionSystem
![Page 31: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/31.jpg)
Spotify Ads infrastructure in 2011
EdgeServiceDesktop
LogDelivery HDFS
User Profile
Batch
Basic Ad Server
CampaignManagement
Billing/Reporting
![Page 32: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/32.jpg)
Spotify Ads infrastructure in 2017
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
![Page 33: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/33.jpg)
Multi-platform clients
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
![Page 34: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/34.jpg)
Data collection
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
![Page 35: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/35.jpg)
Intelligence
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
![Page 36: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/36.jpg)
Ad Delivery
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
![Page 37: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/37.jpg)
Demand fulfillment
iOS
EdgeService
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
Optimization
ModelingSelf-Serve Portal
Creative Generation Payments
Billing/Reporting
![Page 38: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/38.jpg)
Now you know too
Ad server is a powerful messaging platform
![Page 39: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/39.jpg)
Introduction
Ad technology
stack
Architecture Evolution
![Page 40: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/40.jpg)
Architecture overhaul is hard
● While keeping the business running
● While innovating on new products
● When you should have done it yesterday
![Page 41: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/41.jpg)
Why did Spotify evolve Ads architecture?
![Page 42: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/42.jpg)
Future needs●
● Growth in scale
● Emergence of new client platforms
● Cheap cloud computing
● New products to meet business objectives
● Technical debt
![Page 43: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/43.jpg)
The 3 stories
![Page 44: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/44.jpg)
Fixing the legacy mess
Story 1
![Page 45: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/45.jpg)
Original ad server designEdge Service
Routerhash(userid)
Ad server ring with partitions
Ad server instance
Memcache
Memcache
Memcache
Memcache
Campaign DB
User DBDesktop
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
![Page 46: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/46.jpg)
Problems
![Page 47: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/47.jpg)
Stateful service with faulty persistence
![Page 48: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/48.jpg)
Cache as a data store
![Page 49: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/49.jpg)
Service cluster as a hashed ring
![Page 50: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/50.jpg)
Ad decisioning in Client
![Page 51: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/51.jpg)
Batch Client-Server Calls
![Page 52: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/52.jpg)
Fix strategy
![Page 53: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/53.jpg)
Fix strategy tactic
![Page 54: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/54.jpg)
Isolate refactor to one system at a time
![Page 55: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/55.jpg)
The ad server transition
EdgeService
LogDelivery HDFS
User Profile
Batch
Smart Ad Server
CampaignManagement
Billing/Reporting
Ad Server Proxy
(routing) Basic Ad Server
Gradual transition from basic to smart ad serving
Desktop
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
![Page 56: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/56.jpg)
After the ad server transition
Proxy Service
LogDelivery HDFS
User Profile
Batch
CampaignManagement
Billing/Reporting
Smart Ad Server
Desktop
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
![Page 57: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/57.jpg)
Lean, mean and fast
Story 2
![Page 58: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/58.jpg)
Division of responsibilities
Desktop iOS
Android
Ads SDK
Desktop
Web
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
Ad decisioning
Ad fetch orchestration
Client context
Ad Trigger & Render
Before After
![Page 59: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/59.jpg)
Problems
![Page 60: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/60.jpg)
Thick Clients
![Page 61: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/61.jpg)
Logic duplication
![Page 62: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/62.jpg)
Tightly coupled monolith
![Page 63: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/63.jpg)
Fix strategy
![Page 64: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/64.jpg)
Reduce State Management
![Page 65: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/65.jpg)
Break monolith into services
![Page 66: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/66.jpg)
Isolate platform independent logic into a lib
![Page 67: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/67.jpg)
Fix tactic
![Page 68: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/68.jpg)
Design your systems to be master of one thing
![Page 69: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/69.jpg)
Remember division of responsibilities?
Desktop iOS
Android
Ads SDK
Desktop
Web
Rendering Ad trigger decisioning
Ads Ranking
Ads Caching
Ad batching & fetch communication
Ad decisioning
Ad fetch orchestration
Client context
Ad Trigger & Render
BAD GOOD
![Page 70: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/70.jpg)
Multiplatform Client design
iOS
Proxy Service
Android
Ads SDK
Desktop
Web
Chromecast/Playstation/
FireTV
Ad Aggregation
Service
LogDelivery GCS
User Profile
Targeting Service
DMP
Stream Batch
Ad Server
Decision Delivery Ad Exchanges
CampaignManagement
ModelingSelf-Serve Service
Creative Generation Payments
Billing/Reporting
![Page 71: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/71.jpg)
Knowledge is power, Unreliable data is your enemy
Story 3
![Page 72: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/72.jpg)
Event Stream Historical
ETL1 ETL2 ETL3
UserEntity1(attribute1, attribute2) UserEntity1(attribute1, attribute3) UserEntity1(attribute1, attribute3’)
![Page 73: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/73.jpg)
![Page 74: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/74.jpg)
Problems
![Page 75: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/75.jpg)
Duplicate, undiscoverable and fragmented datasets
![Page 76: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/76.jpg)
Metric inaccuracy
![Page 77: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/77.jpg)
Overloaded Data Infra
![Page 78: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/78.jpg)
Fix strategy
![Page 79: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/79.jpg)
Focus on reliable and timely log delivery
![Page 80: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/80.jpg)
![Page 81: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/81.jpg)
Data engineering with SLA
![Page 82: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/82.jpg)
Dataset canonicalization
![Page 83: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/83.jpg)
Some useful lessons learnt from architectural overhaul
![Page 84: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/84.jpg)
Test with minimal impact radius
![Page 85: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/85.jpg)
Mistakes are inevitable
![Page 86: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/86.jpg)
Speed up build decisions
![Page 87: Qcon London 2017 - Architecture overhaul - Ad serving @ Spotify scale](https://reader035.fdocuments.us/reader035/viewer/2022062523/58cf313e1a28ab00168b591b/html5/thumbnails/87.jpg)
Think for tomorrow, Solve for today