SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud...
Transcript of SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud...
![Page 1: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/1.jpg)
Bora Tunca
Availability Objectives of
SoundCloud’s Microservices
![Page 2: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/2.jpg)
SoundCloud is the largest online audio distribution
platform
![Page 3: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/3.jpg)
● Backend developer
● @SoundCloud ~3 years
● Core engineering
Bora Tunca @grandbora
![Page 4: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/4.jpg)
Microservices
![Page 5: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/5.jpg)
![Page 6: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/6.jpg)
~140 services
~140 engineers
![Page 7: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/7.jpg)
How do we improve availability
in a microservices architecture?
![Page 8: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/8.jpg)
![Page 9: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/9.jpg)
![Page 10: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/10.jpg)
MS
API
MS
MS
MS
MS MS
![Page 11: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/11.jpg)
Post Mortem Meetingshelp us spot “risky” services
![Page 12: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/12.jpg)
Service Level Objectivesdefine an availability objective for each microservice
![Page 13: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/13.jpg)
If the availability of a service
drops below its objective,
we raise the red flag.
![Page 14: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/14.jpg)
Service Level Objectives
● SLO infrastructure
● Usage analyses
![Page 15: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/15.jpg)
SLO Infrastructure
![Page 16: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/16.jpg)
Availability
● Error Rate Threshold
● Latency Threshold 99th percentile
![Page 17: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/17.jpg)
SLO Parameters
SLO_ERROR_RATE_THRESHOLD=0.0001SLO_LATENCY_P99_THRESHOLD_SECONDS=0.500SLO_AVAILABILITY_TARGET=0.99999
![Page 18: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/18.jpg)
SLO Metrics
● TotalTransactions
● FailedTransactions
● SlowTransactions
![Page 19: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/19.jpg)
Availability
(FailedTransactions / TotalTransactions) <
&&
(SlowTransactions / TotalTransactions) < 0.0199th percentile
error ratethreshold
![Page 20: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/20.jpg)
SLO Wrapper
MS
Req Res
Prometheus
Metrics
Services Directory
Availability
![Page 21: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/21.jpg)
![Page 22: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/22.jpg)
Usage Analyses
![Page 23: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/23.jpg)
API API API
Microservices
![Page 24: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/24.jpg)
API API API
MS MS MS MS MS MS MS
MS MS MS MS MS MS MS MS
MS MS MS MS MS MS MS
MS MS MS MS MS MS
Edge
Value Added
Data
![Page 25: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/25.jpg)
MSMS
API API API
MS MS MS MS MS
MS MS MS MS MS MS MS MS
MS MS MS MS MS MS MS
MS MS MS MS MS MS
Edge
Value Added
Data
![Page 26: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/26.jpg)
Graceful Degradation
Tracks
API
99.995%95%
Stats
![Page 27: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/27.jpg)
Learnings
![Page 28: SoundCloud’s Microservices Availability Objectives of · SoundCloud’s Microservices.SoundCloud is the largest online audio distribution platform. Bora Tunca @grandbora Backend](https://reader033.fdocuments.us/reader033/viewer/2022060401/5f0e14c17e708231d43d855b/html5/thumbnails/28.jpg)
● Without a formal process identifying risky services is hard
● Don’t over-engineer your services
● Let the data guide your engineering decisions