Microservices at Mercari
-
Upload
google-cloud-platform-japan -
Category
Technology
-
view
10.883 -
download
0
Transcript of Microservices at Mercari
![Page 1: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/1.jpg)
Microservices at MercariCurrent status and challenges
![Page 2: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/2.jpg)
Taichi Nakashima (@deeeet/@tcnksm)
SRE at Mercari, automation obsessed, gopher
![Page 3: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/3.jpg)
SRE mission at Mercari
● To ensure a reliable service that is enjoyable to use at anytime● Takes care of all engineering apart from new service development
○ Performance improvement, automation, security etc
![Page 4: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/4.jpg)
Current Mercari architecture
nginx
HTTP
API API API
MySQL MySQL
solr solr solr
Cache
Simple 3 tiler + α architecture
Single code base
![Page 5: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/5.jpg)
Current Mercari architecture
Same architectureIn 3 region
JPUS
UK
![Page 6: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/6.jpg)
Positive
● A central ops team (SRE) can efficiently handle
![Page 7: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/7.jpg)
Challenges
![Page 8: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/8.jpg)
Challenges
nginx
HTTP
API API API
MySQL MySQL
solr solr solr
Cache
Simple 3 tiler + α architecture
Single code base
![Page 9: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/9.jpg)
Challenges
nginx
HTTP
API API API
MySQL MySQL
solr solr solr
Cache
Simple 3 tiler + α architecture
Monolith?
![Page 10: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/10.jpg)
Challenges
● Code is too huge/complex to understand ● Team is too large to efficiently work on shared code base● Communication overhead is too large ● Velocity (development cycle) is stalled...
![Page 11: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/11.jpg)
Microservices
![Page 12: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/12.jpg)
Microservices?
● Architectural and organizational approach to software development○ To speed up deployment cycles○ Foster innovation and ownership ○ Improve maitainability and scalability
![Page 13: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/13.jpg)
Microservices?
$ cat inside.txt | cut -f 1 -d ' ' | sort | uniq -c | sort -nr
![Page 14: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/14.jpg)
Microservices
● Do one thing well ○ Unix philosophy○ One function in one service, not multiple functions in one service
● Decentralized Governance○ Each team has ownership on each service
● Independent○ Each service can be changed, upgraded, or replaced independently
● Polyglot○ Right framework and tool for each domain
![Page 15: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/15.jpg)
Goal
● Software Engineer○ Without velocity stalled, rather make feature improvement iteration speed fast ○ -> Provide great features to customers faster
● SRE ○ Provide automated platform for microservice ○ Give some responsibility (e.g., deployment, debug) to software engineering○ -> Focus on more SRE related software engineering task
![Page 16: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/16.jpg)
Team
@deeeet @spensnova @babarot
![Page 17: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/17.jpg)
State of microservices in US
![Page 18: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/18.jpg)
Microservices architecture in US
Mercari API
HTTP
![Page 19: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/19.jpg)
Microservices architecture in US
Gateway API
Mercari API
HTTP
HTTP
![Page 20: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/20.jpg)
Microservices architecture in US
Gateway API
Mercari API
HTTP
offer
HTTP
gRPC
![Page 21: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/21.jpg)
Microservices architecture in US
Gateway API
Mercari API
HTTP
search offer
HTTP
gRPC
![Page 22: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/22.jpg)
Microservices architecture in US
Gateway API
Mercari API
HTTP
search
personalization
offer
HTTP
gRPC
![Page 23: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/23.jpg)
Technical stacks
● Docker● Kubernetes (Google Container Engine) ● gRPC
![Page 24: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/24.jpg)
Container
● Resource isolation● Resource limitation● Fast boot (vs. VM)
Docker
● Easy to build container image● Easy to distribute via registry
![Page 25: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/25.jpg)
Why Docker?
● Software engineer control more○ They can include what they want (e.g., runtime, library)
● Environmental parity○ What works on local development (or QA env) is exact same (easy to debug)○ No more “it works on my environment but not in production!”
● Easy to deploy○ Docker image ≒ Single static linked binary○ You already know its benefit if you use Go
![Page 26: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/26.jpg)
Kubernetes (GKE)
● Container orchestration● Derives from Google internal
system named Borg & Omega● Inspired and informed by
Google’s experiences and internal systems
![Page 27: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/27.jpg)
Why kubernetes?
● Best way to maximize container benefit○ Resource isolation/limitation enables us compute resource utilization. But how?
■ K8s can correctly schedule container proper instances○ How to communicate between dynamically scheduled containers?
■ K8s provide the service discovery
● Reduce operation costs ○ Self healing & auto scaling
● Infrastructure of infrastructure○ Industrial standard https://githubengineering.com/kubernetes-at-github○ More tools/software comes top on k8s in future (I guess)
![Page 28: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/28.jpg)
gRPC
● gRPC Remote Procedure Call● High performance, general
purpose, open source, standards-based, RPC framework
● Open source version of stubby RPC in used in Google
![Page 29: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/29.jpg)
gRPC
● Simple service definition○ By default, gRPC uses protocol buffers as the Interface Definition Language (IDL) for
describing both the service interface and the structure of the payload messages.
● Works across languages and platforms○ Write golang server and python client○ Utilize polyglot microservices
![Page 30: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/30.jpg)
Why not REST?
● Who can implement REST correctly?○ High cost to design (Path? Parameters? hah?)○ Eventually it’s just HTTP endpoints
● No more HTTP client implementation ..
![Page 31: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/31.jpg)
Challenges
![Page 32: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/32.jpg)
Challenges
● Deployment ● Observability
![Page 33: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/33.jpg)
Deployment
● Deployment is key in microservices platform○ “Without velocity stalled, rather make iteration speed faster”
● We need easy & safe automated deployment system○ We started chatbot style deployment but it was not scale
![Page 34: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/34.jpg)
Spinnaker
● Continuous Delivery platform● Developed in Netflix
○ Worked with Google and open sourced in 2015
● Support multi cloud○ Kubernetes!, GCE, AWS
![Page 35: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/35.jpg)
Spinnaker GUI
![Page 36: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/36.jpg)
Spinnaker pipeline
![Page 37: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/37.jpg)
Why Spinnaker?
● Kubernetes support● Built-in deployment best practice from Netflix and Google
○ Immutable infrastructure○ Blue/Green deployment, Canary deployment○ Manual judgement (by manager) phase○ Run integration tests
![Page 38: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/38.jpg)
Spinnaker in Mercari
● Currently only for container deployment to kubernetes● Each team uses spinnaker to deploy their own services● One spinnaker handles all microservices in all region
![Page 39: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/39.jpg)
Example pipeline of API gateway deployment (Canary)
![Page 40: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/40.jpg)
One spinnaker cluster manages Mercari global deployment
JPUS
UK
![Page 41: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/41.jpg)
Future of spinnaker
● Pipeline as a Code○ https://github.com/spinnaker/dcd-spec
● Automated canary analysis
![Page 42: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/42.jpg)
Automated canary analysis
https://blog.spinnaker.io/can-i-push-that-building-safer-low-risk-deployments-with-spinnaker-a27290847ac4
![Page 43: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/43.jpg)
Observability
Observability (logging, metrics & tracing) is important
● Each team needs to debug service by themselves without SSH● It’s harder and more complex than monolith
![Page 44: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/44.jpg)
Stackdriver logging
![Page 45: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/45.jpg)
Request ID in log
● Which service caused problem in one request?
![Page 46: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/46.jpg)
Request ID in log
Gateway API
Mercari API
HTTP
search
personalization
offer
HTTP
gRPC
① Generate unique ID
② Annotate log by the ID in same request
HTTP headergRPC metadata
![Page 47: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/47.jpg)
Request ID in log
Search by request ID
Log from gateway
Log from service X
![Page 48: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/48.jpg)
Distributed tracing
● Which services makes the request slow?
![Page 49: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/49.jpg)
Stackdriver tracing
![Page 50: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/50.jpg)
Metrics
Selection of metrics service/software is still on-going discussion & trial
● First support of container and kubernetes ● Integration with kubernetes ecosystem
○ Spinnaker, istio and so on
● Service dependency visualization
![Page 51: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/51.jpg)
Prometheus + grafana
![Page 52: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/52.jpg)
Datadog
![Page 53: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/53.jpg)
Instana
![Page 54: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/54.jpg)
State of microservices in JP
![Page 55: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/55.jpg)
State of microservices in JP
JP is just started
● Some services (Machine learning product) are started to containerized and deployed on GKE
● On-going discussion about the best architecture
![Page 56: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/56.jpg)
Conclusion
● Why we started microservices?● Current state of US microservices and challenges
![Page 57: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/57.jpg)
We’re hiring
● Who loves automation● Technical keywords
○ Docker○ Kubernetes○ gRPC○ Golang ○ Container monitoring
![Page 58: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/58.jpg)
![Page 59: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/59.jpg)
Spinnaker is deployed on GKE
![Page 60: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/60.jpg)
Testing
Testing in microservice is hard?
● Focus on unit tests as usual○ Because each service is supposed to independent ○ Each microservices must measure testing coverage
● Integration tests?○ Use mock instead of working hard for preparing local env
![Page 61: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/61.jpg)
Testing pyramid
Google Testing Blog: Just Say No to More End-to-End Tests
Do this a lot !
Do mock
![Page 62: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/62.jpg)
QA environment
How to test development feature from QA device?
● Pull request (PR) based pod creation
![Page 63: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/63.jpg)
PR based pod creation
Proxy API gateway (master)
API gateway (PR 313)
API gateway (PR 314)
Proxy by PR number
Set RP number
Container is deployed via CI
![Page 64: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/64.jpg)
PR based docker container (QA env)
Easy to switch
![Page 65: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/65.jpg)
PR based pod creation
Proxy API gateway (master)
API gateway (PR 313)
API gateway (PR 314)
Service A (master)
Service A (PR 21)
Proxy by PR number
Set RP number
Container is deployed via CI
![Page 66: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/66.jpg)
Future works
![Page 67: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/67.jpg)
Service mesh
Don’t trust each other!
● Traffic management○ API rate limit, circuit breaker
● Policy enforcement○ Ensure access policies (which service can access which service?)
We should realize above without modifying client/server code!
![Page 69: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/69.jpg)
Service mesh (Istio)
![Page 70: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/70.jpg)
Chaos engineering
● Real world is hard … ○ machine is crashed, network is unstable (especially in distributed system)
● Dependent service fails anytime
![Page 71: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/71.jpg)
Chaos engineering
● Service must be fault tolerance whenever something wrong● Emulate real world problem
○ We need to identify weaknesses ■ Improper fallback settings when a service is unavailable
○ Software Engineer should be aware
![Page 72: Microservices at Mercari](https://reader034.fdocuments.us/reader034/viewer/2022050613/5a6479b67f8b9a63568b4687/html5/thumbnails/72.jpg)
Chaos engineering (Chaos monkey)
https://github.com/Netflix/chaosmonkey