Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017
-
Upload
baruch-sadogursky -
Category
Technology
-
view
293 -
download
0
Transcript of Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017
![Page 1: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/1.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Debugging & Troubleshooting Microservicesw/ Kubernetes
It runs in production, or is it?
![Page 2: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/2.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Ray TsangDeveloper AdvocateGoogle Cloud Platform
@saturnism | +RayTsang
![Page 3: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/3.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
BaruchSadogurskyDeveloper AdvocateJFrog
@jbaruch
![Page 4: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/4.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Shownotes!
● jfrog.com/shownotes○ Slides○ Video (tomorrow)○ Links○ Feedback○ Raffle
![Page 5: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/5.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
![Page 6: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/6.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Hello World Service - Greet
Guestbook Service - Create
Guestbook Service - Retrieve
![Page 7: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/7.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
GuestbookUI
Hello World Service
Redis
session replication
greeting
MySQL
GuestbookService
CRUD
![Page 8: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/8.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Visibility & Insight is Key
![Page 9: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/9.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Let's see it!
![Page 10: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/10.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Repository metadata - key insight into artifacts and their relationships
![Page 11: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/11.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Production vs Non-Production :(
![Page 12: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/12.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Distributed Systems are hard!
![Page 13: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/13.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
We've been there, done that
![Page 14: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/14.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Information - Logs!
kubectl logs -f pod_id
![Page 15: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/15.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Inspecting the process / container
kubectl exec -ti pod_id /bin/bash
![Page 16: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/16.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Port forward to container
kubectl port-forward pod_id local_port:remote_port
![Page 17: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/17.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Isolate problematic instance w/ Labels
kubectl label pod pod_id --overewrite serving=false
![Page 18: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/18.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Pod
frontend
Pod
serving = true
version = 1.0
Pod
serving = true
version = 1.0
Service
Label selectors: version = 1.0 serving = true
Pod
serving = true
version = 1.0
![Page 19: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/19.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Pod
frontend
Pod
serving = true
version = 1.0
Pod
serving = true
version = 1.0
Service
Label selectors: version = 1.0 serving = true
Pod
serving = false
version = 1.0
![Page 20: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/20.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Log aggregation, at scale
![Page 21: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/21.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Stackdriver Logginghttps://cloud.google.com/logging/
![Page 22: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/22.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
![Page 23: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/23.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Metrics from Logs → Alerting
Near-realtime insight
![Page 24: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/24.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Search and Query Your Logs
![Page 25: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/25.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Export to BigQuery, Storage, Pub/Sub
Near-realtime insight
![Page 26: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/26.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
![Page 27: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/27.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
![Page 28: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/28.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Tracing at Google - Dapperhttps://research.google.com/pubs/pub36356.html
![Page 29: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/29.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Zipkin
![Page 30: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/30.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Stackdriver Tracehttps://cloud.google.com/trace/
![Page 31: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/31.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Zipkin → Stackdriver Tracehttps://cloud.google.com/trace/docs/zipkin
![Page 32: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/32.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
![Page 33: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/33.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
![Page 34: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/34.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
![Page 35: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/35.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
![Page 36: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/36.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Automatic reports
Detect performance regression
![Page 37: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/37.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Oops… I forgot a log message. Darn!
![Page 38: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/38.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Don't you wish you can use a debugger?
![Page 39: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/39.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Stackdriver Debugger - Production Debugger
https://cloud.google.com/debugger/
![Page 40: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/40.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Add debugger agent
java -agentpath:/opt/cdbg/cdbg_java_agent.so ... -jar PATH_TO_JAR_FILE
![Page 41: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/41.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Logpoint - dynamically add log messages!
No redeployment, Magic.
gcloud debug logpoints create HelloworldService.java:35 \"Received endpoint: {endpoint}/{name}" --target helloworld-ui-1.0-SNAPSHOT
![Page 42: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/42.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Snapshots
Inspect call stack and variables
![Page 43: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/43.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
![Page 44: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/44.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
![Page 45: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/45.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Trace
Log
Debug
![Page 46: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/46.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Trace → Stackdriver Trace
Logs → Stackdriver Logging
Debug → Stackdriver Debuggerhttps://cloud.google.com/stackdriver/
![Page 47: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/47.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
Thanks!Source: https://github.com/saturnism/spring-boot-dockerShownotes: https://jfrog.com/shownotesStackdriver: https://cloud.google.com/stackdriver/
Ray Tsang@saturnismGoogle Cloud Platform
Baruch Sadogursky@jbaruchJFrog
![Page 48: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/48.jpg)
@saturnism @jbaruch @jfrog @googlecloud #devoxx2017 jfrog.com/shownotes
![Page 49: Troubleshooting & Debugging Production Microservices in Kubernetes as presented in Devoxx 2017](https://reader030.fdocuments.us/reader030/viewer/2022020411/5a6d15547f8b9ab3418b5621/html5/thumbnails/49.jpg)
@saturnism @googlecloud #kubernetes #devoxx