Observability: Less is less and more is more...• Make orchestration itself observable ... •...
Transcript of Observability: Less is less and more is more...• Make orchestration itself observable ... •...
![Page 1: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/1.jpg)
© 2 0 2 0 S P L U N K I N C .
Abstraction Protraction Monitoring in a Kubernetes Environment
Dave McAllister
Feb 13th, 2020
![Page 2: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/2.jpg)
© 2 0 2 0 S P L U N K I N C .
Abstraction adds
Infrastructure to your
Monitoring concerns
![Page 3: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/3.jpg)
© 2 0 2 0 S P L U N K I N C .
In the beginning...
The world was a simpler place
• Server
• App(s)
• Logs
Server
Applications
Logs
![Page 4: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/4.jpg)
© 2 0 2 0 S P L U N K I N C .
Then stuff got interesting1
• Microservices
• Orchestration
• containerd
• Massive Scale
• CI/CD
And so on
(1) “May you live in interesting times”
![Page 5: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/5.jpg)
© 2 0 2 0 S P L U N K I N C .
“It’s Turtles all the Way Down”
https://xkcd.com/1416/
TL; DR:
► Abstractions make life “simpler”
► Abstractions make digging harder
![Page 6: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/6.jpg)
© 2 0 2 0 S P L U N K I N C .
Orchestration
• Containers change the way we develop, deploy, and maintain apps
• Orchestration - a tool that automates the deployment, management, scaling, networking, and availability of apps
• Container Orchestration - a tool that automates the deployment, management, scaling, networking, and availability of container-based apps
Kubernetes is container orchestration at large
![Page 7: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/7.jpg)
© 2 0 2 0 S P L U N K I N C .
Tools add Complexity
• Kubernetes - Container Orchestration - Master - Worker Nodes - Pods
• Containers • App • Sidecar
![Page 8: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/8.jpg)
© 2 0 2 0 S P L U N K I N C .
![Page 9: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/9.jpg)
© 2 0 2 0 S P L U N K I N C .
But Monitoring? Logs?
• Early stacks were simple and static
• But orchestration changed that
![Page 10: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/10.jpg)
© 2 0 2 0 S P L U N K I N C .
So What?
• Monitoring needs to change
• Apps, containers, orchestrators
• “So where is it?”
• Logging needs to change
• Ephemeral
• Scaling as needed
Image C
redit t
o G
eek a
nd P
oke b
log:
htt
p:/
/geekandpoke.t
ypepad.c
om
/.a/6
a00d
83
41
d3
df5
53
ef0
128
76
4d
3a
ef9
70c-p
i
![Page 11: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/11.jpg)
© 2 0 2 0 S P L U N K I N C .
So what about K8s?
![Page 12: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/12.jpg)
© 2 0 2 0 S P L U N K I N C .
Monitor each layer explicitly • Collect logs and monitor for
• Errors
• Failures
• Performance issues
• At each layer
• Pod
• Container
• Orchestrator
Kubernetes Control Plane
API Server
etcd
Scheduler
Controller Manager
Kube-State-Metrics
Kublets
Container Metrics
Logs
![Page 13: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/13.jpg)
© 2 0 2 0 S P L U N K I N C .
Monitor your orchestrated environment
• Centralize your logs and metrics
• Your environment is still decentralized
• Getting things aggregated gives you a fighting chance
• Centralize your logs and metrics
• Your environment is still decentralized
• Getting things aggregated gives you a fighting chance
![Page 14: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/14.jpg)
© 2 0 2 0 S P L U N K I N C .
Simplify, simplify, simplify
• It’s complex enough with 17 tools doing different things
• Don’t do this
• Let your monitoring needs drive your tools
• Responsiveness matters
• In debug
• In deployment
• On scale
![Page 15: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/15.jpg)
© 2 0 2 0 S P L U N K I N C .
Data consistency makes life better
• Ensure data consistency across all the layers
• accurate timestamps
• consistent units of measurement (such as milliseconds vs. seconds)
• collecting a common set of metrics and logs across applications and components
![Page 16: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/16.jpg)
© 2 0 2 0 S P L U N K I N C .
Account for ephemeral containers
• Things start and stop all the time
• Grab the logs and metrics before they’re gone
• The thing you don’t see will get you
• Similar cluster and group activities
![Page 17: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/17.jpg)
© 2 0 2 0 S P L U N K I N C .
Monitor without touching the apps
• DaemonSet or sidecar model
• Use a DaemonSet with fluentd
• Use a DaemonSet with Prometheus monitoring
• Use instrumented libraries
![Page 18: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/18.jpg)
© 2 0 2 0 S P L U N K I N C .
• Be clear as crystal
• Keep services as small as necessary
• Make orchestration itself observable
• Find places not to orchestrate
![Page 19: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/19.jpg)
© 2 0 2 0 S P L U N K I N C .
Summary
• Orchestration and containers change the game
• There are new rules and new ways to apply
• Microservices and orchestration go hand in hand
• You can hurt yourself
• Tread lightly
![Page 20: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/20.jpg)
© 2 0 2 0 S P L U N K I N C .
Closing Thoughts
• We pay for ease of use in additional complexity • Emerging technology need attention just like apps • Keep an eye on on the truths in your logs
“A distributed system is one in which the failure of a
computer you didn't even know existed can render
your own computer unusable.”
– Leslie Lamport, 1987
![Page 21: Observability: Less is less and more is more...• Make orchestration itself observable ... • There are new rules and new ways to apply • Microservices and orchestration go hand](https://reader034.fdocuments.us/reader034/viewer/2022042412/5f2b1f986c67a579e34fc57e/html5/thumbnails/21.jpg)
Thank You
© 2 0 2 0 S P L U N K I N C .
https://www.linkedin.com/in/davemc