To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly
16
| Log management as a service Simplify Log Management Apache Storm To Log or Not to Log: Proven Best Practices for Instrumentation Jon Gifford Founder and Chief Search Officer January 2014
-
Upload
loggly -
Category
Technology
-
view
399 -
download
2
description
To log or not to log? This is an age-old question for developers. Logging everything can be great because you have plenty of data to work from when you have a problem. But it’s not so great if you have to grep and inspect it all yourself. In my mind, developers should instead be thinking about logging the right events in the right format for the right consumer. Instrumentation is one of the most valuable sources of data you can log. At Loggly, we instrument like crazy in our code because we need to understand what our system is doing and where we can improve it. When you’re dealing with billions of events per day, the ability to quickly identify hot-spots is invaluable. However, there are lots of different opinions about what you should instrument and concerns about the performance impact of that instrumentation. What follows are the guidelines and best practices that we’ve developed to help ourselves do a better job of instrumentation. Hopefully they’ll help you too… Read the full blog post on the Loggly blog → http://bit.ly/BestLoggingPractices
Transcript of To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly
| Log management as a service Simplify Log Management
Apache Storm
To Log or Not to Log: Proven Best Practices for Instrumentation
Jon Gifford Founder and Chief Search Officer January 2014
| Log management as a service Simplify Log Management
About Me 15+ years of search experience
§ Lots of different search products § Lots and lots of machines § Lots and lots of logs...
Bias: § Lucene, Java, *nix, Distributed, Realtime, § Measure, Measure, Measure
| Log management as a service Simplify Log Management
§ Centralized logging and archival
§ Real-time processing, analysis and visualization
§ Monitoring, alerting and troubleshooting
Loggly: Addressing the first big data problem every company faces
| Log management as a service Simplify Log Management
To Log or Not to Log?
Lots of data to solve operational problems faster!
Incredibly painful to grep and inspect yourself + -
| Log management as a service Simplify Log Management
Instrumentation: Your Most Valuable Log Data For us, instrumentation is the act of gathering performance data about your application and surfacing it via logs.
§ Provides a low-impact mechanism for gathering production system performance data
§ Visualization is key to understanding this data
| Log management as a service Simplify Log Management
Best Practices for Instrumentation
| Log management as a service Simplify Log Management
Best Practice #1 Treat instrumentation as an ongoing, iterative process.
§ Start by logging everything at a high level § Instrument deeper every time you ask, “What
happened here?” § Instrument deeper in areas that are more variable
or expensive than you thought
| Log management as a service Simplify Log Management
Best Practice #2 Always instrument anything that goes out of process.
§ Can be indicative of larger problems, like the canary in the coal mine § Example: Instrument calls to
your database, even though they should always be fast
§ Distributed system problems are not well behaved – by definition
| Log management as a service Simplify Log Management
Best Practice #3 Always log unacceptable performance.
§ Log “soft exceptions” (e.g. slow queries) § “Trust but verify”
| Log management as a service Simplify Log Management
Best Practice #4 If possible, log enough context so you can get the complete picture of what happened from a single log event.
Process
Log this … not this!
Elapsed time End
End Start
| Log management as a service Simplify Log Management
Best Practice #5 View machines, not people, as your logs’ consumer.
| Log management as a service Simplify Log Management
Best Practice #6 Trends tell the story better than data points.
§ Graph performance over time to see daily fluctuations
§ Use rollups where it’s not practical to log every single event
| Log management as a service Simplify Log Management
Best Practice #7 Instrumentation is NOT a substitute for profiling, and vice versa.
Instrumentation Profiling
Your production environment is not the same as development!
| Log management as a service Simplify Log Management
Best Practice #8 Flying slower is better than flying blind.
§ “Speed advantage” is illusory if you can’t understand system performance
§ Our debate: not whether to instrument but how much
Phot
o by
Eri
c Ki
lby
| Log management as a service Simplify Log Management
Implement Instrumentation Best Practices Faster and Better with Loggly.
About Us: Loggly is the world’s most popular cloud-based log management solution, used by more than 5,000 happy customers to effortlessly spot problems in real-time, easily pinpoint root causes and resolve issues faster to ensure application success.
Try Loggly for Free! → http://bit.ly/BestLoggingPractices
Visit us at loggly.com or follow @loggly on Twitter.
| Log management as a service Simplify Log Management
Proven and Trusted by 5,000+ your peers…
“If we didn’t have Loggly, there would be days where our revenue would be affected by as much as 70 percent.”
– Albert Ho, Executive Producer/
Platform Product Manager, Rumble Entertainment
