Semantic Logging: Avoiding the Logging Chaos

20
Christopher Bennage patterns & practices microsoft.com/practices Semantic Logging Avoiding the Log Chaos

description

Presented at CodeFest 2014 Whether you are logging for the purpose of diagnostics or monitoring, it requires proper, well-designed instrumentation and a sound strategy. The new Semantic Logging Application Block (SLAB) offers a smarter way of logging by keeping the structure of the events when writing log messages to multiple destinations such as rolling flat file, database or Windows Azure table storage. In this talk, we will give an introduction to SLAB and provide a time of Q&A. We will address questions like: * What are the pros and cons of using SLAB? * What is the performance impact? * How can I extend SLAB? * Do I have to commit to using ETW? * Does SLAB support .NET’s EventSoure? * How extensible is SLAB? Can you provide an example? * Can you use SLAB without knowledge of ETW? * What is the trade-off between using SLAB in-process vs out-of-process? * How steep is the learning curve? How do I get started? * How can I contribute to SLAB?

Transcript of Semantic Logging: Avoiding the Logging Chaos

Page 1: Semantic Logging: Avoiding the Logging Chaos

Christopher Bennagepatterns & practicesmicrosoft.com/practices

Semantic LoggingAvoiding the Log Chaos

Page 2: Semantic Logging: Avoiding the Logging Chaos

• No real structure

• What’s in there?

• Sheer number of files & types of logs is overwhelming

• Hard to consume/automate

• Subject to compatibility/inconsistencies.

Logs are Frustrating

Page 3: Semantic Logging: Avoiding the Logging Chaos

Semantic?

Page 4: Semantic Logging: Avoiding the Logging Chaos

Unstructured LogAn example

176 [main] INFO  examples.Sort - Populating an array of 2 elements in reverse order.225 [main] INFO  examples.SortAlgo - Entered the sort method.262 [main] DEBUG SortAlgo.OUTER i=1 - Outer loop.276 [main] DEBUG SortAlgo.SWAP i=1 j=0 - Swapping intArray[0] = 1 and intArray[1] = 0290 [main] DEBUG SortAlgo.OUTER i=0 - Outer loop.304 [main] INFO  SortAlgo.DUMP - Dump of integer array:317 [main] INFO  SortAlgo.DUMP - Element [0] = 0331 [main] INFO  SortAlgo.DUMP - Element [1] = 1343 [main] INFO  examples.Sort - The next log statement should be an error message.346 [main] ERROR SortAlgo.DUMP - Tried to dump an uninitialized array.467 [main] INFO  examples.Sort - Exiting main method.

Page 5: Semantic Logging: Avoiding the Logging Chaos

Structured LogAn example using Azure

Query by payload argument

Page 6: Semantic Logging: Avoiding the Logging Chaos

• Logging cannot be just a checkmark of doing something.• You have to think about consumption and purpose.• Allow appropriate decisions to be made at appropriate

time, explicitly separating:•WHAT to log•WHEN to log it•WHERE to log

We are on a missionChanging the way people think about logging

Page 7: Semantic Logging: Avoiding the Logging Chaos

demo >> EventSource & SLAB in-process

Page 8: Semantic Logging: Avoiding the Logging Chaos

Technologies at PlayEvent Tracing for Windows (ETW)

•Native to Windows platform

•Great performance & OK diagnostic tooling

•Historically hard to publish events

EventSource class

• Introduced in .NET Framework 4.5

•Meant to ease authoring experience

• Extensible but supports ETW-only out of the box

Semantic Logging Application Block (SLAB)

• Provides several destinations for events published with EventSource

•Does not require any knowledge in ETW

•Additional tooling support for authoring events

Page 9: Semantic Logging: Avoiding the Logging Chaos

Technologies at Play

.NET Event Source

Custom Event Source

SLAB in-processObservable EventListen

erSinks

SLAB out-of-process

TraceEvent

Sinks

Third party tools

(e.g. PerfView)

Event Log

ETW

Page 10: Semantic Logging: Avoiding the Logging Chaos

• Sinks• Formatters• Out-Of-Process Service• Event Source Analyzer• Observable Event Listener

SinksFeatures of SLAB

Page 11: Semantic Logging: Avoiding the Logging Chaos

• Azure Tables• SQL Database• Flat file• Rolling flat file• Console• Elasticsearch

SinksFeatures of SLAB

Page 12: Semantic Logging: Avoiding the Logging Chaos

• JSON• XML• Natural (plain-text)

FormattersFeatures of SLAB

Page 13: Semantic Logging: Avoiding the Logging Chaos

• Hosted as a Windows Service or console• All sinks are supported• Configuration-driven with support for re-configuration

Benefits•Increased fault tolerance in case of application crash•Monitored application does not reference SLAB•Can monitor multiple processes from a single service.•Moves the logging overhead from the application to a separate process (but the overhead is still there!)

Out of Process ServiceFeatures of SLAB

Page 14: Semantic Logging: Avoiding the Logging Chaos

// can be run in a unit test[TestMethod]public void AnalyzeAExpenseEvents(){ EventSourceAnalyzer.InspectAll(AExpenseEvents.Log);}

// will verify correctness of events// this example has inconsistent ID and order of parameters[Event(111)]public void MyInvalidEvent(int someArgument, string otherArgument, int userId){ this.WriteEvent(222, someArgument, userId, otherArgument);}

Event Source AnalyzerFeatures of SLAB

Page 15: Semantic Logging: Avoiding the Logging Chaos

• Event listener is IObservable.• Event sinks are IObservers.• Can leverage Reactive Extensions

(Rx) to filter, pre-process or transform the event stream before it’s persisted.

Based on ObservableFeatures of SLAB

Page 16: Semantic Logging: Avoiding the Logging Chaos

demo >> Flush on Error/Alarm Flood Throttle

Page 17: Semantic Logging: Avoiding the Logging Chaos

• Support for ActivityIds• Ability to capture events from source not publicly available• Sink for Elasticsearch • Performance improvements• Improved extensibility story• Minor bug fixes

http://aka.ms/slab1_1

SLAB 1.1What’s Coming Next Now

Page 18: Semantic Logging: Avoiding the Logging Chaos

• Evaluate SLAB and adopt it (search for “slab” in NuGet).• Read the docs - aka.ms/slab • Practice the Hands-on Labs &

Quickstarts - aka.ms/el6hols• Engage with us by providing

feedback and/or submitting contributions - slab.codeplex.com

Call to Action

Page 19: Semantic Logging: Avoiding the Logging Chaos

• http://slab.codeplex.com• http://aka.ms/slab• http://entlib.codeplex.com

Resourcesmicrosoft.com/practices

@bennage

dev.bennage.com

Page 20: Semantic Logging: Avoiding the Logging Chaos

Вопросы?

Christopher Bennagepatterns & practicesMicrosoft

[email protected]