Long live the DevOps team - Edinburgh 2015 - Skelton Thatcher
Why and how to test logging - DevOps Showcase North - Feb 2016 - Matthew Skelton
-
Upload
skelton-thatcher-consulting-ltd -
Category
Software
-
view
2.894 -
download
0
Transcript of Why and how to test logging - DevOps Showcase North - Feb 2016 - Matthew Skelton
Why and how to test loggingDevOps Showcase North / Testing Showcase North
Manchester, UK - UnicomThursday 11th February 2015
Matthew SkeltonSkelton Thatcher Consulting
confession:
I am a big fan of logging
use logging as a channel/vector to make distributed
systems more testable
transforming technology and teams
high impact expertise
Cloud, Agile, DevOps
Why we should test logging
How to test logging effectively
exceptional situationsedge cases
metricsanalytics‘audits’
…@evanphx
logging as an execution trace
reduce time-to-detectincrease team engagement
increase configurabilityenhance DevOps collaboration
#operability
(event sourcing)
(structured logging)
(CQRS)
using logging mainly for errors
inconsistent use of logging
“logging slows things down”
“logging pollutes my domain model”
“logging is just for Ops people”
log aggregation only in Production
logging assumed to be free ($0) to implement
logs not available to Tester or Developers
distinct event IDs
transaction tracing
instrumentation
log aggregation & search
cross-component visibility
increased test coverage
programmatic assertions
deep insights
stories for testing logging
BasketItemAdded
Search: BasketItem
Continuous event IDs
How many distinct event types (state transitions) in
your application?
represent distinct states
enum
Human-readable sets: unique values, sparse, immutable
C#, Java, Python, node(Ruby, PHP, …)
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred
UnexpectedError = 10000,
ApplicationStarted = 20000,
ApplicationShutdownNoticeReceived = 20001,
PageGenerationStarted = 30000,
PageGenerationCompleted = 30001,
MessageQueued = 40000,
MessagePeeked = 40001,
BasketItemAdded = 60001,
BasketItemRemoved = 60002,
CreditCardDetailsSubmitted = 70001,
// ...
}
Technical
Domain
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred
UnexpectedError = 10000,
ApplicationStarted = 20000,
ApplicationShutdownNoticeReceived = 20001,
PageGenerationStarted = 30000,
PageGenerationCompleted = 30001,
MessageQueued = 40000,
MessagePeeked = 40001,
BasketItemAdded = 60001,
BasketItemRemoved = 60002,
CreditCardDetailsSubmitted = 70001,
// ...
}
BasketItemAdded = 60001
BasketItemAdded = 60001
BasketItemRemoved = 60002
BasketItemAdded = 60001
BasketItemRemoved = 60002
represent distinct states
OrderSvc_BasketItemAdded
Monolith to microservices:debugger does not have the full view
Even with remote debugger, it’s boring to attach and detach
Storage I/O
Worker Job
Queue
Upload
Transaction tracing
‘Unique-ish’ identifier for each request
Passed through downstream layers
Unique-ish ID
Testing logging
use logging as a channel/vector to make distributed
systems more testable
track a parcel online
search by event ID
transaction trace
Web App
Doc store
Audit service
Msg queue
Transactional sub-system
Log aggregation sub-system
An action here… …results in log entry here
trace events in a distributed,
asynchronous system
stories for testing logging
AuditRecordCreated
Search: AuditRecordCreated
AuditRecordCreated
Search: AuditRecordCreated
Feature: User trades stocksScenario: User requests a sell before close of trading
Given I have 100 shares of MSFT stockAnd I have 150 shares of APPL stockAnd the time is before close of trading
When I ask to sell 20 shares of MSFT stock
Then I should have 80 shares of MSFT stockAnd I should have 150 shares of APPL stockAnd a sell order for 20 shares of MSFT stock should have been executed
http://martinfowler.com/bliki/GivenWhenThen.html - Pete Hodgson
Given I run a scenario as a LawyerAnd I create a document[And I wait 5 seconds]
When I search the logs APIThen I should find a recent entry for “AuditRecordCreated”
Recap
use logging as a channel/vector to make distributed
systems more testable
unique-ish event type IDs
transaction tracing
log aggregation & search
Stories: expected search results
cross-component visibility
increased test coverage
programmatic assertions
deep insights
trace events in a distributed,
asynchronous system
‘Structured Logging’TW: “Adopt” (May 2015)
https://www.thoughtworks.com/radar/techniques/structured-logging
http://gregoryszorc.com/
.NET: http://serilog.net/Java: https://github.com/fluent/fluent-logger-java
More
Ditch the Debugger and Use Log Analysis Instead
Matthew Skelton
https://blog.logentries.com/2015/07/ditch-the-debugger-and-use-log-analysis-instead/
Thank you
http://skeltonthatcher.com/[email protected]
@SkeltonThatcher
+44 (0)20 8242 4103
@matthewpskelton