Functional and non functional application logging

APRIL 11, 2023 | SLIDE 1


www.realdolmen.com

APPLICATION LOGGING


ABOUT ME

Sander De Vos Software Engineer RealDolmen 2008 Past projects

MaTMa (M*) E-Metro (FOD-ECO) SP10B (FOD-FIN) MIK (FOD-MOB)


QUOTES FROM THE FIELD

“Who needs logs? I sure don’t!” “Debugger is better! Sysout works too!” “Isn’t that what infrastructure does?” “It slows down disk and memory!” “It isn’t in the spec!” “We don’t know what to log – we’ll just log something!” “We don’t know how – we’ll just do it somehow!”


AGENDA

What is logging Definition and Challenges

Why logging Who uses logs What logs are used for Functional or non-functional

What to log What will (not) help you Log message decomposed Guidelines

How to log in Java Logging in production Log processing and analysis


WHAT IS LOGGING


DEFINITION

“Logs may be referred to as log files, audit logs, or audit trails.

Logs also cover alerts, alarms and event records.“

“Logs are the recordings of one or more events occurring on information systems.”

“Logs can be organized based on the program, day, severity, host, or a number of other categories.“


LOG CHALLENGES

Decentralization Cloud SOA

Volatility Multiple tiers and layers Archival Accessability Absence of critical information Random log formats


WHY LOGGING


WHO USES LOGS

Managed Services Security Teams

Monitor Detect Investigate Track Analyze

Auditors Developers

Debugging

Customer


WHAT LOGS ARE USED FOR

Debugging and Forensics Fault monitoring Performance monitoring Troubleshooting Feature usage Security / Incident detection Regulatory and standards compliance


FUNCTIONAL – NON-FUNCTIONAL LOGGING

Functional Audit Governance Security

Non-functional Faults, Errors, Exceptions Execution context Performance Component usage


PUTTING IT ALL TOGETHER

Functional Logs Customer

Audit Always on Requirements

Scope known Years ‘Clutter, yuck, slow’

Non-Functional Logs System operator

Developer Sometimes on Errors, Exceptions,

Debug Scope not known Hours, Days ‘Might come in handy’


WHAT TO LOG


KEEP IN MIND

”What would help YOU at 3 AM in the morning? “

Image: Ambro / FreeDigitalPhotos.net


WHAT WILL HELP YOU

Exact notice of what happened When Where How Who

Clear concise context information Analysis possible without application

Manual Semi-automated Automated

Remotely collectible files Low footprint Proven reliable and authentic


WHAT WILL HELP YOU EXAMPLES

2011-05-31 08:46:11,308 +0200 INFO [eventlog.security] ACCESS OF ProfileServiceImpl.findProfiles BY [login: some_username]; Password: [PROTECTED]; Authenticated: true; Details: RemoteIpAddress: 192.168.156.17; SessionId: 8XyrNkLWv7FdXS… Granted Authorities:ROLE_USER,ROLE…


WHAT WON’T HELP YOU

No details about the record No context information Inconsistent format or bad use Non-correlatable Subjective/interpretative message Too much useless log records Logging to one single big file


WHAT WON’T HELP YOU – EXAMPLE #1

log4j: setFile called: ./logs/be/mobistar/network-status/log_functional.log, false

log4j: setFile ended tis nen default tis nen default Activations : 15 Deactivations : 0 In straigt to resource filter


WHAT WON’T HELP YOU – EXAMPLE #2

INFO [dao.ProductDao] [] - FOUND 40 INFO [web.listeners.PhaseLogger] [] -

+RENDER_RESPONSE INFO [service.ProductFamilyService] [] - product in

family is true INFO [.dao.ProfileDao] [] - [OBJECTS] find profiles


LOG MESSAGE DECOMPOSEDContext Meaning

WHEN Timestamp

WHERE System/Application/Component

WHO User

WHAT Action

RESULT Status

PRIORITY Severity, rank, level


“A log record should be understandable by a human, and easily machine

processable.”


LOGGING GUIDELINES: PRIORITIES

Fatal Application crash&burn

Error Exceptions Monitoring team alert

Warn Monitor health, performance

Info Informative Context Audit trail

Debug Extensive Context Development, Testing Troubleshooting


LOGGING GUIDELINES: EXCEPTIONS

Handling Recover Log the details if critical

Not Handling, Translating Translate Handling logic if not propagating

Not Handling Propagate Someone else will handle (and log!)

Choose exceptions to log Defend from stacktrace chaos Choose what to log for each exception

Catching block has all context details!


LOGGING GUIDELINES: CORRELATION

1 transaction = * log records Correlation necessary

Execution trail

Shared context variable Shared per transaction Reuse of context variables

Nested Diagnostic Context Map Diagnostic Context

Only one correlation Use a context unique variable Correlate transactions


HOW TO LOG IN


HOW LOGGING FRAMEWORKS WORK


JAVA LOGGING FRAMEWORKS

Implementation System.out – System.err

Redirect output Java Util Logging Log4j Logback

Abstraction Commons Logging SLF4J


JAVA UTIL LOGGING

Java 1.4+ Based on Log4J Less features out of the box

Console File Stream


LOG4J

First framework Introduction of Logger, Appender, Level Hiearchical Loggers Configurable appenders, e.g.

MailAppender FileAppender SMTPAppender SocketAppender SyslogAppender

Leader quit and started logback


COMMONS LOGGING

Apache Project API Bridge API-calls forward to

Log4j if on CP Falls back to JUL

Plagued Double Configuration Runtime dependency check Classpath Scanning

Discontinued v1.1.1 dates from end 2007


SLF4J - LOGBACK

Two libraries to rule them all Slf4j

Comparable with commons-logging Parameter parsing Framework bridges Much more powerful

Logback Evolved log4j Implements Slf4j “Picking up where log4j leaves off”

Testing Filters on-need-base


SLF4J BINDING


SLF4J BRIDGING


KEEPING IT PERFORMANT

String concatenation LOGGER.debug("Hello " + name); Protected block

if (logger.isDebugEnabled()) { LOGGER.debug("Hello " + name);}

Slf4j LOGGER.debug("Hello {}", name);


CONTEXT VARIABLES IN PRACTICE

NDC.push(correlationId); app.logMe("Hello world!"); NDC.pop();

Use %x in Pattern ConversionPattern = %x %d %m %n

correlationId 2011-06-21 09:20:41 [INFO] Hello world!


SPECIALIZED LOGGERS

Logger for class Fine-tune log Execution trace How distinguish

Security Performance Business

Specialized Loggers Priorities Output handlers

Grouped logging


LOGGING IN PRODUCTION

Only necessary logging Situation -> turn on debug logging

No restart: Dynamic Reloading ‘Watch’ file for changes

Log file management Rolling file Seperation of concerns

Clustering Different systems generate messages Synchronization of timestamps Correlation on host-specific files


MY OPINION

Importance of logging will grow Mobile / offline applications Cloud integration System interaction

Logging standards enterprisewide Standardization of logging effort Inexpensive exercise

Draft a standard Build adapters / patterns Enforce standards


LOG PROCESSING AND ANALYSIS


LOG ANALYSIS

Bang for your buck XpoLog

http://www.xpolog.com Licensed Faster

Splunk http://www.splunk.com Free Edition ‘Slow’ AJAX feel Nice graphics Modular (http://www.splunkbase.com)

Tuning your pattern is important?

http://www.xpolog.com/

http://www.xpolog.com/

http://www.splunk.com/



http://www.splunkbase.com/

http://www.splunkbase.com/


DEMO


THANK YOU

For more information:visit our website WWW.REALDOLMEN.COM

Follow us on:

Selected presentations are available on:

Or scan this QR code with your Smartphone to immediately go to the website

http://www.realdolmen.com/




http://twitter.com/realdolmentweet

http://blogs.realdolmen.com/

http://www.slideshare.net/RealDolmen

Functional and non functional application logging

Technology

Transcript of Functional and non functional application logging