Logging in Python for large applications
-
Upload
fayaz-yusuf-khan -
Category
Technology
-
view
856 -
download
5
description
Transcript of Logging in Python for large applications
![Page 1: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/1.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Logging in PythonFor complex applications
Fayaz Yusuf Khan
20 December 2013
![Page 2: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/2.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Why use logging?
• All errors happen in runtime.
• Testing can never be 100%
![Page 3: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/3.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Why use logging?
• All errors happen in runtime.
• Testing can never be 100%
![Page 4: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/4.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Why use logging?
• All errors happen in runtime.
• Testing can never be 100%
![Page 5: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/5.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
import logging
logging.warning(’Watch out!’) # will print a message
logging.info(’I told you so’) # will not print anything
Output:
WARNING:root:Watch out!
![Page 6: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/6.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
import logging
logging.warning(’Watch out!’) # will print a message
logging.info(’I told you so’) # will not print anything
Output:
WARNING:root:Watch out!
![Page 7: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/7.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Logging to a file
import logging
logging.basicConfig(filename=’example.log’,
level=logging.DEBUG)
logging.debug(’This message should go to the log file’)
logging.info(’So should this’)
logging.warning(’And this , too’)
Output:
DEBUG:root:This message should go to the log file
INFO:root:So should this
WARNING:root:And this, too
![Page 8: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/8.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Logging to a file
import logging
logging.basicConfig(filename=’example.log’,
level=logging.DEBUG)
logging.debug(’This message should go to the log file’)
logging.info(’So should this’)
logging.warning(’And this , too’)
Output:
DEBUG:root:This message should go to the log file
INFO:root:So should this
WARNING:root:And this, too
![Page 9: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/9.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Initializing
logger = logging.getLogger(__name__)
![Page 10: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/10.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
print()
Display console output for ordinary usage of a command line scriptor program
![Page 11: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/11.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
logging.info()
Report events that occur during normal operation of a program(e.g. for status monitoring or fault investigation)
![Page 12: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/12.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
logging.debug()
Very detailed output for diagnostic purposes
![Page 13: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/13.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
warnings.warn()
In library code if the issue is avoidable and the client applicationshould be modified to eliminate the warning.
![Page 14: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/14.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
logging.warning()
If there is nothing the client application can do about thesituation, but the event should still be noted
![Page 15: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/15.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Raise an exception
Report an error regarding a particular runtime event
![Page 16: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/16.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Raise an exception
Report an error regarding a particular runtime event
![Page 17: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/17.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
logging.error() or logging.exception()
Report suppression of an error without raising an exception (e.g.error handler in a long-running server process)
![Page 18: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/18.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
logging.critical()
A serious error, indicating that the program itself may be unable tocontinue running.
![Page 19: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/19.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Logbookpythonhosted.org/Logbook
Logbook is a logging sytem for Python that replaces the standardlibrary’s logging module.
![Page 20: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/20.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
>>> from logbook import Logger
>>> log = Logger(’Logbook ’)
>>> log.info(’Hello , World!’)
[2010 -07 -23 16:34] INFO: Logbook: Hello , World!
![Page 21: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/21.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Context managers everywhere!
from logbook import SyslogHandler
error_handler = SyslogHandler(’logbook example ’,
level=’ERROR’)
with error_handler.applicationbound ():
# whatever is executed here and an error is logged to
# the error handler
...
![Page 22: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/22.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Getting your libraries and old code to behave
from logbook.compat import redirect_logging
redirect_logging ()
![Page 23: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/23.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Injecting context
import os
from logbook import Processor
def inject_cwd(record ):
record.extra[’cwd’] = os.getcwd ()
with my_handler.applicationbound ():
with Processor(inject_cwd ). applicationbound ():
# everything logged here will have the current
# working directory in the log record.
...
![Page 24: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/24.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Formatting in Logging
import logging
logging.basicConfig(format=’%( levelname)s:%( message)s’,
level=logging.DEBUG)
![Page 25: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/25.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Formatting in Logbook
>>> from logbook import StderrHandler
>>> handler = StderrHandler(
... format_string=’{record.channel }: {record.message}’)
>>> handler.formatter
<logbook.handlers.StringFormatter object at 0x100641b90 >
![Page 26: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/26.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Custom formatters
>>> def my_formatter(record , handler ):
... return record.message
...
>>> handler.formatter = my_formatter
Tip:
Or use Jinja!
![Page 27: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/27.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Custom formatters
>>> def my_formatter(record , handler ):
... return record.message
...
>>> handler.formatter = my_formatter
Tip:
Or use Jinja!
![Page 28: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/28.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Extra fields
handler = StderrHandler(format_string=
’{record.channel }: {record.message) [{ record.extra[cwd]}]’)
![Page 29: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/29.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Redis
import logbook
from logbook.queues import RedisHandler
handler = RedisHandler ()
l = logbook.Logger ()
with handler:
l.info(’Your log message ’)
![Page 30: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/30.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
RabbitMQ
Handler
handler = RabbitMQHandler(’amqp :// guest:guest@localhost //’,
queue=’my_log ’)
Subscriber
with target_handler:
subscriber = RabbitMQSubscriber(
’amqp :// guest:guest@localhost //’, queue=’my_log ’)
subscriber.dispatch_forever ()
![Page 31: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/31.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
ZeroMQ
Don’t use it. (It’s broken)
![Page 32: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/32.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Minimize logging
>>> from logbook import FingersCrossedHandler , StderrHandler
>>> handler = FingersCrossedHandler(StderrHandler ())
>>> with handler.applicationbound ():
... logbook.info(’Hello’)
...
...
>>> handler = FingersCrossedHandler(StderrHandler ())
>>> with handler.applicationbound ():
... logbook.info(’Hello’)
... logbook.error(’Bye’)
...
...
[2013 -12 -16 11:12] INFO: Generic: Hello
[2013 -12 -16 11:12] ERROR: Generic: Bye
![Page 33: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/33.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
TestHandler
>>> from logbook import TestHandler , Logger
>>> logger = Logger(’Testing ’)
>>> handler = TestHandler ()
>>> handler.push_thread ()
>>> logger.warn(’Hello World’)
>>> handler.records
[<logbook.base.LogRecord object at 0x100640cd0 >]
>>> handler.formatted_records
[u’[WARNING] Testing: Hello World ’]
![Page 34: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/34.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Github
github.com/mitsuhiko/logbook
![Page 35: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/35.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Error reporting
Emails?
![Page 36: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/36.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Error reporting
Emails?
![Page 37: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/37.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Error reporting
Emails?
![Page 38: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/38.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Airbrake
Airbrake is the leading exception reporting service, currentlyproviding error tracking for 50,000 applications with support for 18
programming languages.
![Page 39: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/39.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Sign up
airbrake.io
![Page 40: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/40.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Flask
from flask import Flask , request
from airbrake import AirbrakeErrorHandler
import gevent
@app.errorhandler (500)
def internal_error(error ):
if app.config[’EXCEPTION_LOGGING ’]:
handler = AirbrakeErrorHandler(
api_key=app.config[’AIRBREAK_API_KEY ’],
env_name=ENV , request_url=request.url ,
request_path=request.path ,
request_method=request.method ,
request_args=request.args ,
request_headers=request.headers)
gevent.spawn(handler.emit , error)
![Page 41: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/41.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Django
AIRBRAKE = {
’API_KEY ’: ’your_api_key_here ’,
’USE_SSL ’: True ,
’TIMEOUT ’: 5,
’ENVIRONMENT ’: ’unique_name_for_environment ’,
}
![Page 42: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/42.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Errbit
github.com/errbit/errbit
![Page 43: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/43.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Sentrygetsentry.com
Sentry is a realtime event logging and aggregation platform. At itscore it specializes in monitoring errors and extracting all the
information needed to do a proper post-mortem without any of thehassle of the standard user feedback loop.
![Page 44: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/44.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Detail
![Page 45: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/45.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
More detail
![Page 46: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/46.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Install the server
sentry.readthedocs.org
![Page 47: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/47.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Sentry + Logbookraven.readthedocs.org/en/latest/config/logbook.html
Install raven
pip install raven
Your app code will have this at the entry point
from raven.handlers.logbook import SentryHandler
client = Client (...)
sentry_handler = SentryHandler(client)
with sentry_handler.applicationbound ():
# everything logged here will go to sentry.
...
![Page 48: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/48.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Pay attention to formatting!Sentry will intelligently group messages if you use proper string formatting.
logger.error(’There was some {} error’, ’crazy’)
logger.error(’There was some {} error’, ’fun’)
logger.error(’There was some {} error’, 1)
![Page 49: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/49.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Tagging
logger.error(’Could not do that’,
tags={’user email ’: ’foo@bar ’})
![Page 50: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/50.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
TicketingHandlerCreate unique tickets for log records and keep track of the number of times these log
records were created
from logbook import ERROR
from logbook.ticketing import TicketingHandler
handler = TicketingHandler(’postgres :// localhost/database ’,
level=ERROR)
with handler:
# everything in this block and thread will be handled by
# the ticketing database handler
...
![Page 51: Logging in Python for large applications](https://reader038.fdocuments.us/reader038/viewer/2022110115/549e9debb37959be618b4795/html5/thumbnails/51.jpg)
Why use logging Basics When to use what? Logbook Error reporting The end
Phew! It’s finally over!
Reach me at fayaz at dexetra dot com.