The Life of a Web RequestMeasuring and improving

Django application performance

PyCon Australia 2013

Roger Barnes@mindsocketroger@mindsocket.com.auhttp://slideshare.net/mindsocket/

What are you here for

Aspects of site performance

Measuring everything

Fixing the right thing

AND HOW

Performance focus for this talk

✔ user perspective✔ seconds per request✗ benchmarks✗ users per server✗ requests per second

Why should we care?

● Web application performance affects– engagement

Why should we care?

● Web application performance affects– engagement– bounce rate

http://www.slideshare.net/joshfraz/sept-2012rumtalk

Why should we care?

● Web application performance affects– engagement– bounce rate– conversion/abandonment

Why should we care?

● Web application performance affects– engagement– bounce rate– conversion/abandonment– search engine ranking

Why should we care?

● Web application performance affects– engagement– bounce rate– conversion/abandonment– search engine ranking– revenue

Why should we care?

Web sites are

larger&

than ever

Why should we care?

Average web page:

1.3Mb85 requests

- HTTPArchive.org

Why should we care?

Users are increasingly mobile

Their expectations are up

Many "responsive" sites are no smaller than their desktop version

Let's look at aweb request!

There's a redirect!

How about some CSS?

Let's get some

JavaScript and images

Start Render: 1.6s

Document complete: 8.6s

Fully loaded: 9.0s

Requests: 177

Bytes in: 2,689kb

This is on a good connection

The Performance Golden Rule

"80-90% of the end-user response time is spent on the frontend.

Start there."

- Steve Souders

Source: http://www.stevesouders.com/blog/2012/02/10/the-performance-golden-rule/

Front-end Measurement

● Instrument your live application for Real User Monitoring (RUM)– Google Analytics– New Relic

Front-end Measurement

● Synthetic testing● Lots of tools available● 3 in particular...

Chrome Developer Tools

webpagetest.org

Browser navigation timing

http://www.w3.org/TR/navigation-timing/

django-debug-toolbar

Not in PyPi yet,see GitHub version

Back end Front end

Front-end offenders

● Static resources – size and number● Lack of caching and compression● 3rd-party resources● Overdownloading

Static Resources

● Minification/combination of CSS/JavaScript– django-compressor django-pipeline

● Images– Appropriate format and compression level– Correct size– Remove metadata

● jpegoptim, optiPNG

– Consider sprites

Static Resources

● Web server/accelerator config– Caching/expires/compression headers

● Static files to CDN– see Django's storage hooks

● Deferred loading● Silver bullet (?)

– mod_pagespeed (apache)– ngx_pagespeed (nginx)

Ok, now what?

Back-end measurement

Production instrumentation with New Relic

django-statsd and graphite

django-debug-toolbar

runprofileserver indjango-extensions

Web server

WSGI application (WSGIHandler.__call__)

Initialise Request

Get Response (BaseHandler.get_response)

URL Resolution

Request Middleware Response Middleware

View Middleware Template Response Middleware

View Template Rendering

Backend Services

Typical Django Request

Caching Dynamic Requests

● Potentially easy wins● Beware cache invalidation● Different levels

– Caching headers– Web Server/Web Accelerator– Caching middleware– Per view caching– Template fragment caching– Low level object caching

Web server / Reverse proxy

WSGI application (WSGIHandler.__call__)

Initialise request

Get Response (BaseHandler.get_response)

URL Resolution

Request Middleware Response Middleware

View Middleware Template Response Middleware

View Template rendering

Backend services

Typical Django Request

Caching

Browser

Compressing Dynamic Content

Web server

Web accelerator

gzip middleware

Query Performance

● Use debug toolbar● Avoid unnecessary queries

– eg exists followed by separate get

● select_related to get FKs in same query– Be wary of optional foreign keys

● prefetch_related to get collections– 1 additional query instead of N+1

Defer work

● Defer user-irrelevant workloads● Target

– Views– Signals (eg post_save)– Middleware

Defer work with celery

def login(request): # Your login code here # ... user = form.save()

upload_to_nsa.delay(user.id) return HttpResponse(...)

Ok, now what?

Web server

WSGI application (WSGIHandler.__call__)

Initialise request

Get Response (BaseHandler.get_response)

URL Resolution

Request Middleware Response Middleware

View Middleware Template Response Middleware

View Template rendering

Backend services

Django Works Then Responds

Bytes on the wire(Time to First Byte)

Back end

Can we stream?

● StreamingHttpResponse in 1.5● Patch for streaming templates (#13910)● Caveats

– Middleware compatibility– Exception handling– YAGNI?

What about eager streaming?

Hack: View middleware that streams the top of a page before even calling the view

Proof of concept:

https://github.com/mindsocket/django-perf-example

Regular

Streaming

Eager streaming

Visual appearance

Future context

● APIs, websockets and JavaScript, oh my!● SPDY● Mobile● Full stack ownership

Summary

● Measure, measure, measure● Don't ignore the front-end● Beware diminishing returns● Technology is changing

Resources

● General web performance– http://www.slideshare.net/Strangeloopnet/37-lessons-

ive-learned-on-the-performance-front-lines– http://www.webpagetest.org/– https://developers.google.com/speed/pagespeed/– http://www.mnot.net/cache_docs/– http://newrelic.com/– https://dvcs.w3.org/hg/webperf/raw-

file/tip/specs/NavigationTiming/Overview.html– https://www.varnish-cache.org/

Resources

● Django – https://docs.djangoproject.com/en/dev/topics/cache/– https://code.djangoproject.com/wiki/ProfilingDjango– https://code.google.com/p/django-command-extensions/wiki/RunProfileServer– https://developers.google.com/chrome-developer-tools/– https://docs.djangoproject.com/en/dev/howto/static-files/deployment/#staticfiles-

from-cdn– https://docs.djangoproject.com/en/dev/howto/custom-file-storage/– https://github.com/django-debug-toolbar/django-debug-toolbar/issues/319– https://github.com/sorl/sorl-

thumbnail/commit/e5f493a340dea8a33703c05fd66ee76d2f019881– https://github.com/aaugustin/django-c10k-demo– https://github.com/mindsocket/django-perf-example– https://www.djangopackages.com/grids/g/asset-managers/

Questions?

http://slideshare.net/mindsocket

Thank You!