Speed is a Feature - PyConAr 2014
-
Upload
martin-blech -
Category
Software
-
view
189 -
download
1
Transcript of Speed is a Feature - PyConAr 2014
![Page 1: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/1.jpg)
This slide was intentionally left blank
![Page 2: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/2.jpg)
Speed is a featureA mystical journey through Django performance optimization techniques, tools and gotchas
@martinblech @polmuz
![Page 3: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/3.jpg)
Roadmap
How to find performance problems
Sneak peak: Front-end performance
How to fix them in Django
![Page 4: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/4.jpg)
Why?
[Google] found that Half a second delay caused a 20% drop in traffic.
[Amazon] found that even very small delays would result in substantial and costly drops in revenue.
Users really respond to speed2006
![Page 5: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/5.jpg)
There’s nothing like data
Don’t start with the code. Profile and gather real usage data.
![Page 6: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/6.jpg)
There’s nothing like real data
Identify bottlenecks
New Relic
Very goodVery paid
Google Analytics
Free of chargeLess detail
Your logs
No data to third partiesHarder to use
Profiling
Complex setup requiredOverhead
![Page 7: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/7.jpg)
New Relic
![Page 8: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/8.jpg)
New Relic
![Page 9: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/9.jpg)
Google Analytics Site Speed
![Page 10: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/10.jpg)
There’s nothing like data
Let’s find the culprit!
django-debug-toolbar
django-debug-toolbar-template-timings
![Page 11: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/11.jpg)
Time all the things!
![Page 12: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/12.jpg)
Typical backend bottlenecks
Database
External Services
CPU Intensive task
Template Rendering
![Page 13: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/13.jpg)
Database
Missing index
Big, unused fields
Excessive # of queries
Order
![Page 14: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/14.jpg)
Missing Index
class Comment(Model):...created_at = DateTimeField(db_index=True)blogpost = ForeignKey(Blogpost)
class Meta:index_together = [
["created_at", "blogpost"],]
![Page 15: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/15.jpg)
select_related()
>>> for c in Comment.objects.all(): print c.user.name# select * from comments;# select * from users where id = 1;# select * from users where id = 2;...
>>> comments = Comment.objects.all();>>> for c in comments.select_related(“user”): print c.user.name
# select comments.*, users.*# from comments, users# where comments.user_id = users.id;
![Page 16: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/16.jpg)
prefech_related()
>>> for u in User.objects.filter(id__lt=10): print len(u.comments.all())# select * from users;# select * from comments where user_id = 1;# select * from comments where user_id = 2;...
>>> users = User.objects.filter(id__lt=10)>>> for u in users.prefetch_related(“comments”): print len(u.comments.all())
# select * from users where id < 10;# select * from comments# where user_id in (1,2,3,4,5,6,7,8,9);## Joins them in python
![Page 17: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/17.jpg)
Demo
![Page 18: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/18.jpg)
Background Jobs
Celery
from celery import task
@taskdef send_confirmation_email(user_id): ...
def signup(req): ... send_confirmation_email.delay(req.user.id) return HttpResponseRedirect(“/home/”)
![Page 19: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/19.jpg)
Template Compilation
Use django.template.loaders.cached.Loader
TEMPLATE_LOADERS = (
('django.template.loaders.cached.Loader', (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
)),
)
![Page 20: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/20.jpg)
Template Fragment Caching
{% load cache %}
{% cache 500 “last_comments” %}
.. last comments ..
{% endcache %}
{% load cache %}
{% cache 500 “my_recent_comments” user.id %}
.. user’s recent comments ..
{% endcache %}
![Page 21: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/21.jpg)
Caching!
Per view cache
from django.views.decorators.cache import cache_page
@cache_page(60 * 15) # secondsdef my_view(request): ...
![Page 22: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/22.jpg)
Caching!
Per view cache
Gotchas:User-specific content
CSRF token
![Page 23: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/23.jpg)
Caching!
Low level cache API
>>> from django.core.cache import get_cache>>> cache = get_cache('default')>>> cache.set('my_key', 'hello, world!', 30)>>> cache.get('my_key')
>>> from django.core.cache import caches>>> cache = caches['default']
![Page 24: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/24.jpg)
Caching!
from django.core.cache import get_cachecache = get_cache('default')
def last_comments(request):comments_ids = cache.get('last_comments')if comments_ids:
comments = Comment.objects.filter(id__in=comments_ids)else:
comments = fetch_last_comments()comments_ids = [c.id for c in comments]cache.set('last_comments', comments_ids)
...
Low level cache API
![Page 25: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/25.jpg)
Cache Invalidation
“There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors.”
post_save/post_delete are a good place to start, but it doesn’t end there!
@receiver(post_save, sender=Comment)@receiver(post_delete, sender=Comment)def invalidate_last_comments(sender, **kwargs): cache.delete('last_comments')
![Page 26: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/26.jpg)
Caching!
Django Cacheback
from cacheback.decorators import cacheback
@cachebackdef fetch_last_comments_ids():
...
def last_comments(request):comments_ids = fetch_last_comments_ids()comments = Comment.objects.filter(id__in=comments_ids)
...
![Page 27: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/27.jpg)
Don’t forget about the browser
80% or more of the end-user response time is spent in the front end
CombineCompress
CacheLess is more
Load slow things laterFocus on making the
important things faster
![Page 28: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/28.jpg)
Don’t forget about the browser
Google PageSpeed Insights
webpagetest.org
Chrome & Firefox
![Page 29: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/29.jpg)
Assets
Django Compressor{% compress js %}<script src="/static/js/one.js"></script><script>obj.value = "value";</script>{% endcompress %}
Django Assets
{% assets "js_all" %} <script async src="{{ ASSET_URL }}"></script>{% endassets %}
![Page 30: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/30.jpg)
Assets
aload.js
<script data-aload="http://foo.com/foo.js"></script><link data-aload="http://foo.com/foo.css" rel="stylesheet">
![Page 31: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/31.jpg)
Assets
Shameless plug: django-critical
{% critical %}<link rel="stylesheet" href="bootstrap.min.css">{% endcritical %}
Alpha stage, use at your own risk!
![Page 32: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/32.jpg)
Q & Maybe A
![Page 33: Speed is a Feature - PyConAr 2014](https://reader030.fdocuments.us/reader030/viewer/2022032616/55a4244c1a28abae278b4859/html5/thumbnails/33.jpg)
ReferencesPerformance is a feature - http://blog.codinghorror.com/performance-is-a-feature/
Marissa Mayer at Web 2.0 - http://glinden.blogspot.com.ar/2006/11/marissa-mayer-at-web-20.html
Psychology of Web Performance - http://www.websiteoptimization.com/speed/tweak/psychology-web-performance/
New Relic - http://newrelic.com
Google Analytics - http://www.google.com/analytics/
Tracking Application Response Time with Nginx - http://lincolnloop.com/blog/tracking-application-response-time-nginx/
Logging Apache response times - http://www.moeding.net/archives/33-Logging-Apache-response-times.html
Django Debug Toolbar - http://django-debug-toolbar.readthedocs.org/
DDT Template Timings - https://github.com/orf/django-debug-toolbar-template-timings
Django Database Optimizations - https://docs.djangoproject.com/en/1.7/topics/db/optimization/
Two Hard Things - http://martinfowler.com/bliki/TwoHardThings.html
Django Cache Docs - https://docs.djangoproject.com/en/1.7/topics/cache/
Django Cacheback - http://django-cacheback.readthedocs.org/en/latest/index.html
Cached Templates - https://docs.djangoproject.com/en/1.7/ref/templates/api/#django.template.loaders.cached.Loader
Google PageSpeed Insights - https://developers.google.com/speed/pagespeed/insights/
Web Page Test - http://www.webpagetest.org/
Django Compressor - http://django-compressor.readthedocs.org/en/1.3/
Django Assets - http://django-assets.readthedocs.org/en/0.8/
aload.js - https://github.com/pazguille/aload
django-critical - https://github.com/martinblech/django-critical
Demo - https://github.com/martinblech/pyconar2014_perfdemo