Drupal performance and scalability
description
Transcript of Drupal performance and scalability
DRUPAL PERFORMANCE E SCALABILITÀSTEFANO MAINARDI - [email protected] GIACOMASSI - [email protected]
About Stefano
• Drupal developer since 2007
• Twinbit co-founder
• ILDN founder
• I’m a geek and i love Open Source software
@stefanomainardi
About Marco
• web and open source consultant and developer
• interested in knowledge management, GIS, and integration of systems with CMS
• Twinbit co-founder
@marcogiaco
Agenda
• Why speed is so important
• Tips for frontend optimization
• Tips for backend optimization
• Q&A time (we hope!)
What is performance?
“The delay perceived between an action (e.g. click) and a meaningful response”
What is performance?
Performance is money
Why do we care about it?
Why speed is so important?
some numbers
1 Seconddecrease page load time
Amazon's calculated that a page load slowdown of just
one second could cost it $1.6 billion in sales each
year.
Source: http://www.tagman.com/mdp-blog/2012/03/just-one-second-delay-in-page-load-can-cause-7-loss-in-customer-conversions/
Google has calculated that by slowing its search results
by just four tenths of a second they could lose 8 million searches per day
Source: http://www.tagman.com/mdp-blog/2012/03/just-one-second-delay-in-page-load-can-cause-7-loss-in-customer-conversions/
WHAT??
Performance golden rule
80-90% of the end-user response time is spent on the frontend.
Start there.
80-90% of the end-user response time is spent on the frontend.
Source: http://www.stevesouders.com/blog/2012/02/10/the-performance-golden-rule/
But first, the horrible truth about Drupal
Drupal isDatabase intensive!!
Memory intensive!!
Can easily become a resource hog
Why most drupal sites are slow?
Poor frontend implementation!!
Slow mysql queries!!
Serving dynamic contents to anonymous !users!!
Module bloat (“open buffet” syndrome)
Let’s focus on frontend
1. When we request a URL a DNS lookup is done
2. Download the html and start reads from top
3. CSS Block rendering. The browser starts rendering once it has all the style declarations
4. Javascript blocks downloads. Make sure to serve js at last
5. Circa 4/8 assets (images / js / css /fonts) are downloaded in parallel from the same domain
www.twinbit.it ? IP = 188.40.59.145
How does a browser work?
Our goals
1. Put CSS on top (it blocks rendering)
2. Put JS on bottom (it blocks downloads)
3. Minimize the number of requests (CSS / js / images, fonts)
4. Send less data as possible
5. Spread your assets over several domains (CDN)
Put javascript on footer
Remove unneeded cssUsing hook_css_alter() to limit amount of requests
Remove unneeded cssUsing hook_css_alter() to limit amount of requests
Use aggregation
Built-in aggregation
Use aggregation
Advanced CSS/JS Aggregation modulehttp://drupal.org/project/advagg
or smart solution
Use aggregationAdvanced CSS/JS Aggregation module
http://drupal.org/project/advagg
Fully cached CSS/JS assets allow for zero file I/O if the Aggregated file already exists. Results in better page generation performance
On demand generation of CSS/JS Aggregates. If the file doesn't exist it will be generated on demand.
Gzip support. All aggregated files can be pre-compressed into a .gz file and served from Apache. This is faster then gzipping the file on each request.
Use aggregationAdvanced CSS/JS Aggregation module
http://drupal.org/project/advagg
built-in support for !!
HTTP Parallel Request & Threading Libraryhttps://drupal.org/project/httprl
Image requests
1. Sprites: One file for all UI elements (1 request)
2. Use automatic tools like Compass for generate sprites
http://compass-style.org/help/tutorials/spriting/
3. Optimize images https://drupal.org/project/imageapi_optimize
4. Use font-based icons https://drupal.org/project/fontello
Keep in mind to send less data to servers, ever!
SPREAD your assets over several domains
use CDN modulehttps://drupal.org/project/cdn
2 tips1 . DNS Prefetching
2. Cookie-less domainset in settings.php the variable$cookie_domain = ‘www.twinbit.it’
<head> … <link rel=“dns-prefetch” href=“//script.google.com”> … </head>
Monitor your application
Monitor your application
YSlow
Chrome and Firefox developer tools, are your best friends
Google page speed tools
or use external services like
New relic
and know the tools
take away
“Cache saves the cache”cache everything at every level
Let’s focus on backend
thank you for now!
Caching
Anonymous vs authenticated
Identify the type of your application in order to apply appropriate caching policies. !The more static the application is the more it will be served by the “client-side” caches. !Highly dynamic application will need efficient “application-side” caching.
Clustered architecture
Reverse proxy
A proxy server that retrieves resources on behalf of a client from one or more servers. These resources are then returned to the client as though they originated from the server itself. !Varnish is widely adopted, open source and natively supported by Drupal 7.x from 300x to 1000x faster serves static pages and assets powerful configuration language ESI (https://drupal.org/project/esi) can act as load balancer
More links https://drupal.org/node/1054886 https://drupal.org/project/varnish https://www.varnish-cache.org/trac/wiki/VarnishAndDrupal
!!
Web servers
Drupal configuration / coding ! use static caching $foo = &drupal_static(__FUNCTION__);
use cache_set/cache_get disable unneeded modules avoid variable_set() in frontend pages
and cache invalidation keep app logic away from template, use
hook_preprocess functions don’t reinvent the wheel! api.drupal.org
is your best friend! keep drupal performance configuration
in settings.php !
System ! configure Apache MaxClients
((System RAM) – (RAM used by other processes)) / (httpd process size) = MaxClients
KeepAliveTimeout < 5 sec ! use APC
apc.stat ~ 0 apc._num_files_hint > 1000 !
use Nginx if no proxy or CND
More links https://groups.drupal.org/high-performance
Web servers: more is better
! VM-C1: 1 cpu, 2 GB ram VM-C2: 2 cpu, 4 GB ram VM-C3: 4 cpu, 8 GB ram
Web servers: more is much better
Session management
Drupal is saving sessions to database. This can be used in a cluster but we want to save database queries. !To do this we can use different solutions: !!!! Memcache Redis MongoDB
pros very easy and fast
very fast native replication
cons no native replication
no native replication cluster configuration
Drupal compatibility
mature medium medium
Application caching
Drupal has several caches, by default stored in database. To avoid loading the database we can still use different solutions: ! memcache / redis mongodb !!
More links https://drupal.org/project/memcache https://drupal.org/project/mongodb http://www.mongodb.org/ !!!
Memcache tips ! exclude cache_form and cache_views unless different bucket size php mod: Memcache (3.x) vs Memcached bucket size!!! ! module settings !memcache.hash_strategy consistent memcached.sess_consistent_hash 1 !
Drupal settings.php $conf['memcache_stampede_protection'] = TRUE; $conf['lock_inc'] = 'sites/all/modules/memcache/memcache-lock.inc';
Application caching: memcache
Database
MySQL !
3% core queries goes to slaves tag queries to be slave query (also using query_alter) tag queries to be slave query also using views UI !Tune innodb buffer size !SELECT CEILING(Total_InnoDB_Bytes*1.6/POWER(1024,3)) RIBPS FROM (SELECT SUM(data_length+index_length) Total_InnoDB_Bytes FROM information_schema.tables WHERE engine='InnoDB' and table_schema = ‘drupal_db’) A; !!Use MySQL query cache!
Control servers
control server: monitoring software (munin), phpmyadmin, configuration engine (Puppet)
! staging server: should contain a copy of the production
environment. This is used for testing and deploying. ! file storage: user files
Interesting tools: !http://gatling-tool.org/ http://newrelic.com/ !!!!!!
Performances
Anon users: !proxy (varnish): 2000 req/s anonymous users 60 apache processes per cpu core, 1 req/s per process !Auth users: !web server (apache): 14 apache processes per cpu core in virtual env, 1 req/s per process
(consider clustering overhead). Could be much better using other hypervisors, with WMWARE we reached18/20)
web server (apache): 40MB per process
Example
Target utenti (numero di sessioni utente aperte sul sistema in un giorno)
300000
Peso medio hit in KB 100
Page hits per utente (numero di pagine visitate da un utente) 10
Tot hits/giorno 3M
Carico
% utenti intervallo ore intervallo
secondi nell'intervallo
media (req/sec)
triangolare (req/sec)
normale (req/sec)
100% 4.5 16200 185 370 332
Example
Infrastruttura
VM Tipo CPU cores RAM (GB) NumeroTot. CPU cores
Tot. RAM (GB)
haproxy server 2 4 2 4 8
front-end serversfrontend, cpu bounded 4 8 6 24 48
cache serversmemory bound 2 4 3 6 12
mysql serversmemory bound 4 14 2 8 28
control server 2 4 2 4 8
staging environment 2 4 6 12 24
totale risorse 21 58 128
Processi (threads) apache per cpu core 15
Drupal memory footprint (MB)
64
Mysql connection fooprint (MB)
16
Example
• caching is key to performance • set your performance target (capacity plan) • measure performance on changes
Conclusions
Grazie!
Grazie!