Performance Tuning and Optimization of High Traffic Drupal Sites
-
Upload
lucas-silva -
Category
Documents
-
view
16 -
download
0
description
Transcript of Performance Tuning and Optimization of High Traffic Drupal Sites
-
PerformanceTuningandOptimizationforhightrafficDrupal
sitesKhalidBaheyeldin
DrupalCamp,TorontoMay1112,2007
-
Agenda Introduction TheLAMPStack
Linux,Apache,MySQL,PHP Drupal
Databasequeries Modules Caching
Measurementandmonitoringtools Whatcangowrong? Questions,discussion
-
About2bits
BasedinWaterloo,Ontario ActivememberoftheDrupalcommunitysince2003 Memberofsecurityandinfrastructureteams 24+modulesondrupal.org ListedonDrupal.org'sserviceproviderssection Maintainmodulesthatrunondrupal.org(donations,feature,
lists) GoogleSummerofCodementoring(2005,2006,2007)
-
2bitsServices ClientsmainlyinUSAandCanada Subcontractingdevelopmentprojects Customizationofexistingmodules Developmentofnewmodules Installation,upgrades Automatedbackups Performancetuningandoptimization
-
AboutKhalid Developingforcomputersforwaytoo
long(22years),Drupalsince2003
Corecontributions
Sitemaintenancefeature LoggingandalertsinDrupal6 Severalpatches
Memberof
Drupalsecurityteam webmastersteam infrastructureteam
Cofounderof2bits Blogathttp://baheyeldin.com
Contributedmodules Adsense
Userpoints
Nodevote
Jobsearch
Favoritenodes
Flagcontent
StockAPIandmodule
CustomError
Currency
Imagewatermark
Sitemenu
Emailloggingandalerts
SecondLife
Technorati
Clickthru
Referral
-
TheIron Physicalservermatters
Dedicated VPS
Notapplicabletosharedhosting DualOpteronskickass LotsofRAM(cachingthefilesystemandthedatabase,asmuchas
possible) Multipledisksifyoucan Alwaysmirrored!
-
MultipleServers Onedatabaseserver+multiplewebservers CanuseDNSroundrobin Orproperloadbalancers(commercial,free) Evenareverseproxy(squid) Doitonlyifyouhavethebudget
Complexityisarunningcost Tuningasystemcanavoid(ordelay)thesplit
-
TheLAMPstack MostcommonlyusedstackforhostingDrupalandsimilarapplications Linux Apache MySQL PHP
Mostofthispresentationappliesto*BSDaswell.PartsapplytoWindows.
-
Linux
Useaprovenstabledistro(Debian,Ubuntu) Userecentversions(noFedoraCore4please) Beaminimalist Installonlywhatyouneed
(e.g.NoX11,nodesktop,NoPostgreSQLifyouareonlyusingMySQL,...etc.)
Balancecompileyourownvs.upgrades
-
Apache Mostpopular,supportedandfeaturerich Otherwebservers
lighttpd(lighty) PopularwithRuby 1MBperprocess Recentmemoryleaks
nginx Morestablethanlighty(noleaks)
-
Apache Cutthefat
Enableonlymod_phpandmod_rewrite Disableeverythingelse(java,python) MayneedextendedstatusforMunin
TuneMaxClients Toolow:youcan'tserveatrafficspike(Digg,Slashdot)
Toohigh:yourmemorycannotkeepupwiththeload,andyoustartswapping(serverdies!)
-
Apache(cont'd) KeepAlive
5to10secondsOK Morethanthat,ittiesupprocesess
Allowoverrides SettoNone MoveDrupal's.htaccesscontentstovhosts
mod_gzip/mod_deflate CompromiseofCPUusagevs.Bandwidthusage
-
MySQL
MostpopulardatabaseforDrupal Notthebestdatabasefromthetechnologypointofview(ACID,transactions,concurrency),butstilladequateforthejob
Variouspluggableengines
-
MySQLEngines MyISAM
Fasterforreads Lessoverhead Poorconcurrency(tablelocking)
InnoDB Transactional Slowerinsomecases Betterconcurrency Oracleownstheenginenow...
-
MySQLEngines
Twonewengines,ownedbyMySQLAB Falcon.NotmatureenoughtomatchInnoDB,benchmarksshowitisstillslow
SolidDB. PBXT
PrimeBaseXT
-
MySQLtuning
Querycache Probablythemostimportantthingtotune
Tablecache Alsoimportant
Keybuffer
-
PHP
Usearecentversion InstallanOpcodecache/Accelerator
eAccelerator APC Xcache Zend(commerical)
APCvs.eAcceleratorbenchmarkon2bits.
-
Opcodecaches Benefits
Dramaticspeedupofapplications,speciallycomplexoneslikeDrupal
SignificantdecreaseinCPUutilization Considerabledecreaseinmemoryutilization Thebiggestimpactonabusysite
Drawbacks Maycrashoften UselogwatchertoautorestartApache
-
mod_php
Normally,Apachemod_phpisthemostcommonlyusedconfiguration
Sharednothing Nostateretainedbetweenrequests Lessissues
Staywithmod_phpifyoucan. Canbeaslowas1012MBperprocess Sawitashighasmid20s+
-
PHPasCGI
CGIistheoldestmethodfromtheearly90s. Forksaprocessforeachrequest,andhenceveryinefficient.
Somehostsofferitbydefault(security)orasanoption(e.g.runningaspecificPHPversion).
Don'tuseit!
-
FastCGI FCGIisfasterthanCGI(usesasockettothePHPprocess,notforking)
MostlywithLighttpdandnginx,sinceitistheonlywaytorunPHPforthoseservers,butalsowithApache
Therearesomecases(e.g.drupal.orgitself) Betterseparationofpermissions(e.g.Sharedhosting) IfyouhaveoneserverandoneLinuxuser,permissionsmaynotbeanissue.
-
Drupal
Mainlydatabasebottlenecks Bottlenecksareworkedonastheyarefoundbythecommunity
Somemodulesknowntobeslow Notallsitesaffectedbyallbottlenecks
-
Watchdog
Avoiderrors(404songraphics,favicon)TIME STATE INFO
24 updating DELETE FROM watchdog WHERE timestamp < 1176392718
24 Locked INSERT INTO watchdog (uid, type, message, severit
19 Locked INSERT INTO watchdog (uid, type, message, severit
14 Locked INSERT INTO watchdog (uid, type, message, severit
11 Locked INSERT INTO watchdog (uid, type, message, severit
6 Locked INSERT INTO watchdog (uid, type, message, severit
OptionalinDrupal6(syslogasanoption)
-
Sessions Heavilyusedinhightrafficsites
TIME STATE INFO
28 Locked UPDATE sessions SET uid = 0, hostname = '212.154.
28 Copying to t SELECT ... FROM sessions WHERE timestamp >= 11776
28 Locked SELECT ... FROM users u INNER JOIN sessions s ON
27 Locked UPDATE sessions SET uid = 0, hostname = '222.124.
27 Locked UPDATE sessions SET uid = 0, hostname = '201.230.
27 Locked SELECT ... FROM users u INNER JOIN sessions s ON
27 Locked SELECT ... FROM users u INNER JOIN sessions s ON
27 Locked SELECT ... FROM users u INNER JOIN sessions s ON
27 Locked SELECT ... FROM users u INNER JOIN sessions s ON
27 Locked SELECT ... FROM users u INNER JOIN sessions s ON
-
Drupal(cont'd) Disablemodulesthatyoudonotneed. Enablepagecaching
Mayexpiretoooftenonabusysite,causingslowdowns! Considercachingmodules
FSFastpath boost
Makesurecronrunsregulary Enablethrottle
Bewaryaboutthrottleandcache
-
Puggablecaching
Using$confvariableinsettings.php 'cache_include'=>'./includes/yourcache.inc'
Allowsyoutohaveacustomcachingmodule Cachingusingmemcachedisbeingworkedon Tip:canbeusedtodisablecachefordevelopment(stubfunctions)
-
Slowmodules Statisticsmodule
Addsextraqueries EvensloweronInnoDB(COUNT(*)slow) DisablePopularContentblock
gsitemap(XMLsitemap)
Hadanextrajoin,patchaccepted Aggregator2
Abandoned! Manymore...
-
MeasureandMonitor
Howdoyouknowyouhaveaproblem? Userscomplain(siteissluggish,timeouts)? Losingyouraudience?Lossofinterestfromvisitors?
Toolsforvarioustasks
-
Top ClassicUNIX/Linuxprogram Realtimemonitoring(i.e.WhatthesystemisdoingNOW)
Loadaverage CPUutilization(user,system,nice,idle,waitI/O) Memoryutilization Listofprocesses,sorted,withCPUandmemory Canchangeorderofsorting,aswellastimeinterval,andmanyotherthings
-
vmstat
FromBSD/Linux Showsaggregateforthesystem(noindividualprocesses)
Showssnapshotorincremental Processesintherunqueueandblocked Swapping CPUuser,system,idleandiowait
-
netstat
Showsactivenetworkconnections(allandESTABLISHED)
netstatanp netstatanp|grepEST
-
mtop,mytop
mtop Liketop,butforMySQL Realtimemonitoring(nohistory) Showsslowqueries
mytop Similartomtop
SHOWFULLPROCESSLIST
-
mysqlreport/dbtuning Mysqlreport
Perlshellscript Displaysstatistics Norecommendations
Dbtuning AshellscriptthatreadsvariablesfromMySQL Annoyinguseofcolors Usefulrecommendations
-
GraphmonitoringMunin
Niceeasytounderstandgraphs. Historyoveraday,week,monthandyear CPU,memory,network,Apache,MySQL,andmuchmore
CanaddyourownmonitoringscriptsCacti
Similarfeatures
-
Drupaltools Develmodule
Totalpageexecution Queryexecutiontime Querylog Memoryutilization
Tracemodule Morefordebugging,butalsousefulinknowingwhatgoesonunderthehood
-
Whatcangowrong?
CPUusageistoohighMemoryoverutilizationToomuchdiskI/OToomuchnetworktraffic
-
CPU
FindoutwhoisusingtheCPU?Findoutwhichtype(user,system,waitI/O)
-
CPU IfitisanApacheprocess,theopcodecachewillhelp,unless
youhaveabug. IfitisMySQL,thensomeofthatisnormal(intensivequeries),
otherwise tunetheindexes splittheservertotwoboxes. Tunethequerycache
Ifitissomethingelse,andconsistent,thenconsiderremovingit.
-
CPU100%
OutputfromToptop - 10:16:58 up 75 days, 59 min, 3 users, load average: 152.70, 87.20, 46.98
Tasks: 239 total, 157 running, 81 sleeping, 0 stopped, 1 zombie
Cpu(s):100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2075932k total, 1558016k used, 517916k free, 13212k buffers
Swap: 1574360k total, 49672k used, 1524688k free, 442868k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
659 www-data 21 0 61948 14m 4060 R 3 0.7 0:14.35 apache2
960 www-data 20 0 62084 14m 4076 R 3 0.7 0:10.51 apache2
989 www-data 20 0 62036 14m 4052 R 3 0.7 0:09.95 apache2
.... hundreds of them
-
CPU100%
Vmstatoutput# vmstat 15
procs -----------memory---------- ----cpu----
r b swpd free buff cache us sy id wa
152 0 40868 1190640 13740 465004 22 6 71 2
153 0 40868 1190268 13748 464996 100 0 0 0 155 0 40868 1189740 13756 464988 100 0 0 0 154 0 40868 1189540 13768 465044 100 0 0 0
-
CPU100% Whatwasit? eAccelerator(svn303+PHP
5) AttempttogetoverPHP
crashes NoteCPUutilization(100%,
thenhigh,thendroppedlowwhengoodversionused)
-
Memory
Swappingmeansyoudon'thaveenoughRAM Excessiveswapping(thrashing)isserverhell! ReducethesizeofApacheprocesses ReducethenumberofApacheprocesses(MaxClients) Turnoffprocessesthatarenotused(e.g.Java,extracopiesofemailservers,otherdatabases)
Buymorememory!Costeffectiveandworthit.
-
Memory
Impactonmemoryusagewhenthereisnoopcodecachevs.withanopcodecache
-
DiskI/O
Firsteliminateswappingifgethitbyit. Getthefastestdisksyoucan.7200RPMataminimum. TurnoffPHPerrorloggingto/var/log/*/error.log Considerdisablingwatchdogmoduleinfavorofsyslog(Drupal6willhavethatoption),orhackthecode
OptimizeMySQLonceaweek,oronceaday
-
Network
Normallynotanissue Occasionallyyouwillhaveastubborncrawlerthough
OrevenaDdoS Orworse,extortion Caneatupresources,includingnetwork
-
Diggfrontpage?
OnGoodFriday,adsoftheworld.comwasonDigg'sfrontpage. Thefounderwroteaboutit
http://creativebits.org/webdev/surviving_the_digg_effect Survivedthediggwell. Anotherserver(untuned)gotdiggedtwiceanddied
-
ResourcesandLinks
General http://2bits.com/articles/drupalperformancetuningandoptimizationforlargewebsites.html http://www.lullabot.com/articles/performance_and_scalability_seminar_slides
Apache http://httpd.apache.org/docs/2.0/misc/perftuning.html
MySQL http://www.mysqlperformanceblog.com/
http://dev.civicactions.net/moin/CodeSprint/SanFransiscoMarch2007/PerformanceAndScalabilitySeminar
-
Conclusion
Questions? Comments? Discussions?