Modern, scalable deployment for plone
-
Upload
christian-theune -
Category
Technology
-
view
858 -
download
1
description
Transcript of Modern, scalable deployment for plone
Tuesday, October 16, 12
Tuesday, October 16, 12
Tuesday, October 16, 12
Tuesday, October 16, 12
Infrastructure
Tuesday, October 16, 12
EntropyTuesday, October 16, 12
Platform
Man
agem
ent
Virtualization
Components
OS
Aut
omat
ion
Mon
itori
ng
Hardware
Data center
Application
Supp
ort
Tuesday, October 16, 12
Platform
Man
agem
ent
Virtualization
Components
OS
Aut
omat
ion
Supp
ort
Mon
itori
ng
Hardware
Data center
Application
Tuesday, October 16, 12
Data center
Tuesday, October 16, 12
Platform
Man
agem
ent
Virtualization
Components
OS
Aut
omat
ion
Supp
ort
Mon
itori
ng
Hardware
Application
Data center
Tuesday, October 16, 12
Hardware
Tuesday, October 16, 12
Platform
Man
agem
ent
Virtualization
Components
OS
Aut
omat
ion
Supp
ort
Mon
itori
ng
Data center
Application
Hardware
Tuesday, October 16, 12
• Virtualization is very affordable
• Platform creates interface between app and infrastructure
Virtualization
Components
OS
Tuesday, October 16, 12
Platform
Man
agem
ent
Aut
omat
ion
Supp
ort
Mon
itori
ng
Hardware
Data center
Application
Virtualization
Components
OS
Tuesday, October 16, 12
• convergence
• idempotency
• versioning
• everything as code
• continuous repairs
Automation
Tuesday, October 16, 12
Platform
Man
agem
ent
Virtualization
Components
OS
Aut
omat
ion
Supp
ort
Mon
itori
ng
Hardware
Data center
Application
Tuesday, October 16, 12
• Take care of updates and “production-ready” configuration
• Backup and other day-to-day operations
• OS and regularly used components
• Gradual process moving components from user-space installations to platform-wide features
Management
Tuesday, October 16, 12
Platform
Man
agem
ent
Virtualization
Components
OS
Aut
omat
ion
Supp
ort
Mon
itori
ng
Hardware
Data center
Application
Tuesday, October 16, 12
• Nagios :(
• (Un-)fortunately: #monitoringsucks
Monitoring
Tuesday, October 16, 12
Platform
Man
agem
ent
Virtualization
Components
OS
Aut
omat
ion
Supp
ort
Mon
itori
ng
Hardware
Data center
Application
Tuesday, October 16, 12
• Request tracker
• custom glue code
• Daily review
• Interrupt shield
Support
Tuesday, October 16, 12
Platform
Components
Man
agem
ent
Virtualization
OS
Aut
omat
ion
Supp
ort
Mon
itori
ng
Hardware
Data center
Application
Tuesday, October 16, 12
Application deployment and operations
Tuesday, October 16, 12
Deployment
• Quick
• Repeatable
• Platform-independent
• single-click
Tuesday, October 16, 12
Batou
• Model-based
• Use everywhere
• No effect on runtime
• Orchestration
• Convergency
• Idempotency
• Reusability
Tuesday, October 16, 12
class Solr(Component):
def configure(self): self.address = Address(self.host.fqdn, 9000) self.provide('solr:server', self.address)
self += Buildout('solr', python='2.7')
self += Program('solr', command='java', command_absolute=False, args='-jar start.jar', directory=self.expand( '{{component.workdir}}/parts/instance'), restart=True)
Tuesday, October 16, 12
[hosts]oshaweb00 = nginx, www, varnish, supervisor, haproxy, docsoshaweb01 = zope, hwthemes, supervisor...oshaweb20 = zope, hwthemes, supervisoroshaweb21 = zeo, openldap, supervisoroshaweb22 = nginx, media2oshaweb23 = nginx, birt, lms, ugm, supervisoroshaweb24 = solr, memcached, supervisoroshaweb26 = mailout
[component:haproxy]strategy = production_strategybots = 3whitelist = 8default = 3...
Tuesday, October 16, 12
Components
goodnginx
varnish
haproxy
PostgreSQL
memcached
supervisor
postfix
badapache
mysql, BDB
OpenLDAP
Tuesday, October 16, 12
Minimal "production-ready" checklist
• Sane front-end configuration
• System startup/shutdown
• Log rotation
• Database maintenance
• Monitoring: processes and ports
• Service user
Tuesday, October 16, 12
Secrets
• One AES-encrypted config file per environment
• Transparent decryption
• Safety-belt to avoid accidental checking in unencrypted files
• Future: PGP-encryption
Tuesday, October 16, 12
Environments
• Production vs. Staging
• Structurally identical but smaller
• Only one of many (dev, integration, ...)
• "Fly what you test, test what you fly."
Tuesday, October 16, 12
Web applications
• Shared nothing in your app!
• Many small processes
• Do not listen early
• Never store runtime config in the database
Tuesday, October 16, 12
Web applications II
• Log long running requests
• Avoid synchronous external requests
• Avoid accidental write requests.
• Use feature switches.
Tuesday, October 16, 12
Process control
• only one component to integrate
• turn anything into a daemon
• use superlance (httpok, memmon) for active restarts
• use 3.0 release candidate
Tuesday, October 16, 12
Python: virtualenv and buildout
• use both, combined
• buildout might be a hammer, but not everything is a nail
Tuesday, October 16, 12
more buildout
• use extends to keep your configs clean and structured
• allow-picked-versions=False
• -t 3
• buildout >= 1.6
• mr.developer
• allowed-hosts = ...
Tuesday, October 16, 12
and even more buildout
• multiple small better than one big
• different Python versions
• different subsets of egg versions
• use mirrors - see pypi-mirrors.org
• latency to the PyPI server is the biggest factor in buildout performance
Tuesday, October 16, 12
Databases
• As much RAM as you can afford - seriously
• Low latency network (<0.1ms)
• Multi-core if necessary
• Disk usage < 40%
Tuesday, October 16, 12
story: more hardware = slower
• Application is slow.
• Customer buys more hardware. Specifically: RAM.
• System is slower now.
• System runs many components. Hard to see through.
Tuesday, October 16, 12
story: more hardware = slower
• One number stands out: process exit rate is 20/sec. PosgreSQL. What?
• Customer creates new connection for each query.
• PostgreSQL forks master for new connection.
• Fork went from few 10ms to few 100ms
Tuesday, October 16, 12
story: more hardware = slower
• Diagnose. Find root cause.
Tuesday, October 16, 12
HTTP/S termination
• HTTP and SSL
• Virtual hosting
• Redirects
• host normalization
• Proxy to load balancer
• Static files
Tuesday, October 16, 12
Load balancing
• LB + HA
• Classify requests
• redispatch + retries
• balance: leastconn
• minconn 1, maxconn 2, threads +2
• Live statistic UI
Tuesday, October 16, 12
Monitoring
• again :(
• processes, ports, logs
• model dependencies
• check_webpage.rb
• Business processes (BPI)
Tuesday, October 16, 12
Reporting
• report + phone call
• incidents
• changes
• availability
• performance
• trends
Tuesday, October 16, 12
SLAs
• Availability
• more keeps your business safe
• less reduces costs
• Friends don't let friends think in "nines".
• Alignment of technical and organisational measures
Tuesday, October 16, 12
References
• "The practice of system and network administration" (Limoncelli, et. al.)
• Mark Burgess on config management
• "The twelve-factor app" (Heroku)
• gocept.net/doc
Tuesday, October 16, 12
Conclusion I
• Use infrastructure to build platforms
• Platforms should be transparent.
•Deploy applications on platforms.
Tuesday, October 16, 12
Conclusion II
• Do not cobble together random stuff from the internet.
• Careful work pays. Buzzwords don't.
• If you hear "enterprise". Run.
• Sapere aude.
Tuesday, October 16, 12
Conclusion III
• I don't think IaaS and PaaS in the current extremes are a good answer to many projects.
• Many people experiment with ideas. Everything is becoming more flexible.
• Let's play!
Tuesday, October 16, 12
Questions?Tuesday, October 16, 12