Practical project automation (PyGrunn conference)

Practical project automation

donderdag 19 mei 2011


• Reinout van Rees

• Nelen & Schuurmans

• Utrecht

• Water consultancy

• Lots of python and django


Project: more than code

• Projects are important

• Deployment

• Colleagues

• Rebuilding

• Solution: automation


Generic idea: automation

• Cobbler’s children

• Automate your projects

• Less manual work

• Less menial work

• Fewer errors

• More of the fun work


Build automation

• ./configure;make

• Ant, maven

• Buildout (python)

• Single command to get started


Build tool: good spot for more automation

• Most are extensible

• Examples:

• Apache config file generation

• Setting up cronjobs


Apache config file:[apacheconf]recipe = collective.recipe.templateinput = ${buildout:directory}/etc/mysite.apache.conf.inoutput = ${buildout:directory}/etc/mysite.apache.conf

<VirtualHost *:80> ServerName mysite.example.org CustomLog ${buildout:directory}/var/log/access.log combined ErrorLog ${buildout:directory}/var/log/error.log DocumentRoot ${buildout:directory}/var/www ... RewriteRule ^(.*) http://localhost:${django:gunicorn_port}$1 [P]

<VirtualHost *:80> ServerName mysite.example.org CustomLog /srv/mysite/var/log/access.log combined ErrorLog /srv/mysite/var/log/error.log DocumentRoot /srv/mysite/var/www ... RewriteRule ^(.*) http://localhost:10003$1 [P]


http://localhost

http://localhost

http://localhost

http://localhost

Prevent missing apache log dir (automatically!)

[mkdir]recipe = z3c.recipe.mkdirpaths = ${buildout:directory}/generated ${buildout:directory}/var/geotiffs ${buildout:directory}/var/log ${buildout:directory}/var/media ${buildout:directory}/var/sqlite ${buildout:directory}/var/static


Automatic crontab

[supervisor-cronjob]recipe = z3c.recipe.usercrontabtimes = @rebootcommand = ${buildout:bin-directory}/supervisord

[whoosh-cronjob]recipe = z3c.recipe.usercrontabtimes = 13 */8 * * *command = ${buildout:bin-directory}/django update_index


Test automation

• Jenkins/buildbot

• Deployment ok...

• ... so tests possible

• Prevent mistakes


Extra jenkins automation: quality checks

• Code coverage (‘coverage’ on pypi)

• Pyflakes and pep8 (also on pypi)

• Jslint (or jshint)


Quality: make it easy to do the right thing

• “I really should do x, but it takes too much time so I’ll skip it”

• Releasing packages

• Skeleton


Releasing python packages

• zest.releaser (on pypi)

• prerelease

• release

• postrelease



• Make it easier to do the right thing: let laziness work for you

• Automate your builds

• Automate tests

• Extend where practical


Questions?

reinout.vanrees.org

@reinoutvanrees

[email protected]

www.nelen-schuurmans.nl

photos.reinout.vanrees.org


http://reinout.vanrees.org








Practical project automation (PyGrunn conference)

Technology

Transcript of Practical project automation (PyGrunn conference)