delivering applicationswith zc.buildout

and a distributed modelTarek Ziadé <tarek.ziade@ingeniweb.com>

1

Who am I ?New to Plone (~1 year) -> I worked on peripheral matters Used to Zope (CPS core developer)

CTO at Ingeniweb - a Plone company (>20 developers)

Python fan - wrote some books about it(even in english, because you don’t hear my accent in a book)

involved in plone.org migration, PSC, zc.buildout

2

Special promotion during the PloneConf :buy one book == get a big hug

It will look good on your deskbesides Martin Aspeli’s book

3

Thanks ! * Andreas Jung * Veda Williams * David Glick * Jeff Kowalczyk * Youenn Boussard * Christian Klinger * Jesse Snyder * Alec Mitchell * John Habermann * Maurits van Rees * Jean-François Roche * Martin Aspeli * Alain Meurant * Aleksandr Vladimirskiy * Jon Stahl * Alexander Limi * Stephen McMahon

4

Part 1 - working with packages

Part 2 - working with zc.buildout

Part 3 - application lifecycle

5

Part 1 - working with packages

6

7

builds and distributes a packageregisters and uploads it to PyPI

distutils

7

8

get the screencast at http://ziade.org/ploneconf

8

9

distutils is

the standardin the whole Python community.

+1

9

10

distutils is

broken (but there’s hope)

-1

10

11

setuptools

simple dependencies management, namespaced package egg distribution

provides easy_install

11

12

“python setup.py bdist_egg”

one single .egg file

12

13

egg=

deployment formatin a zip archive

"Eggs are to Pythons as Jars are to Java..."

13

14

“sdist” is enough, though(and should be mandatory)

14

15

get the screencast at http://ziade.org/ploneconf

15

16

easy_install my_package=

will get and install “my_package”from “PyPI”

16

17

setuptools is

the de facto standardin the Plone community.

+1

17

18

setuptools is

brokentoo (pje says)

(but there’s hope)

-1

18

19

packaging future in Python

?

19

20

PJE Debian packagers BSD packagers Us !

DiscussionsRepoze guys

TG2 guys

Pylons guys

PEPs

distutils 2

Pythoncore

developers

Heaven

Zope 3 guys

Guido

Ian

20

21

Packaging future in Python

hard to know.but community-drivencome sprint with us saturday !

21

22

It’s OK, fellowswe will eventuallymake it through.

22

23

#1 PyPI == SPOF

#2 packages need privacy sometimes

#3 plone.org/products is *dying*

Problems with packaging ?

23

24

Make a PyPI mirror

Run you own private PyPI

Make plone.org/products PyPI compatible

Solutions ?

#1 PyPI == SPOF

#2 packages need privacy sometimes

#3 plone.org/products is *dying*

24

25

easy_install collective.eggproxy

#1 Make a PyPI mirror : Smart mirroring

25

26

26

27

all calls will go through the proxythe mirror is filled on-demand

In zc.buildout : the index option[buildout]

index = http://my.mirror:8888

27

28

#2 Run your own private PyPI

PloneSoftwareCenter !

28

29

get the screencast on http://ziade.org/ploneconf

29

30

collective.dist

python 2.6 new “register” and “upload” commands

“mregister” and “mupload” in 2.4 and 2.5

$ python setup.py mregister sdist mupload$ python setup.py mregister sdist mupload -r plone.org$ python setup.py mregister sdist mupload -r my-company...

30

31

Developer side : multiple servers with .pypirc[distutils]index-servers = pypi plone.org my-company

[pypi]username:tarekpassword:hahaha

[plone.org]username:tarekpassword:hahaha...repository:http://plone.org/products

[my-company]username:tarekpassword:hahaha..hahaha.haha...ha..pfiiiuuuuuu..and..hahahaahah.i..am..dying..repository:http://my-company.com/products

31

32

Let’s try our new PyPIwith collective.dist

32

33

get the screencast on http://ziade.org/ploneconf

33

34

#3 Make plone.org/productsPyPI compatible

plone.org was suppose

to switch to that for months.Lazy guys

34

35

Let’s try new.plone.org

35

36

get the screencast on http://ziade.org/ploneconf

36

37

The big picture

37

38

Yes !!!

developer #138

39

Yepeee !!!Releasing everywhere

let’s spam PyPI and plone.orgtchak ! tchak ! tchak !

developer #2

39

40

Hahaha !

We have our package all over the place.

developer #3

40

41

end of part 1

questions ?

#1 Make mirror

#2 Run your own PyPI

#3 push to several servers

#3 use “mregister” and “mupload”

41

42

Part 2 - working with zc.buildout(reminders)

42

43

5 hours in 2006

43

44

5 hours in 2006get zopeinstall zope

install python extra packages

get extra products

install extra products dependency

doesn’t work

ahhh right, install python-ldap

create an instance

checkout products in development

start to work

doesn’t work

ahhh right, wrong python-ldap version

read extra products doc

install extra products

44

45

Me in 2006

45

46

5 minutes in 2008

46

47

5 minutes in 2008get the “buildout”$ python bootstrap.py$ bin/buildoutstart to work

47

48

But that was not the main reason

48

49

Zopeone big-Opackage.

zope.interface

zope.schemas

zope.stuff

zope.whatsthat

zope.makecoffee

zope.dodishes

“eggification”

over 100 packages...

2005 Now

49

50

Ploneproducts

plone.ista

plone.portlets

plone.stuff

plone.whatsthat

plone.makecoffee

plone.enolp

“eggification”

...

2005 Now

50

51

YourKillerPloneApp

app.site

app.skin

...

51

52

YourDistribution

app.site

app.skin

zope.stuff

plone.stuff

...

...

52

+100 packagesto run Plone

Pythonsite-packages

Increase conflictprobabilities

something is going wrong here

53

54

zc.buildout =

isolated environment

54

55

zc.buildout best practices

#1 use the same layout for all your projects

#2 make sure all developers have the same environment

#3 use one cfg per target

55

56

project1 docs buildout packages releases

project2 docs buildout packages releases....

#1 same layout for all projects

56

57

easy_install collective.releaser

57

58

get the screencast on http://ziade.org/ploneconf

58

59

#2 make sure all developers have the same environment

Plone buildouts are source based Warning

Get my Windows installer : python2.4.4-win32.zipGoogle “An installer for a buildout-ready Windows”

Windows developers

59

60

Typical buildout layout uses the extends feature

dev.cfg (extends buildout.cfg)buildout.cfg

prod.cfg (extends buildout.cfg)+ bootstrap.py

#3 use one cfg per target

60

61

[buildout]extends = buildout.cfgparts = threedevelop = ...

[buildout]parts = one two

buildout.cfg :

dev.cfg :

[buildout]parts = one two threedevelop = ...

61

62

democreating a fresh Plone 3 buildout (Paste)adding the dev.cfghooking a new development packageadding a prod.cfg

62

63

get the screencast on http://ziade.org/ploneconf

63

64

end of part 2

questions ?

#1 use the same layout for all projects

#2 make sure all developers have the same environment

#3 use one cfg per target

64

Part 3 - application lifecycle

65

66

Releasing packages Releasing the buildout

67

68

Releasing packages

for package in packages: raise the version edit CHANGES.txt create a branch (svn) go to that branch remove the dev tag (setup.cfg) release it with “mregister sdist mupload -r somehwere” release it with “mregister sdist mupload -r somehwereelse”

68

69

Releasing packages with collective.releaser:

for package in packages: python setup.py release

69

70

Defining targets in .pypirc

[distutils]

index-servers = pypi plone.org private

glob-style = true

[pypi]...release-command = mregister sdist muploadrelease-packages = iw.*

[plone.org]...release-command = mregister sdist muploadrelease-packages = plone.* collective.*

[private]...

70

71

get the screencast on http://ziade.org/ploneconf

71

72

collective.releaser makes a lot of assumptionsit’s svn-oriented (the first use case)

72

73

- recipes

- your released packages - exceptions (security fixes, major bug fixes)

What packages should be frozen ?

Release the buildout

73

74

authentication: use lovely.buildouthttp

Release the buildout

[buildout]...extensions=lovely.buildouthttp...repository:http://my-company.com/products

pypi,http://my-company.com/products,tarek,hahahapypi,http://plone.org/products,tarek,hahaha

HOME/.buildout/.httpauth

74

75

$ cd buildout$ svn cp . http://somewhere/releases/0.1

Release the buildout -> project layout

project ... buildout packages release/0.1 <- tag for the buildout

75

76

Release the buildout with collective.releaser :with project_release

$ cd buildout$ project_releaseWhat version you are releasing ? 0.1Added version file.

76

77

- bin/buildout on target system

- remove some stuff

- offline mode to ‘true’ - tar -czvf release-0.1.tgz release/0.1

Build your release distribution

77

78

$ svn co http://somesvn/my_projet/releases/0.1 project$ cd project$ project_deploy prod.cfg

Build your distribution with collective.releaserwith project_deploy

78

79

Upgrading the application

79

80

Upgrading the application : 3 strategies

#1 full release#2 diff release, using collective.releaser#3 egg pack release, using collective.releaser

80

81

collective.releaser :

the releaser_project templateproject_releaseproject_deployproject_eggs

project_diff

81

82

That’s it ! Thanks !

questions ?

82

83

http://tarekziade.wordpress.com

zc.buildout @ PyPI

collective.dist @ PyPI

collective.releaser @ PyPI

http://alterway.fr

83