Paver For PyWorks 2008

Smoothing Software Project Scripting

Kevin DangoorBlueSkyOnMars.comphp|works PyWorks 2008

Python is not compiled

(actually, it is, but that’s not important right now)

Python projects certainly do not need• Redistributable files to move

around

• Deployment to other machines

• Interaction with source control systems

• Documentation that’s built from the source code

• Documentation that’s built from a format other than its final display format

If only there was some scripting

language that we already knew.

Waitasec. Isn’t Python a scripting

language?

Project-related scripts need...

• Command line argument handling

• Configuration

• Often work with files

• Sometimes need to use distutils/setuptools, but wish they could do a little more

• Need to work with other common tools (Sphinx, svn, virtualenv)

Command-line argument handling

bin/start-server --port 8675309

Configuration

[messages]greeting=Hello

[thing1]who=Worldmessage=${messages.greeting}, ${who}

Lots of working with files

import os

if not os.path.exists(“foo”): os.mkdir(“foo”)if not os.path.exists(os.path.join(“foo, “bar”)): open(os.path.join(“foo”, “bar”), “w”).write(“Hi”)

Working with distutils/setuptools

#!/bin/sh

sphinx-build (blah blah blah)python setup.py sdist upload

Other common tools

sphinx-build ...

subprocess.Popen(“svn info”... # and do something with the output

virtualenv.create_bootstrap_script(“# more code”)

Do you really want separate scripts for

everything?• /usr/local/bin/git /usr/local/bin/git-merge-resolve

• /usr/local/bin/git-add /usr/local/bin/git-merge-stupid• /usr/local/bin/git-add--interactive /usr/local/bin/git-merge-subtree

• /usr/local/bin/git-am /usr/local/bin/git-merge-tree

• /usr/local/bin/git-annotate /usr/local/bin/git-mergetool

• /usr/local/bin/git-apply /usr/local/bin/git-mktag

• /usr/local/bin/git-archimport /usr/local/bin/git-mktree

• /usr/local/bin/git-archive /usr/local/bin/git-mv

• /usr/local/bin/git-bisect /usr/local/bin/git-name-rev

• /usr/local/bin/git-blame /usr/local/bin/git-pack-objects

• /usr/local/bin/git-branch /usr/local/bin/git-pack-redundant

• /usr/local/bin/git-bundle /usr/local/bin/git-pack-refs

• /usr/local/bin/git-cat-file /usr/local/bin/git-parse-remote

• /usr/local/bin/git-check-attr /usr/local/bin/git-patch-id

• /usr/local/bin/git-check-ref-format /usr/local/bin/git-peek-remote

• /usr/local/bin/git-checkout /usr/local/bin/git-prune

• /usr/local/bin/git-checkout-index /usr/local/bin/git-prune-packed

• /usr/local/bin/git-cherry /usr/local/bin/git-pull

• /usr/local/bin/git-cherry-pick /usr/local/bin/git-push

• /usr/local/bin/git-citool /usr/local/bin/git-quiltimport

• /usr/local/bin/git-clean /usr/local/bin/git-read-tree

• /usr/local/bin/git-clone /usr/local/bin/git-rebase

• /usr/local/bin/git-commit /usr/local/bin/git-rebase--interactive

• /usr/local/bin/git-commit-tree /usr/local/bin/git-receive-pack

• /usr/local/bin/git-config /usr/local/bin/git-reflog

• /usr/local/bin/git-convert-objects /usr/local/bin/git-relink

• /usr/local/bin/git-count-objects /usr/local/bin/git-remote

• /usr/local/bin/git-cvsexportcommit /usr/local/bin/git-repack

• /usr/local/bin/git-cvsimport /usr/local/bin/git-repo-config

• /usr/local/bin/git-cvsserver /usr/local/bin/git-request-pull

• /usr/local/bin/git-daemon /usr/local/bin/git-rerere

• /usr/local/bin/git-describe /usr/local/bin/git-reset

• /usr/local/bin/git-diff /usr/local/bin/git-rev-list

• /usr/local/bin/git-diff-files /usr/local/bin/git-rev-parse

• /usr/local/bin/git-diff-index /usr/local/bin/git-revert

• /usr/local/bin/git-diff-tree /usr/local/bin/git-rm

• /usr/local/bin/git-fast-import /usr/local/bin/git-runstatus

• /usr/local/bin/git-fetch /usr/local/bin/git-send-email

• /usr/local/bin/git-fetch--tool /usr/local/bin/git-send-pack

• /usr/local/bin/git-fetch-pack /usr/local/bin/git-sh-setup

• /usr/local/bin/git-filter-branch /usr/local/bin/git-shell

• /usr/local/bin/git-fmt-merge-msg /usr/local/bin/git-shortlog

• /usr/local/bin/git-for-each-ref /usr/local/bin/git-show

• /usr/local/bin/git-format-patch /usr/local/bin/git-show-branch

• /usr/local/bin/git-fsck /usr/local/bin/git-show-index

• /usr/local/bin/git-fsck-objects /usr/local/bin/git-show-ref

• /usr/local/bin/git-gc /usr/local/bin/git-ssh-fetch

• /usr/local/bin/git-get-tar-commit-id /usr/local/bin/git-ssh-pull

• /usr/local/bin/git-grep /usr/local/bin/git-ssh-push

• /usr/local/bin/git-gui /usr/local/bin/git-ssh-upload

• /usr/local/bin/git-hash-object /usr/local/bin/git-stash

• /usr/local/bin/git-http-fetch /usr/local/bin/git-status

• /usr/local/bin/git-imap-send /usr/local/bin/git-stripspace

• /usr/local/bin/git-index-pack /usr/local/bin/git-submodule

• /usr/local/bin/git-init /usr/local/bin/git-svn

• /usr/local/bin/git-init-db /usr/local/bin/git-svnimport

• /usr/local/bin/git-instaweb /usr/local/bin/git-symbolic-ref

• /usr/local/bin/git-local-fetch /usr/local/bin/git-tag

• /usr/local/bin/git-log /usr/local/bin/git-tar-tree

• /usr/local/bin/git-lost-found /usr/local/bin/git-unpack-file

• /usr/local/bin/git-ls-files /usr/local/bin/git-unpack-objects

• /usr/local/bin/git-ls-remote /usr/local/bin/git-update-index

• /usr/local/bin/git-ls-tree /usr/local/bin/git-update-ref

• /usr/local/bin/git-mailinfo /usr/local/bin/git-update-server-info

• /usr/local/bin/git-mailsplit /usr/local/bin/git-upload-archive

• /usr/local/bin/git-merge /usr/local/bin/git-upload-pack

• /usr/local/bin/git-merge-base /usr/local/bin/git-var

• /usr/local/bin/git-merge-file /usr/local/bin/git-verify-pack

• /usr/local/bin/git-merge-index /usr/local/bin/git-verify-tag

• /usr/local/bin/git-merge-octopus /usr/local/bin/git-whatchanged

• /usr/local/bin/git-merge-one-file /usr/local/bin/git-write-tree

• /usr/local/bin/git-merge-ours /usr/local/bin/gitk

• /usr/local/bin/git-merge-recursive

easy_install Paver

paver paverdocs

pavement.py

• You already know Python

• The language rules are well-defined

• The language rules are well-documented

• Python is powerful, so you’ll never be left hanging or need an escape hatch

Why Python build files?

Pavements are almost standard Python

from paver.defaults import *

Configuration

options( setup = setup_meta, minilib=Bunch( extra_files=['doctools', 'virtual'] ), sphinx=Bunch( builddir="build", sourcedir="source" ), virtualenv=Bunch( packages_to_install=["nose", "sphinx", "docutils", "virtualenv"], install_paver=False, script_name='bootstrap.py', paver_command_line=None ))

Dynamic config values

>>> from paver.defaults import *>>> import time>>> options(current=lambda: time.time())>>> options.current1216726815.0027969

Namespace searching

>>> options(... setup=Bunch(version="1.0"),... sphinx=Bunch(builddir="docbuild")... )>>> options.version'1.0'

Namespace searching (continued)

>>> options(... setup=Bunch(version="1.0"),... sphinx=Bunch(builddir="docbuild")... )>>> options.order('sphinx')>>> options.versionTraceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/admin/projects/paver/paver/runtime.py", line 31, in __getattr__ raise AttributeError(name)AttributeError: version

Configuration is still standard

Python• You can treat options like a normal,

nested dictionary

• The only unusual thing would be that callables are called.

Tasks

@taskdef clean(): """Cleans up this paver directory. Removes the virtualenv traces and the build directory.""" pass

paver help

$ paver help---> helpPaver 0.8.1

Usage: paver [global options] [option.name=value] task [task options] [task...]

Run 'paver help [section]' to see the following sections of info:

options global command line optionssetup available distutils/setuptools taskstasks all tasks that have been imported by your pavement

'paver help taskname' will display details for a task.

Tasks defined in your pavement: bootstrap - Build a virtualenv bootstrap for developing

paver help tasks

Tasks defined in and imported by your pavement: bootstrap - Build a virtualenv bootstrap for developing paver clean - Cleans up this paver directory cog - Runs the cog code generator against the files matching your specification commit - Removes the generated code from the docs and then commits to bzr deploy - Copy the Paver website up doc_clean - Clean (delete) the built docs generate_setup - Generates a setup help - Displays the list of commands and the details html - Build Paver's documentation and install it into paver/docs minilib - Create a Paver mini library that contains enough for a simple

paver help <taskname>

Details for minilib:Create a Paver mini library that contains enough for a simple pavement.py to be installed using a generated setup.py. This is a good temporary measure until more people have deployed paver. The output file is 'paver-minilib.zip' in the current directory. Options: extra_files list of other paver modules to include (don't include the .py extension)

@needs

@task@needs("uncog")def commit(): """Removes the generated code from the docs and then commits to bzr.""" pass

call_task

call_task(‘commit’)

@cmdopts

@task@cmdopts([("username=", "u", "Username for remote server"), ("server=", "s", "Server to deploy to")])def deploy(): """Copy the Paver website up."""

pass

paver <taskname>

$ paver generate_setup minilib---> generate_setupWrite setup.py---> minilibGenerate paver-minilib.zip

Paver and Distutils/Setuptools

“Using the Distutils is quite simple, both for module

developers and for users/administrators installing

third-party modules.”– Distributing Python Modules, section 1.1

python setup.py install

Paver extends Distutils

paver install

setup.py example

from distutils.core import setupsetup(name='foo', version='1.0', py_modules=['foo'], )

Upgrading to Paver

options( setup = Bunch(name='foo', version='1.0', py_modules=['foo'], ))

Keeping it simple for users

$ paver generate_setup minilib---> generate_setupWrite setup.py---> minilibGenerate paver-minilib.zip

Generated setup.py

import osif os.path.exists("paver-minilib.zip"): import sys sys.path.insert(0, "paver-minilib.zip")

import paver.commandpaver.command.main()

The Paver Standard Library(is actually newer and less musty)

The Paver Standard Library

• Runtime helpers (paver.runtime)

• Distutils integration (paver.setuputils)

• File handling (paver.path)

• Documentation tools (paver.doctools)

• Subversion (paver.svn)

• SSH (paver.ssh)

• Virtualenv (paver.virtual)

• Miscellaneous Tasks (paver.misctasks)

paver.runtime

# display text if verbose is setdebug(“Hi there. Feeling chatty today?”)

# display text if quiet is not setinfo(“Glad we don’t have to keep quiet”)

# display text regardless of settingerror(“HA! I’M SHOUTING AND YOU CAN’T STOP ME!”)

paver.runtime (continued)

# run a command, as long as dry-run is off# capture the output into myvalmyval = sh(“cat /tmp/foo”, capture=True)

# run a function (delete_all(‘/’) to be exact). # if dry-run is set, then# just print the message insteaddry(“Delete everything”, delete_all, ‘/’)

paver.setuputils

options.setup.package_data = setuputils.find_package_data("paver", package="paver",

only_in_packages=False)

paver.path

• Jason Orendorff’s path.py module (also available separately)

• It’s a subclass of string!

• Fun use of operator overloading

• Lots of great methods

• Makes working with files/directories fun and easy

paver.path examples

p = path("docs")tmpdir = p / "tmp"tmpdir.mkdir()fn = tmpdir / "myfile.txt"fn.write_text("Hi there!")

paver.doctools

# add this to useimport paver.doctools

paver.doctools – Sphinx

paver.doctools.html()¶Build HTML documentation using Sphinx. This uses the following options in a “sphinx” section of the options.

docrootthe root under which Sphinx will be working. Default: docsbuilddirdirectory under the docroot where the resulting files are put. default: buildsourcedirdirectory under the docroot for the source files default: (empty string)

paver.doctools.doc_clean()¶Clean (delete) the built docs. Specifically, this deletes the build directory under the docroot. See the html task for the options list.

paver.doctools – Cog & SectionedFile• Solutions to common problems of

creating high-quality docs

• Ideally, your code samples will be in convenient runnable code files and have unit tests.

• But you also want nice, minimal code samples in your text as you’re writing.

paver.doctools sample code

# mysample.py

# [[[section mysample]]]def sample_func(): print "To sample, or not to sample?"# [[[endsection]]]

paver.doctools in docs

And then when you want to print the sample string, you just call::

# [[[cog include(“code/mysample.py”, “mysample”)]]] # [[[end]]]

paver.doctools in docs (2)

And then when you want to print the sample string, you just call::

# [[[cog include(“code/mysample.py”, “mysample”)]]] def sample_func(): print "To sample, or not to sample?" # [[[end]]]

paver.doctools uncog before commit

@task@needs("uncog")def commit(): """Removes the generated code from the docs and then commits to bzr.""" sh("bzr commit")

paver.svn

paver.svn.checkout(url, dest, revision='')Checks out the specified URL to the given destination.

paver.svn.checkup(url, dest, revision='')Does a checkout or update, depending on whether the destination exists and is up to date (if a revision is passed in). Returns true if a checkout or update was performed. False otherwise.

paver.svn.info(path='')Retrieves the svn info for the path and returns a dictionary of the values. Names are normalized to lower case with spaces converted to underscores.

paver.svn.update(path='', revision='')Run an svn update on the given path.

Example at: https://projects.sitepen.com/toolbox/svn/trunk/pavement.py

https://projects.sitepen.com/toolbox/svn/trunk/pavement.py

https://projects.sitepen.com/toolbox/svn/trunk/pavement.py

paver.ssh

paver.ssh.scp(source, dest) Copy the source file to the destination.

paver.virtual

paver.virtual.bootstrap()Creates a virtualenv bootstrap script. The script will create a bootstrap script that populates a virtualenv in the current directory. The environment will have paver, the packages of your choosing and will run the paver command of your choice.

This task looks in the virtualenv options for:

script_namename of the generated script

packages_to_installpackages to install with easy_install. The version of paver that you are using is included automatically. This should be a list of strings.

paver_command_linerun this paver command line after installation (just the command line arguments, not the paver command itself).

paver.misctasks

paver.misctasks.generate_setup()

paver.misctasks.help()

paver.misctasks.minilib()Options:

extra_fileslist of other paver modules to include (donʼt include the .py extension)

paver.misctasks.paverdocs()

Pulling it all together

• write functions with the @task decorator.

• Just plain old Python with lots of conveniences

• Paver’s pavement.py is a good example

• Even better, take a look at “Getting Started with Paver”http://bit.ly/starting_paver

http://bit.ly/starting_paver

http://bit.ly/starting_paver

Remember this?

#!/bin/sh

sphinx-build (blah blah blah)python setup.py sdist upload

Paver needs to do that too

@task@needs(['html', "minilib", "generate_setup", “setuptools.command.sdist”])def sdist(): """Builds the documentation and the tarball.""" pass

But wait, it does more!

@task@needs(['cog', 'paver.doctools.html'])def html(): """Build Paver's documentation and install it into paver/docs""" builtdocs = path("docs") / options.sphinx.builddir / "html" destdir = path("paver") / "docs" destdir.rmtree() builtdocs.move(destdir)

Changes coming to Paver

• The little bit of magic goes away

• Support for CloudControl

• Ability to run sub-builds easily

• Virtualenv/pyinstall management

• zc.buildout recipe running

Credits

• Thanks to SitePen for supporting Paver

• Bumpy road http://flickr.com/photos/ilya/442034/sizes/o/

• Smooth roadhttp://flickr.com/photos/nicholas_t/317528561/

• Options (Anna’s hairdo)http://flickr.com/photos/alangley/2136034468/

• Taskshttp://flickr.com/photos/mpwillis/470557535/

http://flickr.com/photos/ilya/442034/sizes/o/

http://flickr.com/photos/ilya/442034/sizes/o/

http://flickr.com/photos/nicholas_t/317528561/

http://flickr.com/photos/nicholas_t/317528561/

http://flickr.com/photos/alangley/2136034468/

http://flickr.com/photos/alangley/2136034468/

http://flickr.com/photos/mpwillis/470557535/

http://flickr.com/photos/mpwillis/470557535/

http://www.blueskyonmars.

com/projects/paver/

http://www.blueskyonmars.com/projects/paver/








Paver For PyWorks 2008

Technology

Transcript of Paver For PyWorks 2008