Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration

under contract DE-AC04-94AL85000.

Installing Python Software Packages:The Good, The Bad and the Ugly

William E. HartSandia National Laboratories

wehart@sandia.gov

Overview

This talks focuses on installing on Python packages– Installing Python itself is usually a no-brainer

The Good1. Installing on Windows with an installer executable2. Installing with Linux application utility3. Installing a Python package from the PyPI repository4. Installing a Python package from source

The Bad5.Using a virtual environment to isolate package installations6.Using an installer executable on Windows with a virtual

environmentThe Ugly

7. Installing a Python extension package from source8.PyCoinInstall – Managing builds for Python extension packages

Slide 2

About Python

Recent releases:– 2.5 Very stable– 2.6 Very stable– 2.7 Recently released. Supports transition to Python 3.x– 3.0 Includes changes that are not backwards compatible.– 3.1 Stable 3.x release. Resolves 3.0 performance issues.

Package management for 3.x is still a bit snarky. We’ll focus on examples for 2.x

Slide 3

Python Packages

Package distributions– Include one or more *.py files– May include compiled source files (e.g. C++)– Include a top-level setup.py file to coordinate installation

Three categories:– Pure-Python packages

• Only include Python *.py files

– Independent extension packages• Include compiled source files only on standard system

libraries

– Dependent extension packages• Include compiled source files that rely on third-party

libraries

Slide 4

The Good

Assumption: you have administrative privileges– Windows administrator group– Linux super-user

Impact: you can install Python in system directories!

Idea:– Install packages in the system Python directories

• Specifically, the site-packages directory is used

– Can easily install pure-Python packages or pre-compiled extension packages

Slide 5

Example 1

Installing on Windows with an installer executable

Steps:1.Download installer executable from package website2.Execute installer

Notes:– This uses the registry to find the Python installation– The installer can contain pre-compiled extension packages

• These may differ for 32- and 64-bit installations– The installer will likely be specific to the Python version– This may work OK for dependent extension packages if the

dependent libraries are stored in a standard location

Slide 6

Example 2

Installing with Linux application utility

Idea: use a standard Linux package manager– Ubuntu (and other Debian-based Linux variants)

• apt-get install packagename– Fedora

• yum install packagename

Notes:– The package manager can install pre-compiled extension packages– The package manager handles package dependencies

automatically– Not all Python packages are available within these managers– It’s unclear whether these packages are updated frequently

Slide 7

Example 3

Installing a Python package from the PyPI repository

PyPI:– A web-based service for hosting Python packages– Includes some pre-compiled extension packages– Several Python packages are used to install from PyPI

• setuptools, distribute, pip

Installing with pip:– pip install packagename

Notes:– The package manager handles package dependencies automatically– Most Python packages are available from PyPI, including most releases– PyPI does not include many pre-compiled extension packages– The PyPI service has become a bit overloaded in the past year– You need to specify the HTTP_PROXY environment to talk to proxy servers

Slide 8

Example 3(cont)Problem: pip is not a standard Python package!

Installing pip:– wget http://peak.telecommunity.com/dist/ez_setup.py– python ez_setup.py– easy_install pip

Notes:– pip is now generally recommended over setuptools and

distribute– The wget command is not a standard MS Windows utility

• Download from http://users.ugent.be/~bpuype/wget/

Slide 9

Example 4

Installing a Python package from source

Steps:– Download package source from PyPI or project website– Extract package from tarball or zipfile– python setup.py install

Notes:– This process avoids the need to install pip– This is useful for packages that are not available on PyPI– This installation process can build and install some extension packages

• Configuration of the compiler may be a problem• Configuration of extension package dependencies is a problem

– This installation does not handle package dependencies• Package dependencies may cause an install to fail

Slide 10

The Bad

Problem: users do not have administrative privileges– This is not uncommon in industry

Problem: different users on a computer need different package versions

Idea:– A user can create a virtual Python environment– The user installs packages in this virtual Python directories

Impact:– Users have complete control over their Python environment

• Can even have multiple environments (e.g. for developers)– Can install both pure-Python packages or pre-compiled

extension packages

Slide 11

Virtual Python

Idea: create an isolated Python environment

virtualenv:– Creates a directory structure that mimics the system Python

installation– Works on MS Windows, Linux and Mac OS

Installing virtualenv (with administrative privileges)– Debian Linux

• apt-get install python-virtualenv

– Other• wget http://peak.telecommunity.com/dist/ez_setup.py• python ez_setup.py• easy_install virtualenv

Slide 12

Virtual Python (cont)Creating a virtual Python environment:

– virtualenv directory

Virtual Python executable:– MS Windows: directory/Scripts/python.exe– Linux: directory/bin/python

Notes:– On MS Windows the executables are installed in the Scripts

directory• For simplicity, all subsequent examples will use the bin directory

– The setuptools package is installed in a virtual environment• Can use the easy_install command to install packages

Example:– directory/bin/easy_install pip

Slide 13

Virtual Python(cont)Problem: setup a virtual environment without administrative privileges

vpy_install:– Bootstrap a virtualenv installation from a script

Steps:– wget http://doiop.com/pyutilib/vpy_install– python vpy_install directory

Notes:– Standard tools like setuptools and pip are installed by vpy_install– The specified directory is a virtualenv Python environment– On Linux, you can use a simpler command-line:

• vpy_install directory– The –-site-packages option can be used to expose packages that are

installed in the system Python installation• This is particularly useful when using precompiled extension packages,

which are installed in system Python directories

Slide 14

Example 5

Using a virtual environment to isolate package installations

Idea:– Create a virtual python installation with vpy_install– Install packages with pip

Example (Linux):– wget http://doiop.com/pyutilib/vpy_install– vpy_install mypy– mypy/bin/pip install packagename

Notes:– Within the virtual environment, pip is installed as an executable

• You do not need to execute it with the Python executable

Slide 15

Example 5(cont)Using a virtual environment to isolate package installations

Note: source package installations using a virtual Python executable are installed in the virtual environment

Example (Linux):– wget http://doiop.com/pyutilib/vpy_install– vpy_install mypy– cd packagename– ../mypy/bin/python setup.py install

Slide 16

Example 6

Using an installer executable on Windows with a virtual environment

Problem:– MS Windows installers use the system Python installation– If a user does not have administrative privileges, then an installer

cannot be used directly!

Idea:– “Install” the executable with the easy_install command

Example:– easy_install.exe matplotlib-1.0.0.win32-py2.6.exe

Note: this only works if the executable was created by the Python distutils package!

Slide 17

Example 6(cont)Using an installer executable on Windows with a virtual

environment

Idea:– Modify the MS Windows registry to register the virtual python

executable

Steps:

1.Download the the_python.py script that is described at

http://doiop.com/nedbatchelder/register_python.html

2.Register the virtual python executable (as administrator)

• mypy\Scripts\python.exe the_python.py

3.Execute the MS Windows installer

4.Reregister the original python executable (as administrator)

Slide 18

The Ugly

Problem:– you need to use an extension package that has dependencies– prebuilt packages are not available

Observation:– There is very little consistency between installation setup for

different packages– Few packages use configuration management tools like

autotools or cmake– Build instructions can be very different for different platforms

There are few options beyond diving deep into the package build process!

Slide 19

Example 7

Installing a Python extension package from source

Note: there is no standard way for doing this

Example: SciPy documentation– General build instructions– Separate instructions for Linux, Mac OS, Windows, and Solaris

Recommendations:– Make sure that this package is going to be worth your time– Don’t hesitate to ask the developer team for help– Have patience ...

Slide 20

Example 8

PyCoinInstall – Managing builds for Python extension packages

Note: this script is a relatively new project within CoinBazaar– https://projects.coin-or.org/CoinBazaar/wiki/Projects/PyCoinInstall

Idea:– Create a virtual environment with virtualenv– Define package installation with a simple plugin system

• Packages define download, build and/or install operations– A user creates a simple configuration file to specify the packages that

will be installed– The pci command is used to manage the installation

Notes:– Current development is focused on COIN-OR related packages– Current plugins are tailored for building under Linux

Slide 21

Final Thoughts

Installing and distributing pure-Python packages is quite easy.

Installing extension packages is much more difficult.– Try to leverage pre-compiled extension packages!

Virtual Python environments offer a flexible paradigm for managing different Python configurations

– These are worth using even if you have administrative privileges!

Slide 22