A tutorial on Virtualenv to isolate python installations

  1. You don’t have sudo / root access on a development machine?
  2. You want to try out a new package without polluting the system directories?
  3. You want to test or experiment with a new version of a library without uninstalling the previous version?
  4. You want to test your code in an isolated environment before you deploy it to a production server?

If your answer to any of the questions above is YES, then this post if for you!

What is virtualenv?

Virtualenv is a tool to create isolated Python environments quite like chroot jail on Unix systems. In a chroot, programs cannot access anything outside of chroot but in virtualenv as the name implies, it creates isolated environments only with respect to libraries, but the programs can still access the files and folders normally.

Imagine if you are working on your next Web2.0 project and you want to upgrade to a newer version of a library to test your code with it first without breaking your existing python installation. Or say you want to ensure that you have same set of libraries as on the systems where you will be deploying your code. Moreover if your hosting service is shared, hence you don’t have sudo access to install any package in global python installation. Virtualenv, comes to your rescue in all these scenarios.

Virtualenv setup and usage

Step 1: Install virtualenv

$ pip install virtualenv

Step 2: Create a virtualenv

The following command creates a clean virtualenv devoid of any packages in a directory called “ve”. However you can use any name you want for the name of virtualenv directory.

$ virtualenv --no-site-packages ve
The --no-site-packages flag is deprecated; it is now the default behavior.
New python executable in ve/bin/python2.6
Also creating executable in ve/bin/python
Installing setuptools............done.
Installing pip...............done.

Note that –no-site-packages is the default behavior so it is not really necessary to pass this option but I included this for clarity.

The way it works is by creating symlinks to your global python installation, as you can see clearly using the tree utility.

$ tree -d ve/
ve/
|-- bin
|-- include
| `-- python2.6 -> /usr/include/python2.6
`-- lib
 `-- python2.6
 |-- config -> /usr/lib/python2.6/config
 |-- distutils
 |-- encodings -> /usr/lib/python2.6/encodings
 |-- lib-dynload -> /usr/lib/python2.6/lib-dynload
 `-- site-packages
 `-- pip-1.1-py2.6.egg
 |-- EGG-INFO
 `-- pip
 |-- commands
 `-- vcs

15 directories

Step 3: Use the isolated python environment

To use or enter the environment, use the following command:

$ source ./ve/bin/activate

Now you are in an new environment and any globally installed packages won’t be available here. If I list the packages before entering the virtual environment it shows all the installed packages in my global python installation.

$ pip freeze
Brlapi==0.5.5
CouchDB==0.8
Django==1.3
Fabric==1.3.4
GnuPGInterface==0.3.2
Mako==0.3.6
MarkupSafe==0.9.2
PAM==0.4.2
PIL==1.1.7
Scrapy==0.14.1
South==0.7.3
Twisted==12.0.0
...

After I run enter the environment, I see that no system packages exist in this installation/ virtualenv:

$ source ./ve/bin/activate
(ve)$ pip freeze
(ve)$

Also note that the command prompt on the shell now has a prefix ‘ve’ (the name of your virtual environment) which indicates you are in the virtual environment.

Step 4: Install the required packages

Now you can simply install the packages using pip and does not require sudo access as all the packages are installed in the virtualenv ‘ve’ created above:

(ve)$ pip install <package-name>

or from a requirements / dependency file:

(ve)$ pip install -r requirements.txt

Now you can continue your to develop and test your code in this environment.

Step 5: Exit the virtual environment

To leave the virtual environment simply run:

(ve)$ deactive
$

Notice that the command prompt has now reverted back to $ without the prefix ‘ve’.

Summary

Virtualenv and Pip are must use tools for advanced as well as beginners in python development. It seems daunting but both Pip and Virtualenv are quite easy to setup and use. Using them in combination would avoid a lot headache in the long run.

Advertisements

A short tutorial on Python packaging

If you are a beginner to Python, then you might find it confusing that there are several ways to install Python packages and which one to use. They are:

  1. distutils – With distutils you would typically download the package, untar it and then run:
     $ python setup.py install

    This will install the library files (packages and modules) into the ‘site-packages’ directory of the Python interpreter. On my ubuntu11.04 the location of site-packages is /usr/local/lib/python2.6/site-packages/. Note that distutils does not install dependencies.

     

  2. easy_install (part of setuptools package) – It automates the process by downloading a named package from PyPi (which is a repository of Python Packages). To install packages using easy_install run:
    $ easy_install <package-name>

    and it downloads and installs the package in ‘dist-packages’ directory of the Python interpreter.

     

  3. pip – succeeds easy_install as it has several advantages. A few of them are:
    • support for uninstallation of packages, which is not possible with easy_install
    • installs dependencies
    • partial / incomplete installation do not happen with pip
    • pip automatically installs all packages specified in a file with pip install -r requirements.txt

There are other benefits of pip which are listed here: http://stackoverflow.com/questions/3220404/why-use-pip-over-easy-install

To install packages using pip simply run:

$ pip install <package_name>

or to install all packages listed in a file run:

$ pip install -r requirements.txt

To see the list of installed packages run:

$ pip freeze

Apart from the tools listed above there are several others which succeed like disutils2 and distribute but they are not used very commonly.

Summary

Pip is the future and is the recommended way to install python packages. For best results use pip with virtualenv. I will talk about virtualenv in another post of mine in this series.