launchpad-dev team mailing list archive

[Barry Warsaw <barry@xxxxxxxxxxxxx>]

Now that we're on Python 2.5, we have a few new language features at our
disposal.  In today's AMEU reviewer meeting, I brought up the 'with' statement
and was asked to write something up about it.  The with statement will be okay
to use in Launchpad code, so you'll see more of it crop up, and it's good to
know how and when to use it.

First off, in Python 2.5 you need a future import to enable the with
statement.  'with' is a keyword in Python 2.6, so when we move to that version
we can get rid of all the future imports:

>>> from __future__ import with_statement

The 'with' statement utilizes what's called the 'context manager protocol'.
This protocol is described here:

http://www.python.org/doc/current/reference/datamodel.html#context-managers

and there is a module of some helpers here:

http://www.python.org/doc/current/library/contextlib.html

Where is a with-statement useful?  IMO, any time you've got a try/finally
block, it's a good candidate for a with statement.  Context managers, as their
name implies, manages contexts and resources.  A with statement is useful any
time you allocate a resource and need to guarantee that it is cleaned up.

For example, if you wanted to print every line in /etc/password, and you were
anal like me, you'd do something like:

>>> fp = open('/etc/passwd')
>>> try:
...     for line in fp:
...         sys.stdout.write(line)
... finally:
...     # Don't wait for gc to free the file descriptor.
...     fp.close()

Look at how much nicer this is using the with statement.  This works because
built-in file objects already support the context manager protocol:

>>> with open('/etc/passwd') as fp:
...     for line in fp:
...         sys.stdout.write(line)

Notice that you do not need to explicitly close 'fp', because the file
object's context manager does this for your.  The 'as fp' part is optional;
it's the way to grab a handle on the target expression of the with statement.
In this case it's useful because you want the file object.  To write to a file
is similarly easy:

>>> with open('foo', 'w') as fp:
...     print >> fp, 'Look Ma, no hands!'

When the with-statement is complete, it automatically closes the file.

It's also very easy to write your own context managers to work with the
with-statement.  The documentation has all the gory details, but essentially,
you just need to implement an __enter__() method and an __exit__() method.
The former gets called when the with-statement starts, and the latter gets
called when it's done.  The signature of __exit__() is a little tricky to
allow it to handle exceptions in the body of the with-statement, but usually
you don't care and will just let those percolate up.

Here's a simple context manager for handling temporarily changing directories:

>>> class chdir:
...     """A context manager for temporary directory changing."""
...     def __init__(self, directory):
...         self._curdir = None
...         self._directory = directory
...     def __enter__(self):
...         self._curdir = os.getcwd()
...         os.chdir(self._directory)
...     def __exit__(self, exc_type, exc_val, exc_tb):
...         os.chdir(self._curdir)
...         # Don't suppress exceptions.
...         return False

You'd use this like so:

>>> with chdir('/tmp/var'):
...     shutil.rmtree('launchpad')

At the end of this, the cwd is restored, but inside the with, cwd is
/tmp/var.  Pretty easy right?  Well, it can be even easier! :)  The contextlib
module has a helper that works with generators to make defining these things
really simple.

>>> from contextlib import contextmanager
>>> @contextmanager
... def chdir(directory):
...    cwd = os.getcwd()
...    os.chdir(directory)
...    yield  # pass control back to the body of the with statement
...    os.chdir(cwd)
>>> print os.getcwd()
/tmp
>>> with chdir('/var/log'):
...   print os.getcwd()
... 
/var/log
>>> print os.getcwd()
/tmp

That example also shows you where the 'as' clause is omitted because it's not
needed.

I think that's the basics, and gives a good feel for how to use the with-
statement.  When to use them is also (IMO) fairly straightforward.  If you've
got some complex resource or environment you need to set up and you have to
guarantee that it's restored at the end of the task, use a with-statement.

A common question is whether to use these to manage transactions.  Jeroen and
I have debated this one.  It's worked well for me in my Storm-based
non-Launchpad applications, but Jeroen doesn't like their semantics for this.
I'll let him take it from there.

Enjoy,
-Barry

Attachment: signature.asc
Description: PGP signature