← Back to team overview

nova team mailing list archive

Re: Why not python threads?

 

On Wed, Aug 04, 2010 at 07:50:01PM +0100, Ewan Mellor wrote:
>>> I feel like Nova is greatly complicated by the async code, and I'm
>>> starting to see some of the pain of Twisted: it seems that
>>> _everything_ needs to be async in the long run, because if something
>>> calls a function that is (or could be) async, it must itself be
>>> async.  So yields and @defer.inlineCallbacks start cropping up
>>> everywhere.
>> Just to be clear: defer.inlineCallbacks (and thus yields) are by no
>> means required to deal with deferreds. Before Nova, I had never used
>> it.
> All this aside, could you write a few simple examples of how we're
> supposed to use deferreds, inlineCallbacks, yields etc for now?  I
> thought that I was copying the idioms in use elsewhere, but I've ended
> up with code that is definitely blocking, even though I thought that
> it wouldn't be.

Ok, the first two things to realise are:

 a) Twisted is single threaded by default.
 b) Using a deferred to do something does not make it asynchronous.

Paraphrasing a little bit:

 a) If you make a blocking call in your code, your application will hang
    until the call returns.
 b) Calling it through something like defer.maybeDeferred will not help.


E.g.:

from twisted.internet import reactor
import time

def wait_a_little_bit():
    time.sleep(2)
    print "Slept until %s" % (time.asctime())

def do_stuff():
    wait_a_little_bit()
    wait_a_little_bit()

reactor.callWhenRunning(do_stuff)
reactor.run()


If you run this, you'll see that the calls to wait_a_little_bit happen
in sequence rather than in parallel. This is because time.sleep blocks
the whole application.

Adding deferreds to the mix does little to help us:

from twisted.internet import reactor, defer
import time

def wait_a_little_bit():
    time.sleep(2)

def done_waiting(_):
    print "Slept until %s" % (time.asctime())

def do_stuff():
    d = defer.maybeDeferred(wait_a_little_bit)
    d.addCallback(done_waiting)
    d = defer.maybeDeferred(wait_a_little_bit)
    d.addCallback(done_waiting)

reactor.callWhenRunning(do_stuff)
reactor.run()

If you need to call out to a blocking API (which is rather common), what
you really want to do is to use reactor.callInThread. This creates a
new thread where the blocking call is made without interfering with the
rest of your app.


from twisted.internet import reactor import time

def wait_a_little_bit():
    time.sleep(2)
    print "Slept until %s" % (time.asctime())

def do_stuff():
    reactor.callInThread(wait_a_little_bit)
    reactor.callInThread(wait_a_little_bit)

reactor.callWhenRunning(do_stuff)
reactor.run()

If you run this, you'll see the two calls to wait_a_little_bit finish at
roughly the same time.

If you need to get data out of the call, you can pass create a deferred,
attach your callbacks to it, and pass it to wait_a_little_bit. Example:

from twisted.internet import reactor, defer
import logging
import time

def wait_a_little_bit(d):
    time.sleep(2)
    d.callback("Slept until %s" % (time.asctime()))

def do_stuff():
    logging.basicConfig(level=logging.INFO)

    d = defer.Deferred()
    d.addCallback(logging.info)
    reactor.callInThread(wait_a_little_bit, d)
    d = defer.Deferred()
    d.addCallback(logging.info)
    reactor.callInThread(wait_a_little_bit, d)

reactor.callWhenRunning(do_stuff)
reactor.run()


I hope this helps a little bit.

-- 
Soren Hansen
Ubuntu Developer
http://www.ubuntu.com/



Follow ups

References