nova team mailing list archive
-
nova team
-
Mailing list archive
-
Message #00143
Re: Why not python threads?
On Wed, Aug 04, 2010 at 07:50:01PM +0100, Ewan Mellor wrote:
>>> I feel like Nova is greatly complicated by the async code, and I'm
>>> starting to see some of the pain of Twisted: it seems that
>>> _everything_ needs to be async in the long run, because if something
>>> calls a function that is (or could be) async, it must itself be
>>> async. So yields and @defer.inlineCallbacks start cropping up
>>> everywhere.
>> Just to be clear: defer.inlineCallbacks (and thus yields) are by no
>> means required to deal with deferreds. Before Nova, I had never used
>> it.
> All this aside, could you write a few simple examples of how we're
> supposed to use deferreds, inlineCallbacks, yields etc for now? I
> thought that I was copying the idioms in use elsewhere, but I've ended
> up with code that is definitely blocking, even though I thought that
> it wouldn't be.
Ok, the first two things to realise are:
a) Twisted is single threaded by default.
b) Using a deferred to do something does not make it asynchronous.
Paraphrasing a little bit:
a) If you make a blocking call in your code, your application will hang
until the call returns.
b) Calling it through something like defer.maybeDeferred will not help.
E.g.:
from twisted.internet import reactor
import time
def wait_a_little_bit():
time.sleep(2)
print "Slept until %s" % (time.asctime())
def do_stuff():
wait_a_little_bit()
wait_a_little_bit()
reactor.callWhenRunning(do_stuff)
reactor.run()
If you run this, you'll see that the calls to wait_a_little_bit happen
in sequence rather than in parallel. This is because time.sleep blocks
the whole application.
Adding deferreds to the mix does little to help us:
from twisted.internet import reactor, defer
import time
def wait_a_little_bit():
time.sleep(2)
def done_waiting(_):
print "Slept until %s" % (time.asctime())
def do_stuff():
d = defer.maybeDeferred(wait_a_little_bit)
d.addCallback(done_waiting)
d = defer.maybeDeferred(wait_a_little_bit)
d.addCallback(done_waiting)
reactor.callWhenRunning(do_stuff)
reactor.run()
If you need to call out to a blocking API (which is rather common), what
you really want to do is to use reactor.callInThread. This creates a
new thread where the blocking call is made without interfering with the
rest of your app.
from twisted.internet import reactor import time
def wait_a_little_bit():
time.sleep(2)
print "Slept until %s" % (time.asctime())
def do_stuff():
reactor.callInThread(wait_a_little_bit)
reactor.callInThread(wait_a_little_bit)
reactor.callWhenRunning(do_stuff)
reactor.run()
If you run this, you'll see the two calls to wait_a_little_bit finish at
roughly the same time.
If you need to get data out of the call, you can pass create a deferred,
attach your callbacks to it, and pass it to wait_a_little_bit. Example:
from twisted.internet import reactor, defer
import logging
import time
def wait_a_little_bit(d):
time.sleep(2)
d.callback("Slept until %s" % (time.asctime()))
def do_stuff():
logging.basicConfig(level=logging.INFO)
d = defer.Deferred()
d.addCallback(logging.info)
reactor.callInThread(wait_a_little_bit, d)
d = defer.Deferred()
d.addCallback(logging.info)
reactor.callInThread(wait_a_little_bit, d)
reactor.callWhenRunning(do_stuff)
reactor.run()
I hope this helps a little bit.
--
Soren Hansen
Ubuntu Developer
http://www.ubuntu.com/
Follow ups
References