nova team mailing list archive

[Eric Day <eday@xxxxxxxxxxxx>]

Hi Justin,

To compare with a pure thread implementation, the same benchmark on
my blog put the total time for the heaviest load at about 3 minutes
(as opposed to two 1.5 minutes for twisted). The memory consumption
was higher even with the minimal thread stack size (about 4x twisted)
and context switching of course went through the roof.

Having said all that, we don't really need these levels of concurrency
with all components within the system. Any API endpoints should
certainly be based on twisted/eventlet/whatever, but host, volume, and
network workers could easily scale with a one-thread per connection
model. As Justin points out, with the async frameworks it's all or
nothing or you're going to be blocking the event thread with a single
task. This can be worked around with threads or external process
management, but at that point the coordination could probably be
easier without involving twisted event loops.

A question for the Nasa guys: what were the reasons for choosing
twisted for the workers?

-Eric

On Tue, Aug 03, 2010 at 06:30:14PM -0700, Justin Santa Barbara wrote:
>    Without meaning to make the twisted/eventlet flamewar any worse, can I
>    just ask why we're not just using 'good old threads'? *I've asked Eric Day
>    for his input based on his great benchmarks (http://oddments.org/?p=494).
>    *My background is*from the Java world, where threads work wonderfully -
>    possibly even better than
>    async:*http://www.thebuzzmedia.com/java-io-faster-than-nio-old-is-new-again
>    I feel like Nova is greatly complicated by the async code, and I'm
>    starting to see some of the pain of Twisted: it seems that _everything_
>    needs to be async in the long run, because if something calls a function
>    that is (or could be) async, it must itself be async. *So yields and
>    @defer.inlineCallbacks start cropping up everywhere.
>    One of the project goals seems to be simplicity of the code, for fewer
>    bugs and to reduce barriers to entry, and it seems that if we could use
>    'plain old Python' that we would better achieve this goal than if we have
>    to use an async framework.
>    I know that Python has its issues here with the GIL, but I'm just
>    wondering whether, in the case of nova, threads might be good enough, and
>    produce much easier to understand code? *I'm guessing that maybe the
>    project started with threads - what happened?
>    Justin

> _______________________________________________
> Mailing list: https://launchpad.net/~nova
> Post to     : nova@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~nova
> More help   : https://help.launchpad.net/ListHelp