nova team mailing list archive

[Justin Santa Barbara <justin@xxxxxxxxxxxx>]

I've spent today repeating Eric Day's performance benchmarks, including
simple python threading, on a quad core instead of a dual core (I actually
wanted to evaluate Jython, but I digress.)  Turns out threading then beats
Twisted and Eventlet.  Twisted can get the upper hand again over threading,
with a multi-process model, but we're not talking significant differences
(64 seconds vs 52 seconds).

My conclusion is simple: it isn't clear that Twisted buys us the great
performance it promises.  It is clear that it locks us in to the Twisted
deferred model.

If we stick with 'clean code', we bet on 3 horses:

   - Python 3 may fix some of the GIL issues that make threaded code perform
   poorly, in particular the ones that David Beazley recently highlighted
   (though this will depend on how much time the apps spends on CPU vs I/O)
   - Jython doesn't even have a GIL
   - Eventlet can offer async without code changes

If we choose Twisted, we're betting everything on Twisted being the best
option.

If the Twisted proponents want to demonstrate a benchmark that simulates the
nova-api use case, where Twisted offers a sufficient performance
benefit (over both eventlet and simple threading) to justify the lock-in and
the code complexity, _then_ we should consider Twisted.

Until that result happens, I vote that we should avoid the Twisted lock-in.
 To me, the language choice of Python indicates that we care about developer
productivity and code simplicity & correctness more than we care about raw
performance.  IMHO, choosing Twisted is inconsistent with that choice.

Incidentally, this talk is a real eye opener:
http://blip.tv/file/2232349  It's the
'opening act' to David Beazley's GIL talk, and describes very well how
Python threading is actually much better than generally thought.

Justin


On Sat, Aug 7, 2010 at 6:03 AM, Rick Clark <rick@xxxxxxxxxxxxx> wrote:

> On 08/04/2010 01:35 PM, Gustavo Niemeyer wrote:
> > I'm not actually arguing either way, but if I had to make this
> > decision, I'd have a slightly more even comparison between eventlet
> > and twisted.  Twisted is stable, well known, has several maintainers,
> > and *many* users.  Canonical, Google, Apple, ITA, and many many other
> > use Twisted.  You can trust on it being around and well maintained in
> > the foreseeable future.
>
> I am not making a judgment here, but I agree with Gustavo. I think the
> health and viability of a project should be considered in it's
> selection.  The decision isn't purely technical.
>
> On another note, this is not an either/or question.  Twisted does many
> things that eventlet does not, they are not mutually exclusive.
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~nova
> Post to     : nova@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~nova
> More help   : https://help.launchpad.net/ListHelp
>
>