launchpad-dev team mailing list archive

[Gary Poster <gary.poster@xxxxxxxxxxxxx>]

On Jul 11, 2011, at 2:52 AM, Robert Collins wrote:

> On Mon, Jul 11, 2011 at 6:47 PM, Thomas Hervé
> <thomas.herve@xxxxxxxxxxxxx> wrote:
>> Le 11/07/2011 01:24, Robert Collins a écrit :
>>> On Mon, Jul 11, 2011 at 8:58 AM, Thomas Hervé
>>> <thomas.herve@xxxxxxxxxxxxx> wrote:
>>>> Not arguing with those last points, I just wanted to point out that
>>>> using Twisted with a Zope server is not that hard. There are 2 cases:
>>>>  * You're already using Twisted as WSGI container. In which case, you
>>>> are already running Twisted, so it's "just" a matter of being careful
>>>> when you're making from from threads. blockingCallFromThread is really
>>>> useful for that.
>>> 
>>> We have that turned off due to instability.
>> 
>> I'd be interested to know the problems.
> 
> Gary may be able to fill you in, AIUI it was tried but not stable so untried.

I was around when Zope Corp tried it and around Launchpad when we tried it here.  In both cases, as soon as we deployed it we got instability at production load--I don't remember details except that we got error log messages, and broken responses.  In both cases, we removed the Twisted integration and went back to Zope's asyncore-based "ZServer" wsgi server implementation and things were fine again.

The integration at the time was with Twisted web2, which has been deprecated for a long time now, I think.  Digging into problems was not judged as valuable for ZC or Launchpad.  Twisted web (the original package) now (for a while) has wsgi support, which looked very nice and simple the last time I looked at it.  I've idly wanted to try out with Launchpad, but there's not been a strong case for the time and risk.  In theory, it ought to work, and Twisted web and Zope both have years of production usage behind them, as opposed to web2, so I'd be cautiously optimistic.

> 
>>>>  * You're using another container. In this case, you can use a dedicated
>>>> thread for running Twisted. There are a couple of pitfalls (not install
>>>> the signal handlers, thus not starting processes with Twisted), but
>>>> otherwise it works fine. blockingCallFromThread still works as a bonus.
>>> 
>>> The container we're using is zope itself. The security model is thread
>>> based, twisted is not. This leads to a range of terrible side effects
>>> - from silent security vulnerabilities to attribute access failing for
>>> undiagnosable reasons.
>>> 
>>> Your points about how one *in general* might integrate twisted are
>>> reasonable, but *zope* specifically plus *twisted* are specifically
>>> considerably harder to work well reliably together.
>> 
>> It seems to be you're talking about integrating a threaded application
>> with Twisted, and there is not much specific to Zope here. Indeed, you
>> should make sure you only send stateless objects to the Twisted thread.
>> It's a problem which can happen with Storm as well for example. Granted,
>> it's a problem.
> 
> Well, individual requests in zope aren't threaded, but the framework
> is deeply threaded, and state isn't trivial to isolate.
> 
> So I'm sure it can be done, but its not well understood and easy to do right! ;)

Mm.  Both of you have good points.

I've done Twisted and Zope integration, and it's worked very well.  All of the basics are open-sourced (zc.twist, zc.async) though none of it is of interest to us, because it uses ZODB.  

At least for the jobs I was doing, it was fairly trivial to establish which kinds of Twisted tasks do not need security or other application context. Similarly, paranoid-but-still-useful rules of what's safe to pass around threads are trivial and reasonably well-known (e.g., "don't pass mutables" works nicely), so we followed them when possible; and we allowed explicit gestures to pass across database (ZODB) references, and even security context, in a safe way.

LP has extra threaded state, but Zope itself only has a single one that we care about: the security interaction, which contains the request, which references a principal.  If you establish the right security policy class as an interaction, and have the right principals, the security is pretty much good to go, which is sufficient in many cases.

I'm simply sharing experiences here; I'm not arguing a particular position.  It's not entirely trivial, but it's been done successfully and productively.

Gary