maas-devel team mailing list archive

[Gavin Panella <gavin.panella@xxxxxxxxxxxxx>]

On 9 May 2012 02:15, Robert Collins <...> wrote:
> On Wed, May 9, 2012 at 7:59 AM, Gavin Panella <...> wrote:
...
>> PostgreSQL can enforce transaction and/or query time-outs, can't it?
>> If not, then Twisted has callLater.
>
> We use PostgreSQL to enforce them, but you have to teach the ORM to
> set them up and adjust them - see the relevant logic in LP for an
> example.

This would still work.

>
> callLater does not help as you cannot interrupt the request cleanly
> (dropping the TCP session is very hit-and-miss, python threads aren't
> sanely interruptible when in C modules...)

No, but it can be used to give the client a response, cancel the
operation from the PostgreSQL end, or arrange for results to be sent
instead via long-poll, for example.

...
>> [1] Or: *never ever ever never ever* do transactions in the reactor
>> thread. Partly because, iirc, the transaction module expects to be
>> able to use thread-locals to keep track of transaction state, hence
>> doing all transactions in a single thread leads us right into the
>> belly of the beast.
>
> Can you do it and should you do it are two different questions. [1]
> above is a classic example of designing *to fail*: humans will make
> mistakes,

Using threads is another classic example of designing to fail, but
that's the prevalent model for web applications because everyone
always all remembers to be very careful when mucking around with
global state... don't they? ;)

> and when the result of the mistake is a performance drop off that is
> only visible at scale, its hard to effectively detect the mistake
> before it reaches trunk, without having a complex, costly, process
> around vetting revisions coming into trunk. Much simpler to avoid
> the entire *possibility* of the problem occuring, by having no
> special reactor thread.

It's actually not hard to detect: a transaction monitor that ensures
that the current thread != the reactor thread.

>
> The more things to remember, the more likely it is that a
> contributor will forget one, and the more subtle the interactions,
> the more likely it will escape review, etc etc. This is a big part
> of why simplicity matters : keeping the load that our brains *have*
> to carry to do things well low.

That's true for anything, be it Twisted, Django, Celery, YUI. I mean,
YUI has *so* much crap to remember yet it's a preferred technology.
Twisted, for what it's worth, allows you to use a threaded or event
driven model as the problem dictates. There's little friction between
the two. And Twisted is fairly simple underneath it all.

I don't want to argue that Twisted is perfect, I just want to address
what I see as unwarranted FUD around it. It's a relatively simple and
mature framework, into which you can buy a lot or a little. There's a
reactor, which runs in a single thread; everything else in Twisted is
optional library code. Django and YUI, for example, demand a lot more
deference from developers.