maas-devel team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

On Wed, May 9, 2012 at 7:59 AM, Gavin Panella
<gavin.panella@xxxxxxxxxxxxx> wrote:
> On 8 May 2012 19:38, Robert Collins <...> wrote:
>> In addition to Clint's excellent points (all of which I agree with),
>> I'd also add two more points:
>>
>> * pserv, being twisted, means that it will have a hate-hate
>> relationship with ORM state, just keeping it from doing silly things
>> like keeping a transaction open for days will be an exercise in
>> great care and diligence.
>
> PostgreSQL can enforce transaction and/or query time-outs, can't it?
> If not, then Twisted has callLater.

We use PostgreSQL to enforce them, but you have to teach the ORM to
set them up and adjust them - see the relevant logic in LP for an
example.

callLater does not help as you cannot interrupt the request cleanly
(dropping the TCP session is very hit-and-miss, python threads aren't
sanely interruptible when in C modules...)

>> * all the protections we (eventually) put in place around the DB
>> (such as timeouts and worker concurrency limits) will have to be
>> replicated for pserv, and as it has a different programming model,
>> that means double work. In LP we haven't done this yet, and we have
>> had the failure modes (like a script that goes nutty keeping backps
>> from running, or a concurrent script causing unanticipated load) at
>> one time or another. MAAS, being deployed on customer sites, outside
>> of our ops teams reach, has to insulate itself from these sorts of
>> things.
>
> We used Storm and transactions and all that stuff with ease and
> without incident in clouddeck with no special provisions for Twisted
> other than: do transactional stuff in a thread that is not the reactor
> thread [1].
>
> I'm almost certain that it would be trivial to use Django's ORM within
> Twisted too.
>
>
> [1] Or: *never ever ever never ever* do transactions in the reactor
> thread. Partly because, iirc, the transaction module expects to be
> able to use thread-locals to keep track of transaction state, hence
> doing all transactions in a single thread leads us right into the
> belly of the beast.

Can you do it and should you do it are two different questions. [1]
above is a classic example of designing *to fail*: humans will make
mistakes, and when the result of the mistake is a performance drop off
that is only visible at scale, its hard to effectively detect the
mistake before it reaches trunk, without having a complex, costly,
process around vetting revisions coming into trunk. Much simpler to
avoid the entire *possibility* of the problem occuring, by having no
special reactor thread.

The more things to remember, the more likely it is that a contributor
will forget one, and the more subtle the interactions, the more likely
it will escape review, etc etc. This is a big part of why simplicity
matters : keeping the load that our brains *have* to carry to do
things well low.

-Rob