launchpad-dev team mailing list archive

[Graham Binns <graham@xxxxxxxxxxxxx>]

Calling all folks, hello folks...

I've been staring at bug 194558 (Project file uploads time-out but don't
OOPS)[1] for the last few days. The problem is twofold:

 1. Uploads occasionally fail at the Apache level - i.e. Apache has the
    upload but can't pass it on to the appserver.
 2. We don't get OOPSes when this happens.

I've been looking at problem #1 (even though #2 is probably the easier
to fix). Our fearless TA made the following suggestion, complete with
hand-waving, in the bug comments [2]:

> It may be easiest to fix this by moving uploads out of the zope
> appservers, which our infrastructure is not geared around supporting
> big uploads too. E.g. upload directly to the librarian, then register
> the content with LP (/handwave)

And I'm interested in investigating this further. As far as I can tell,
we have the following issues to solve:

 1. How do we upload directly to the librarian from the browser? (AIUI
    the librarian uses an almost-but-not-quite-HTTP-like protocol, which
    makes things awkward, but if I've misunderstood then please feel
    free to tell me and so make me very happy).
 2. How do we register the content with LP once the upload has
    completed? (The answer that both Gary and I have arrived at involves
    the words "hey, look at this RabbitMQ thing over here...")

Really, I'm looking for any information that anyone has on how the
Librarian works, since I don't have a great understanding of it. If I'm
wrong about the protocol then this is a relatively simple bug to fix. If
I'm right then I suspect I'll need your help to work it out.

 [1] https://bugs.launchpad.net/launchpad/+bug/194558
 [2] https://bugs.launchpad.net/launchpad/+bug/194558/comments/6

-- 
Graham Binns | PGP Key: EC66FA7D
http://launchpad.net/~gmb