launchpad-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On Tue, Oct 4, 2011 at 6:13 AM, Graham Binns <graham@xxxxxxxxxxxxx> wrote:
> Calling all folks, hello folks...
>> It may be easiest to fix this by moving uploads out of the zope
>> appservers, which our infrastructure is not geared around supporting
>> big uploads too. E.g. upload directly to the librarian, then register
>> the content with LP (/handwave)
>
> And I'm interested in investigating this further. As far as I can tell,
> we have the following issues to solve:

\o/

>  1. How do we upload directly to the librarian from the browser? (AIUI
>    the librarian uses an almost-but-not-quite-HTTP-like protocol, which
>    makes things awkward, but if I've misunderstood then please feel
>    free to tell me and so make me very happy).

Theres no reason its not an HTTP POST (PUT implies users choosing the
URL so it perhaps not appropriate) other than hysterical raisins.
Changing it to be one is a small task for a twisted fluent person, or
a moderate one for a non fluent person, IMNSHO. (Switch from custom
protocol to a standard one pulling our metadata out of the the form
variables)

>  2. How do we register the content with LP once the upload has
>    completed? (The answer that both Gary and I have arrived at involves
>    the words "hey, look at this RabbitMQ thing over here...")

Less handwavy time then :P - Several options appear to me here:
A: Have the user do it: on completion of the upload return HTTP 30x
and a link to the linking page in the appserver. There are some
security ramifications here (e.g. can I trigger someone to link
hostile content to their account somehow) but that only means pay
attention to detail, AFAICT

B: Have the librarian do it: include the object to link to as a form
header, have the upload service run in the same domain as the
appserver (so auth cookies are available) and make an XMLRPC call to
the appserver before returning the HTTP status code to the user. In
fact the XMLRPC call to the appservers could return the next_url to
redirect to, giving a seamless experience.

C: Offload it to a backend asynchronously - e.g. Rabbit. I think this
will prevent us giving the user a clear signal about their upload and
so is undesirable. Remember that there is nothing to wait /for/ if the
upload went straight to the librarian - we wouldn't be double handling
uploads anymore, so I don't think there is any need to queue things
up.

> Really, I'm looking for any information that anyone has on how the
> Librarian works, since I don't have a great understanding of it. If I'm
> wrong about the protocol then this is a relatively simple bug to fix. If
> I'm right then I suspect I'll need your help to work it out.

Its a twisted daemon, launched via a tac file. That means that the tac
file declares a Service with associated protocol factories that listen
on various ports. Each of those then fires up a protocol instance when
something connects to it. You don't need to touch the existing
factories to fix this bug - you need to add a new HTTP POST upload
factory (and I suggest this set all LFA's it creates to private, and
let the linking step in the appserver decide if it should instead be
public).

If you need further details to get the ball rolling, I, or one of the
other twisted heads here, can certainly do that :)

my recommendation is B.

-Rob