launchpad-dev team mailing list archive

["Francis J. Lacoste" <francis.lacoste@xxxxxxxxxxxxx>]

Hi Graham,

tl;dr  Thanks for looking in this issue.
I think the let's upload to twisted is a big hand wave at the moment and
we should probably investigate why apache is failing.

I don't understand how allowing direct upload to the librarian would
solve this bug. It might, but I think there are quite a few loose ends
that need to be connected first.

For reminder, the problem is that the file upload form fails at the
apache level in some nebulous but - not so uncommon - cases. That form
handles upload both the content (which ends up in the librarian) and the
metadata (creating the release and linking it to the file content).

So loose ends:

  * As a general deployment aren't we always running apache in front of
    twisted? So uploading to the librarian would fail similarly than
    now.

  * It's interesting that users report that creating the release using
    the API script doesn't suffer from this bug. In which case, this
    might point to a apache <-> browser bad interaction.

  * How are we going to handle the cross domain issue (loads from
    launchpad.net, posts to launchpadlibrarian.net - without going
    through apache?)

  * Are we going to write a "specific" view in twisted to take care of
     all the data handled by this form. Or more specifically, devise a
     generic protocol where once the content is stored, the librarian
     redirect to the real view with the file alias reference + the
     other metadata originally passed in)?

And I'm pretty sure that "solutions" to any of these problems will open
up a bunch of other issues. Making that path really too much for what we
are trying to achieve (work-around a weird apache bug).

Simpler fix:

  * See if playing if timeouts on apache change the behavior.

  * Change (or make an alternative yui-based upload form) that uses the
    yui flash-based upload widget (maybe the flash http doesn't
    trigger the apache bug, like the launchpadlib script does).

  * Just add warnings to users that they use the launchpadlib script if
    they encounter repeated upload errors.

Cheers
On 11-10-03 01:13 PM, Graham Binns wrote:
> Calling all folks, hello folks...
> 
> I've been staring at bug 194558 (Project file uploads time-out but don't
> OOPS)[1] for the last few days. The problem is twofold:
> 
>  1. Uploads occasionally fail at the Apache level - i.e. Apache has the
>     upload but can't pass it on to the appserver.
>  2. We don't get OOPSes when this happens.
> 
> I've been looking at problem #1 (even though #2 is probably the easier
> to fix). Our fearless TA made the following suggestion, complete with
> hand-waving, in the bug comments [2]:
> 
>> It may be easiest to fix this by moving uploads out of the zope
>> appservers, which our infrastructure is not geared around supporting
>> big uploads too. E.g. upload directly to the librarian, then register
>> the content with LP (/handwave)
> 
> And I'm interested in investigating this further. As far as I can tell,
> we have the following issues to solve:
> 
>  1. How do we upload directly to the librarian from the browser? (AIUI
>     the librarian uses an almost-but-not-quite-HTTP-like protocol, which
>     makes things awkward, but if I've misunderstood then please feel
>     free to tell me and so make me very happy).
>  2. How do we register the content with LP once the upload has
>     completed? (The answer that both Gary and I have arrived at involves
>     the words "hey, look at this RabbitMQ thing over here...")
> 
> Really, I'm looking for any information that anyone has on how the
> Librarian works, since I don't have a great understanding of it. If I'm
> wrong about the protocol then this is a relatively simple bug to fix. If
> I'm right then I suspect I'll need your help to work it out.
> 
>  [1] https://bugs.launchpad.net/launchpad/+bug/194558
>  [2] https://bugs.launchpad.net/launchpad/+bug/194558/comments/6
> 


-- 
Francis J. Lacoste
francis.lacoste@xxxxxxxxxxxxx

Attachment: signature.asc
Description: OpenPGP digital signature