launchpad-dev team mailing list archive

[Edwin Grubbs <edwin.grubbs@xxxxxxxxxxxxx>]

On Wed, Aug 11, 2010 at 6:40 AM, Danilo Šegan <danilo@xxxxxxxxxxxxx> wrote:
> Hi Rob,
>
> У сре, 11. 08 2010. у 10:55 +1200, Robert Collins пише:
>
>> What we can do now is:
>>  - we receive the request
>>  - we commit the *intent* and return a 'pending' message to the
>> browser, which then renders a spinner locally and polls an API for
>> completion of the queued item
>>  - the queue processes the work
>>  - the polling js sees the work is complete and renders completion
>>
>> Now, queue latency and when the user gets their benefits. The current
>> job system has on average 30 second latency + processing time. We can
>> address that for specific queues with a small amount of work to poll
>> the DB every second in a new transaction.
>>
>> This should be pretty transparent to the user, will make the page time
>> come -way- down, and handle very large team rearrangements gracefully.
>
> We want this for POFile exports as well.  At the moment, we have an
> export queue and we notify the user by email.  In most cases, we can
> figure out that the generation of the POFile will be quick (even doable
> in 2s[1]), but because there are many (0.5%?) POFiles where it wouldn't,
> we have this old queue mechanism.  Users hate it.


I think this shows that if we run everything through a queue, there
needs to be a way to run really quick tasks in parallel with really
big tasks. You wouldn't want a single POFile export for a project to
be stuck behind the processing of 75k files for all of ubuntu. There
could be a slow lane and a fast lane. If the jobs have a method for
estimating the length of time reasonably well, we could also have a
super-fast lane, where the page decides not to queue the task at all.

Though _fillTeamParticipation() should run sufficiently quick for most
teams after turning it into a single query, I looked at the worst case
scenario, which is the team with the most members (3,800) being added
to the team with the most super teams (60). That could require 228,000
(60 * 3,800) new rows in TeamParticipation, assuming no overlap with
existing members of the new super teams. In that case, queued jobs
would probably need to be split apart into smaller queued jobs that
could be processed reasonably quickly in a single db transaction. Of
course, processing the same type of job in parallel slow/fast lanes
could wreak havoc on the TeamParticipation table if adding and removal
of team members is done out of order.

-Edwin

> If this can be made easier to implement and support, I'd be very
> happy :)
>
> Cheers,
> Danilo
>
> [1] For a full language pack export for Ubuntu, containing ~75k files,
> we produce it in 9h or roughly 0.5s per POFile.
>
> PS. To be honest, in the past, entire PO file exports were not doable in
> 5s for anything but the most trivial cases.  We've since improved so can
> think about better solutions for users.