launchpad-dev team mailing list archive

[Michael Hudson <michael.hudson@xxxxxxxxxxxxx>]

Jonathan Lange wrote:
> On Sun, Nov 1, 2009 at 12:22 AM, Michael Hudson
> <michael.hudson@xxxxxxxxxxxxx> wrote:
>> Tom Haddon wrote:
>>> On Fri, 2009-10-30 at 16:14 +1300, Michael Hudson wrote:
>>>> Michael Hudson wrote:
>>>>> James Westby wrote:
>>>>>> On Thu Oct 29 14:42:59 +0000 2009 James Westby wrote:
>>>>>>> This means that in probably the next 2-24 hours I will get a notification that
>>>>>>> lucid is created and we should create the branches for it.
>>>>>> I have just recieved this notification. Lucid is now created and so we can
>>>>>> create branches for it.
>>>>> I'll work with a LOSA to get this done today.
>>>> So we couldn't run it today because I forgot that the script I wrote
>>>> equired adding a user to the database config of the production database
>>>> until after the people who can approve this had left for the day :-(
>>>>
>>>> Assuming such approval can be gotten tomorrow, here's what needs to be
>>>> done:
>>>>
>>>> 1) Get approval to add the 'branch-distro' dbuser from the first hunk of
>>>> https://pastebin.canonical.com/24075/ to the production database, and
>>>> allow codehost@crowberry to connect as this user.
>>> We got approval and I've opened up DB access.
>>>
>>>> 2) Run this command on crowberry:
>>>>
>>>> /srv/bazaar.launchpad.net/production/mwh-rev-8539-plus/scripts/branch-distro.py ubuntu karmic lucid
>>> This is running. It was called as follows:
>>>
>>> codehost@crowberry:/srv/bazaar.launchpad.net/production/mwh-db-stable
>>> $ ./scripts/branch-distro.py ubuntu karmic lucid >
>>> ~/mwh-branch-lucid.log 2>&1
>>>
>>>> Save the output somehow.  You can measure it's progress by seeing how
>>>> many branches are at https://code.edge.launchpad.net/ubuntu/lucid (it
>>>> should produce as many there as there are official branches in lucid,
>>>> which is a little under 16000).
>>>>
>>>> 3) The script may fall over, in which case the right thing to do is
>>>> restart it.  If it seems to be repeatedly falling over without making
>>>> progress (you can run with -v to check this) debugging will be required.
>>>> I recommend jml for this job :-)
>>> I'll keep an eye on it.
>> It ran fine, although re-pulling the 32000 change branches overloaded
>> the puller rather.
> 
> Do we have any plans for avoiding this at lucid+1?

Given that the puller didn't actually do anything to the branches at the
bzr level it should be possible to fake all of its effects on the
database (updating last_mirrored_id, stacked_on, etc) in the
branch-distro.py script.  The scanner too I guess (duplicate the
branchrevision rows across).

Making the puller much more efficient (using a process pool perhaps?)
would be another approach.

>>  You can see the new branches at:
>>
>>    https://code.edge.launchpad.net/ubuntu/lucid
>>
> 
> Yay.
> 
>> I think we should still restack all the branches that are stacked on the
>> karmic branches onto the lucid branches for better performance, but that
>> can be trickled through over the next few weeks, possibly even with a
>> Launchpad API using script rather than anything involving an admin.
>>
> 
> Although maybe that should be added to the base script for use with lucid+1?

I actually don't think so: the base script should do the minimum that
has to be done immediately post release to keep the run time down.
Adjusting the stacking isn't something that needs to be done then.  I
guess some back of the envelope calculations should be done to get an
idea how many branches will need to be restacked.

Cheers,
mwh