launchpad-dev team mailing list archive

[Karl Fogel <karl.fogel@xxxxxxxxxxxxx>]

Julian Edwards <julian.edwards@xxxxxxxxxxxxx> writes:
> We *could* generate a daily tarball automatically if it helps people on flaky 
> connections to get hold of our source.

I can't think of any other way to help people on flaky connections.
Generating a daily downloadable tarball is one solution, but see below
for another workaround.

> From memory of the conversation on IRC, he said that he can resume http 
> downloads if the session drops.  I don't think bzr pull does that does it?  
> Maybe it should! :)

No, Bazaar doesn't do resumption in any of its access methods (sure it
would be great if it did, but it doesn't right now).  However, John
Arbash Meinel just sent an interesting workaround via private mail:

> I feel like I'm starting in the middle, rather than at the beginning,
> but I'll give it a shot.
> 
> At least reading here:
> https://dev.launchpad.net/Getting
> 
> It doesn't say anything about a "launchpad.tar.gz" it just says to grab
> the latest "rocketfuel-setup" script using 'bzr cat' and then run it.
> 
> I guess that is the 'rf-setup' that is mentioned?
> 
> If the specific problem is the time it is taking for the download of the
> launchpad codebase, then you can do something like this:
> 
> bzr init-repo --2a ~/launchpad/lp-branches
> cd ~/launchpad/lp-branches
> 
> STEP_SIZE=500
> LAST_REVNO=`bzr revno lp:launchpad`
> bzr branch lp:launchpad -r $STEP_SIZE launchpad-incremental
> cd launchpad-incremental
> for i in `seq $STEP_SIZE $STEP_SIZE $LAST_REVNO`; do
>   bzr pull -r $i
> done
> bzr pull -r $LAST_REVNO
> 
> cd ..
> rm -rf launchpad-incremental
> 
> 
> That will seed the shared repository with the launchpad history, and
> only pull down a portion of the data at a time. After doing that, you
> should be able to run "rocketfuel-setup" as normal, and when it grabs
> the appropriate launchpad branches, you should already have most of the
> history locally, so it should go fast.
> 
> My bash-fu isn't great, but I think if one of those 'bzr pull' lines
> fails, it won't kill the for loop. Instead it will just try again,
> though the next time it will be trying to download 2x the data. If your
> connection is a little flakey, that will probably be ok. If you are
> having a lot of problems, you can set the STEP_SIZE as small as you want.

Yoel, is your scripting ability good enough to give this a try?  If so,
can you please modify rocketfuel-setup.sh to try this method?  If it
works well for you on a flaky connection, I'll enthusiastically help
shepherd the patch into rocketfuel-setup in trunk.

If you can't, I'll have a go at it myself; but it's always better to
keep the change as close to the tester as possible, that's why I'm
asking you first if you can try it.

Thanks,
-Karl