← Back to team overview

yellow team mailing list archive

Re: Odd ec2 behavior

 

Thanks, Brad.

Separately, I discovered (and confirmed per your suggestion, gmb) that
our new zope.testing branch was breaking the subunit output.  I think
gmb knows why.  StevenK filed bug 1003696 and I think this was also the
cause.  At my suggestion he reverted the pertinent revision.

Gary

On 05/23/2012 07:20 PM, Brad Crittenden wrote:
> Suggested soundtrack: http://www.youtube.com/watch?v=ivWfRQ4V5ao
> 
> Today Gary and I both ran into the same problem with running parallel tests on ec2.  The symptom is that /var/tmp/bazaar.launchpad.dev is being created by the 'make build' step and it is owned by root with group/other write access.  Tests that need to write to that directory or children fail with permission denied errors.
> 
> When run locally, 'make clean' will remove /var/tmp/bazaar.launchpad.dev and 'make build' does not create it.  That's great because any codehosting tests that need the directory will create it on demand and we don't see any problems.
> 
> Also, when inside an lxc container (from the slave) running 'make clean build' works as expected too.  Hurrah.
> 
> But the 'make build' step, when run from /usr/local/bin/lp-setup-lxc-build, results in /var/tmp/bazaar.launchpad.dev being created with the wrong ownership.  It is unclear why 'make build' creates that directory.
> 
> Findings:
> 
> 1) The LP Makefile has not changed recently to cause this problem,
> 
> 2) Gary and I both changed lp-setup-lxc-build to run each step individually and print out 'ls /var/tmp' between.  The modified script is at http://paste.ubuntu.com/1003875/ and the results are at http://pastebin.ubuntu.com/1003905/
> 
> The results are a little hard to read because the commands were not echoed.  At line 121 'make clean' has just completed and 'ls /var/tmp' executed on the container.  It shows there is no bazaar.launchpad.dev.  'make build' is then called.
> 
> It completes at line 177 and then 'ls /var/tmp' is run and shows:  
> 
> drwxr-xr-x  2 buildbot buildbot 4096 May 23 19:24 archive
> drwxr-xr-x  2 root     root     4096 May 23 23:04 bazaar.launchpad.dev
> drwxr-xr-x  5 buildbot buildbot 4096 May 23 23:06 launchpad_mailqueue
> drwxrwsr-x 10 buildbot buildbot 4096 May 23 23:06 mailman
> 
> So, 'make build' has created the directory and it is owned by root and unwritable.  
> 
> We cannot figure out why this is happening and why it just started.  Parallel tests on ec2 are stuck until we fix it.
> 
> --Brad
> 
> 



Follow ups

References