yellow team mailing list archive

[Gary Poster <gary.poster@xxxxxxxxxxxxx>]

On 03/20/12 23:58, Gary Poster wrote:
> 1) The containers (in the branch-rewriter) are looking for files like
> this, according to lsof:
> /dependencies/sourcecode/pygettextpo/gettextpo.so . I'm not sure why.
>
> 2) The containers on the 8-core machine are waiting on fd 11 when they
> wait. If I'm using lsof correctly, that means it is waiting on
> /rootfs/dev/random (see bug 959352), which didn't exist. I tried making
> a symlink but it didn't change anything, so maybe that's not the cause.
> I was hopeful that this was the cause. :-/ It still is something we
> should get sorted.
>
> 3) Eventually, it turns out, the code proceeds anyway. We eventually got
> just four test failures http://pastebin.ubuntu.com/893076/ in 3 hrs, 46
> mins, 49 secs, but I think we need to fix or work around that bug from
> #1 before we get too disheartened that eight cores only gives us two or
> three hours speed increase. I think, once we get rid of these hangs,
> things will look noticeably better.

Here's strace starting with the hang, that eventually is not a hang:

read(11, "\305F\237\221\2\376~\262", 4096) = 8
close(11)                               = 0
munmap(0xf72bd000, 4096)                = 0
stat64("/var/lib/buildbot/dependencies/eggs/dnspython-1.7.1-py2.6.egg/dns/hashlib", 
0xff846ac8) = -1 ENOENT (No such file or directory)
open("/var/lib/buildbot/dependencies/eggs/dnspython-1.7.1-py2.6.egg/dns/hashlib.so", 
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/var/lib/buildbot/dependencies/eggs/dnspython-1.7.1-py2.6.egg/dns/hashlibmodule.so", 
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/var/lib/buildbot/dependencies/eggs/dnspython-1.7.1-py2.6.egg/dns/hashlib.py", 
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/var/lib/buildbot/dependencies/eggs/dnspython-1.7.1-py2.6.egg/dns/hashlib.pyc", 
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
close(8)                                = 0
stat64("/var/lib/buildbot/dependencies/eggs/dnspython-1.7.1-py2.6.egg/dns/rcode", 
0xff84a658) = -1 ENOENT (No such file or directory)
open("/var/lib/buildbot/dependencies/eggs/dnspython-1.7.1-py2.6.egg/dns/rcode.so", 
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/var/lib/buildbot/dependencies/eggs/dnspython-1.7.1-py2.6.egg/dns/rcodemodule.so", 
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/var/lib/buildbot/dependencies/eggs/dnspython-1.7.1-py2.6.egg/dns/rcode.py", 
O_RDONLY|O_LARGEFILE) = 8
fstat64(8, {st_mode=S_IFREG|0644, st_size=3108, ...}) = 0
open("/var/lib/buildbot/dependencies/eggs/dnspython-1.7.1-py2.6.egg/dns/rcode.pyc", 
O_RDONLY|O_LARGEFILE) = 11
fstat64(11, {st_mode=S_IFREG|0644, st_size=3182, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xf72bd000
read(11, 
"\321\362\r\n\10\326hOc\0\0\0\0\0\0\0\0\7\0\0\0@\0\0\0s1\1\0\0d\0"..., 
4096) = 3182
fstat64(11, {st_mode=S_IFREG|0644, st_size=3182, ...}) = 0
read(11, "", 4096)                      = 0
close(11)                               = 0
munmap(0xf72bd000, 4096)                = 0
close(8)                                = 0

as I said, as best I could tell, fd 11 was /dev/random that I had 
symlinked over to /rootfs/dev/random.  Maybe the randomness is 
exhausted?  ssh uses a lot of it, I understand...that would explain why 
it doesn't happen when benji or I try it locally on our desktops...8 
processes using ssh at once might use a lot of entropy.