openstack team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

On Mon, Dec 12, 2011 at 4:45 PM, Dan Prince <dan.prince@xxxxxxxxxxxxx> wrote:
> Hello all,
>
> We just turned on a Bellows feature that will automatically update Gerrit
> reviews with SmokeStack test results. Each Gerrit review should have a
> comment that looks something like this:
>
> SmokeStack Results (patch set 2):
>
> Unit Success: http://smokestack.openstack.org/?go=/jobs/5570
> Libvirt Success: http://smokestack.openstack.org/?go=/jobs/5568
> XenServer Success: http://smokestack.openstack.org/?go=/jobs/5569

Thanks Dan, looks like a great start on this. Not sure what's going on
with the unit test runs in Glance, though... As an example, see:

http://smokestack.openstack.org/?go=/jobs/5600

There are dozens of errors like this:

======================================================================
ERROR: test_add_member (glance.tests.unit.test_api.TestGlanceAPI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.ZaWdEoPfhP/glance_source/glance/tests/unit/test_api.py",
line 1937, in setUp
    stubs.stub_out_filesystem_backend()
  File "/tmp/tmp.ZaWdEoPfhP/glance_source/glance/tests/stubs.py", line
63, in stub_out_filesystem_backend
    os.mkdir(FAKE_FILESYSTEM_ROOTDIR)
OSError: [Errno 17] File exists: '/tmp/glance-tests'

That FAKE_FILESYSTEM_ROOTDIR is cleaned up in the tearDown() method of
API unit tests, and these tests run perfectly fine for me locally. I
was thinking that one of the following things may be occurring:

* The unlink() of the FAKE_FILESYSTEM_ROOTDIR is not fsync'ing fast
enough, resulting in setUp()'s call to create the /tmp/glance-tests
directory is stumbling over itself
* The tests are being run in parallel somehow?

I've seen the fsync behaviour cause havoc in some of the image cache
tests before, and the solution ended up putting a small wait loop in
the test code to wait until disk buffers were flushed and a cache file
was fully removed from the filesystem. That may be happening here?
These are being run on Cloud Servers, right?

/me just trying to figure out why tests would run differently in
smokestack than Jenkins or locally...

Thanks in advance for any insight.

Cheers,
-jay