openstack team mailing list archive

["Dan Prince" <dan.prince@xxxxxxxxxxxxx>]

Hi Jay,
 

The tests should not be running concurrently.
 
We currently have 4 Natty Cloud Servers configured as unit test workers. The machines are shared between nova/glance/keystone.
 
The unit test runner is pretty simple:
 
https://github.com/dprince/smokestack/blob/master/app/templates/unittest_runner.sh.erb
 
Would using a unique FAKE_FILESYSTEM_ROOTDIR for each test run or test help out here? Perhaps prefixing each one w/ /tmp/glance-tests so that we could also add an explicit tmp dir cleanup command in the test suite runners as well.
 
In the meantime I'm happy to add an explicit cleanup for /tmp/glance-tests to make sure we have a clean slate for each test run.
 
Dan
 
-----Original Message-----
From: "Jay Pipes" <jaypipes@xxxxxxxxx>
Sent: Tuesday, December 13, 2011 7:14am
To: "Dan Prince" <dan.prince@xxxxxxxxxxxxx>
Cc: "Openstack" <openstack@xxxxxxxxxxxxxxxxxxx>
Subject: Re: [Openstack] unit and integration tests results for Gerrit


Thanks Dan, looks like a great start on this. Not sure what's going on
with the unit test runs in Glance, though... As an example, see:

http://smokestack.openstack.org/?go=/jobs/5600

There are dozens of errors like this:

======================================================================
ERROR: test_add_member (glance.tests.unit.test_api.TestGlanceAPI)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/tmp/tmp.ZaWdEoPfhP/glance_source/glance/tests/unit/test_api.py",
line 1937, in setUp
 stubs.stub_out_filesystem_backend()
 File "/tmp/tmp.ZaWdEoPfhP/glance_source/glance/tests/stubs.py", line
63, in stub_out_filesystem_backend
 os.mkdir(FAKE_FILESYSTEM_ROOTDIR)
OSError: [Errno 17] File exists: '/tmp/glance-tests'

That FAKE_FILESYSTEM_ROOTDIR is cleaned up in the tearDown() method of
API unit tests, and these tests run perfectly fine for me locally. I
was thinking that one of the following things may be occurring:

* The unlink() of the FAKE_FILESYSTEM_ROOTDIR is not fsync'ing fast
enough, resulting in setUp()'s call to create the /tmp/glance-tests
directory is stumbling over itself
* The tests are being run in parallel somehow?

I've seen the fsync behaviour cause havoc in some of the image cache
tests before, and the solution ended up putting a small wait loop in
the test code to wait until disk buffers were flushed and a cache file
was fully removed from the filesystem. That may be happening here?
These are being run on Cloud Servers, right?

/me just trying to figure out why tests would run differently in
smokestack than Jenkins or locally...

Thanks in advance for any insight.

Cheers,
-jay