yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #04737
[Bug 1162047] Re: lockutils.synchronized does not work properly if lock_path=none
** Changed in: nova
Status: Fix Committed => Fix Released
** Changed in: nova
Milestone: None => havana-3
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1162047
Title:
lockutils.synchronized does not work properly if lock_path=none
Status in OpenStack Compute (Nova):
Fix Released
Status in Oslo - a Library of Common OpenStack Code:
Fix Released
Bug description:
We discovered today that two processes will not lock properly if
external=True and lock_path=None.
nova/openstack/common/lockutils.py tries to use tempfile.mkdtemp() to
create a temporary directory for tracking the locking but this doesn't
work as tempfile.mkdtemp() returns two different temporary
directories. Here is an example of a simple test case running using
locking that demonstrates the problem:
openstack@issue147007:~/devel$ ./run_tests.sh -N nova.tests.test_sync
Running ` python setup.py testr --slowest --testr-args='--subunit nova.tests.test_sync'`
running testr
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --list
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --load-list /tmp/tmpsB1Gxo
running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --load-list /tmp/tmpo09Jxy
('Running modified test_sync!!!! CONF.lock_path is %s', None)
('Running modified test_sync!!!! CONF.lock_path is %s', None)
('JSBRYANT: local_lock_path is %s', '/tmp/tmp80M3nJ')
Thread 1 Start
Thread 1 End
nova.tests.test_sync.TestLocking.test_one (subunit.RemotedTestCase)
nova.tests.test_sync.TestLocking.test_one ... ok
('JSBRYANT: local_lock_path is %s', '/tmp/tmp898Hj_')
Thread 2 Start
Thread 2 End
nova.tests.test_sync.TestLocking.test_two (subunit.RemotedTestCase)
nova.tests.test_sync.TestLocking.test_two ... ok
Slowest Tests
Test id Runtime (s)
----------------------------------------- -----------
nova.tests.test_sync.TestLocking.test_two 120.102
nova.tests.test_sync.TestLocking.test_one 60.062
----------------------------------------------------------------------
Ran 2 tests in 122.583s
Note the two test cases shouldn't run in parallel but they do as the
locking isn't working. The output shows the two different lock
directories which were created here:
if external and not CONF.disable_process_locking:
LOG.debug(_('Attempting to grab file lock "%(lock)s" '
'for method "%(method)s"...'),
{'lock': name, 'method': f.__name__})
cleanup_dir = False
# We need a copy of lock_path because it is non-local
local_lock_path = lock_path
if not local_lock_path:
local_lock_path = CONF.lock_path
if not local_lock_path:
cleanup_dir = True
local_lock_path = tempfile.mkdtemp()
print("JSBRYANT: local_lock_path is %s",
local_lock_path)
if not os.path.exists(local_lock_path):
fileutils.ensure_tree(local_lock_path)
This can be worked around by either specifying the lock_path on the call to synchronized or by updating the conf file to have a lock_path. Regardless, this seems like something that cause unpredictable behavior for users who don't get those options into their conf files.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1162047/+subscriptions
