yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

Public bug reported:

Seen here:

http://logs.openstack.org/22/606122/7/check/nova-tox-
functional/1f3126b/testr_results.html.gz

There are different failures, but the tests were introduced with this
change:

https://review.openstack.org/#/c/591733/

For example, in
nova.tests.functional.regressions.test_bug_1550919.LibvirtRbdEvacuateTest.test_evacuate_shared_instancedir
I'm seeing:

2018-11-30 17:03:58,424 ERROR [nova.virt.libvirt.driver] Failed to start
libvirt guest

And this (during evacuate from compute0 to compute1 in the test):

Traceback (most recent call last):
  File "nova/compute/manager.py", line 7897, in _error_out_instance_on_exception
    yield
  File "nova/compute/manager.py", line 3040, in rebuild_instance
    migration, request_spec)
  File "nova/compute/manager.py", line 3102, in _do_rebuild_instance_with_claim
    self._do_rebuild_instance(*args, **kwargs)
  File "nova/compute/manager.py", line 3265, in _do_rebuild_instance
    self._rebuild_default_impl(**kwargs)
  File "nova/compute/manager.py", line 2929, in _rebuild_default_impl
    block_device_info=new_block_device_info)
  File "nova/virt/libvirt/driver.py", line 3079, in spawn
    destroy_disks_on_failure=True)
  File "nova/virt/libvirt/driver.py", line 5648, in _create_domain_and_network
    destroy_disks_on_failure)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "nova/virt/libvirt/driver.py", line 5607, in _create_domain_and_network
    self.plug_vifs(instance, network_info)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1062, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1118, in _mock_call
    raise effect
TestingException


I'm guessing something is mocking global state and causing that failure...we likely need a fake virt driver that extends the libvirt driver and does what we want for these tests to avoid global mocks.


In the other test, I see a timeout:

nova.tests.functional.regressions.test_bug_1550919.LibvirtFlatEvacuateTest.test_evacuate_nonshared_instancedir

And it seems to be stuck here:

2018-11-30 17:03:05,111 INFO [nova.virt.libvirt.driver] Creating image


http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22INFO%20%5Bnova.virt.libvirt.driver%5D%20Creating%20image%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

32 hits in the last 48 hours, check and gate, all failures.

** Affects: nova
     Importance: High
         Status: Confirmed


** Tags: gate-failure libvirt testing

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1806126

Title:
  LibvirtRbdEvacuateTest and LibvirtFlatEvacuateTest tests race fail

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seen here:

  http://logs.openstack.org/22/606122/7/check/nova-tox-
  functional/1f3126b/testr_results.html.gz

  There are different failures, but the tests were introduced with this
  change:

  https://review.openstack.org/#/c/591733/

  For example, in
  nova.tests.functional.regressions.test_bug_1550919.LibvirtRbdEvacuateTest.test_evacuate_shared_instancedir
  I'm seeing:

  2018-11-30 17:03:58,424 ERROR [nova.virt.libvirt.driver] Failed to
  start libvirt guest

  And this (during evacuate from compute0 to compute1 in the test):

  Traceback (most recent call last):
    File "nova/compute/manager.py", line 7897, in _error_out_instance_on_exception
      yield
    File "nova/compute/manager.py", line 3040, in rebuild_instance
      migration, request_spec)
    File "nova/compute/manager.py", line 3102, in _do_rebuild_instance_with_claim
      self._do_rebuild_instance(*args, **kwargs)
    File "nova/compute/manager.py", line 3265, in _do_rebuild_instance
      self._rebuild_default_impl(**kwargs)
    File "nova/compute/manager.py", line 2929, in _rebuild_default_impl
      block_device_info=new_block_device_info)
    File "nova/virt/libvirt/driver.py", line 3079, in spawn
      destroy_disks_on_failure=True)
    File "nova/virt/libvirt/driver.py", line 5648, in _create_domain_and_network
      destroy_disks_on_failure)
    File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "nova/virt/libvirt/driver.py", line 5607, in _create_domain_and_network
      self.plug_vifs(instance, network_info)
    File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1062, in __call__
      return _mock_self._mock_call(*args, **kwargs)
    File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1118, in _mock_call
      raise effect
  TestingException

  
  I'm guessing something is mocking global state and causing that failure...we likely need a fake virt driver that extends the libvirt driver and does what we want for these tests to avoid global mocks.

  
  In the other test, I see a timeout:

  nova.tests.functional.regressions.test_bug_1550919.LibvirtFlatEvacuateTest.test_evacuate_nonshared_instancedir

  And it seems to be stuck here:

  2018-11-30 17:03:05,111 INFO [nova.virt.libvirt.driver] Creating image

  
  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22INFO%20%5Bnova.virt.libvirt.driver%5D%20Creating%20image%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

  32 hits in the last 48 hours, check and gate, all failures.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1806126/+subscriptions