yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

** Changed in: nova
       Status: Confirmed => Fix Released

** Changed in: nova
     Assignee: (unassigned) => Matt Riedemann (mriedem)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1806126

Title:
  LibvirtRbdEvacuateTest and LibvirtFlatEvacuateTest tests race fail

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Seen here:

  http://logs.openstack.org/22/606122/7/check/nova-tox-
  functional/1f3126b/testr_results.html.gz

  There are different failures, but the tests were introduced with this
  change:

  https://review.openstack.org/#/c/591733/

  For example, in
  nova.tests.functional.regressions.test_bug_1550919.LibvirtRbdEvacuateTest.test_evacuate_shared_instancedir
  I'm seeing:

  2018-11-30 17:03:58,424 ERROR [nova.virt.libvirt.driver] Failed to
  start libvirt guest

  And this (during evacuate from compute0 to compute1 in the test):

  Traceback (most recent call last):
    File "nova/compute/manager.py", line 7897, in _error_out_instance_on_exception
      yield
    File "nova/compute/manager.py", line 3040, in rebuild_instance
      migration, request_spec)
    File "nova/compute/manager.py", line 3102, in _do_rebuild_instance_with_claim
      self._do_rebuild_instance(*args, **kwargs)
    File "nova/compute/manager.py", line 3265, in _do_rebuild_instance
      self._rebuild_default_impl(**kwargs)
    File "nova/compute/manager.py", line 2929, in _rebuild_default_impl
      block_device_info=new_block_device_info)
    File "nova/virt/libvirt/driver.py", line 3079, in spawn
      destroy_disks_on_failure=True)
    File "nova/virt/libvirt/driver.py", line 5648, in _create_domain_and_network
      destroy_disks_on_failure)
    File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "nova/virt/libvirt/driver.py", line 5607, in _create_domain_and_network
      self.plug_vifs(instance, network_info)
    File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1062, in __call__
      return _mock_self._mock_call(*args, **kwargs)
    File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1118, in _mock_call
      raise effect
  TestingException

  
  I'm guessing something is mocking global state and causing that failure...we likely need a fake virt driver that extends the libvirt driver and does what we want for these tests to avoid global mocks.

  
  In the other test, I see a timeout:

  nova.tests.functional.regressions.test_bug_1550919.LibvirtFlatEvacuateTest.test_evacuate_nonshared_instancedir

  And it seems to be stuck here:

  2018-11-30 17:03:05,111 INFO [nova.virt.libvirt.driver] Creating image

  
  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22INFO%20%5Bnova.virt.libvirt.driver%5D%20Creating%20image%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

  32 hits in the last 48 hours, check and gate, all failures.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1806126/+subscriptions