yahoo-eng-team team mailing list archive

[OpenStack Infra <1696125@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/441204
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=563c0927d14d052e3f1fad80df95fe4a7c48d38b
Submitter: Jenkins
Branch:    master

commit 563c0927d14d052e3f1fad80df95fe4a7c48d38b
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Tue Jun 6 14:08:37 2017 -0400

    libvirt: Check if domain is persistent before detaching devices
    
    Previously the libvirt driver would always assume that it was only
    detaching devices (volumes or virtual interfaces) from a persistent
    domain however that is not always the case.
    
    For example when rolling back from a live migration an attempt is made
    to detach volumes from the transient destination domain that is being
    cleaned up. This attempt would fail with the previous assumption of the
    domain being persistent in place.
    
    This change introduces a simple call to has_persistent_configuration
    within detach_device_with_retry to confirm the state of the domain
    before attempting to detach.
    
    Closes-Bug: #1669857
    Closes-Bug: #1696125
    Change-Id: I95948721a0119f5f54dbe50d4455fd47d422164b


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1696125

Title:
  Detach interface failed - Unable to detach from guest transient domain
  (pike)

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Seeing this in Tempest runs on master (pike):

  http://logs.openstack.org/24/471024/2/check/gate-tempest-dsvm-neutron-
  linuxbridge-ubuntu-
  xenial/6b98d38/logs/screen-n-cpu.txt.gz?level=TRACE#_Jun_06_02_16_02_855503

  Jun 06 02:16:02.855503 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: WARNING nova.compute.manager [None req-b4a50024-a2fd-4279-b284-340d2074f1c1 tempest-TestNetworkBasicOps-1479445685 tempest-TestNetworkBasicOps-1479445685] [instance: 2668bcb9-b13d-4b5b-8ee5-edbdee3b15a8] Detach interface failed, port_id=3843caa3-ab04-45f1-94d8-f330390e40fe, reason: Device detach failed for fa:16:3e:ab:e3:3f: Unable to detach from guest transient domain.: DeviceDetachFailed: Device detach failed for fa:16:3e:ab:e3:3f: Unable to detach from guest transient domain.
  Jun 06 02:16:02.884007 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server [None req-b4a50024-a2fd-4279-b284-340d2074f1c1 tempest-TestNetworkBasicOps-1479445685 tempest-TestNetworkBasicOps-1479445685] Exception during message handling: InterfaceDetachFailed: Failed to detach network adapter device from 2668bcb9-b13d-4b5b-8ee5-edbdee3b15a8
  Jun 06 02:16:02.884180 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Jun 06 02:16:02.884286 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
  Jun 06 02:16:02.884395 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Jun 06 02:16:02.884538 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
  Jun 06 02:16:02.884669 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Jun 06 02:16:02.884777 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
  Jun 06 02:16:02.884869 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Jun 06 02:16:02.884968 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 77, in wrapped
  Jun 06 02:16:02.885069 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  Jun 06 02:16:02.885171 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jun 06 02:16:02.885272 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jun 06 02:16:02.885367 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jun 06 02:16:02.885461 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jun 06 02:16:02.885554 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 68, in wrapped
  Jun 06 02:16:02.885649 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Jun 06 02:16:02.885755 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 214, in decorated_function
  Jun 06 02:16:02.885856 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  Jun 06 02:16:02.885950 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jun 06 02:16:02.886053 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jun 06 02:16:02.886143 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jun 06 02:16:02.886232 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jun 06 02:16:02.886322 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 202, in decorated_function
  Jun 06 02:16:02.886415 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jun 06 02:16:02.886505 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 5224, in detach_interface
  Jun 06 02:16:02.886601 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     raise exception.InterfaceDetachFailed(instance_uuid=instance.uuid)
  Jun 06 02:16:02.886695 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server InterfaceDetachFailed: Failed to detach network adapter device from 2668bcb9-b13d-4b5b-8ee5-edbdee3b15a8
  Jun 06 02:16:02.886794 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server 

  This probably started with: https://review.openstack.org/#/c/349014/
  (merged 5/31)

  http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Detach%20interface%20failed%5C%22%20AND%20message:%5C%22reason:%20Device%20detach%20failed%20for%5C%22%20AND%20message:%5C%22Unable%20to%20detach%20from%20guest%20transient%20domain%5C%22%20AND%20tags:%5C%22screen-n-cpu.txt%5C%22&from=10d

  Might need something like this:
  https://review.openstack.org/#/c/441204/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1696125/+subscriptions