← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #43791

[Bug 1529785] [NEW] Create snapshot on VM containing multiple SRIOV vNICs configured with the same MAC fails

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Use case: SRIOV protection on the guest VM with Intel NICs on the host
requires (at one scenario) 2 SRIOV ports attached to different physical
NICs (e.g. phyNet1 & phyNet2) configured with the same MAC address (and
as a result, same IP).

Problem: Create snapshot on such VM fails

Reason: (from nova-compute.log)
2015-12-28 12:03:37.594 4616 ERROR oslo_messaging.rpc.dispatcher [req-57a8c147-b945-43e8-9915-5a52d3b7deb9 18800eca7d674a60995039349089e8da 0b5c257568424704854f8d10342edf80 - - -] Exception during message handling: operation failed: multiple devices matching mac address fa:16:3e:e1:7f:3f found
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6917, in snapshot_instance
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     return self.manager.snapshot_instance(ctxt, image_id, instance)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     payload)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 341, in decorated_function
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 312, in decorated_function
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in decorated_function
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 357, in decorated_function
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 417, in decorated_function
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     instance=instance)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 407, in decorated_function
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     *args, **kwargs)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3297, in snapshot_instance
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     task_states.IMAGE_SNAPSHOT)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3327, in _snapshot_instance
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     update_task_state)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1422, in snapshot
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     self._detach_sriov_ports(context, instance, virt_dom)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3121, in _detach_sriov_ports
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     libvirt.VIR_DOMAIN_AFFECT_LIVE)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     rv = execute(f, *args, **kwargs)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(c, e, tb)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     rv = meth(*args, **kwargs)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1193, in detachDeviceFlags
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher libvirtError: operation failed: multiple devices matching mac address fa:16:3e:e1:7f:3f found
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher

Observation: when trying to manually call "virsh detach-interface instance-0000005b hostdev --mac fa:16:3e:e1:7f:3f" (MAC addr being the MAC addr of both SRIOV ports) the result is:
error: Domain has multiple interfaces matching MAC address fa:16:3e:e1:7f:3f. You must use detach-device and specify the device pci address to remove it.

Solution: in driver.py call the equivalent of "virsh detach-device <pci addr>" instead of: guest.detach_device(cfg, live=True) (cfg being the <interface>...</interface> XML (e.g.:
<interface type="hostdev" managed="yes">
  <mac address="fa:16:3e:e1:7f:3f"/>
  <source>
    <address type="pci" domain="0x0000" bus="0x05" slot="0x1e" function="0x6"/>
  </source>
  <vlan>
    <tag id="2145"/>
  </vlan>
</interface>
)

** Affects: nova
     Importance: Undecided
     Assignee: David Edery (david-edery)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => David Edery (david-edery)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1529785

Title:
  Create snapshot on VM containing multiple SRIOV vNICs configured with
  the same MAC fails

Status in OpenStack Compute (nova):
  New

Bug description:
  Use case: SRIOV protection on the guest VM with Intel NICs on the host
  requires (at one scenario) 2 SRIOV ports attached to different
  physical NICs (e.g. phyNet1 & phyNet2) configured with the same MAC
  address (and as a result, same IP).

  Problem: Create snapshot on such VM fails

  Reason: (from nova-compute.log)
  2015-12-28 12:03:37.594 4616 ERROR oslo_messaging.rpc.dispatcher [req-57a8c147-b945-43e8-9915-5a52d3b7deb9 18800eca7d674a60995039349089e8da 0b5c257568424704854f8d10342edf80 - - -] Exception during message handling: operation failed: multiple devices matching mac address fa:16:3e:e1:7f:3f found
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6917, in snapshot_instance
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     return self.manager.snapshot_instance(ctxt, image_id, instance)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     payload)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 341, in decorated_function
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 312, in decorated_function
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in decorated_function
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 357, in decorated_function
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 417, in decorated_function
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     instance=instance)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 407, in decorated_function
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     *args, **kwargs)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3297, in snapshot_instance
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     task_states.IMAGE_SNAPSHOT)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3327, in _snapshot_instance
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     update_task_state)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1422, in snapshot
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     self._detach_sriov_ports(context, instance, virt_dom)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3121, in _detach_sriov_ports
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     libvirt.VIR_DOMAIN_AFFECT_LIVE)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     rv = execute(f, *args, **kwargs)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(c, e, tb)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     rv = meth(*args, **kwargs)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1193, in detachDeviceFlags
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher libvirtError: operation failed: multiple devices matching mac address fa:16:3e:e1:7f:3f found
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher

  Observation: when trying to manually call "virsh detach-interface instance-0000005b hostdev --mac fa:16:3e:e1:7f:3f" (MAC addr being the MAC addr of both SRIOV ports) the result is:
  error: Domain has multiple interfaces matching MAC address fa:16:3e:e1:7f:3f. You must use detach-device and specify the device pci address to remove it.

  Solution: in driver.py call the equivalent of "virsh detach-device <pci addr>" instead of: guest.detach_device(cfg, live=True) (cfg being the <interface>...</interface> XML (e.g.:
  <interface type="hostdev" managed="yes">
    <mac address="fa:16:3e:e1:7f:3f"/>
    <source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x1e" function="0x6"/>
    </source>
    <vlan>
      <tag id="2145"/>
    </vlan>
  </interface>
  )

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1529785/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next