yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1529785] Re: Create snapshot fails on VM containing multiple SRIOV vNICs configured with the same MAC

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1529785@xxxxxxxxxxxxxxxxxx>
Date: Thu, 18 Feb 2016 15:11:25 -0000
Reply-to: Bug 1529785 <1529785@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/262341
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f93d808bc3ba1c0a97ec46f345fad9c0d00865f8
Submitter: Jenkins
Branch:    master

commit f93d808bc3ba1c0a97ec46f345fad9c0d00865f8
Author: David Edery <david.edery@xxxxxxxxxxxxxxxxxx>
Date:   Wed Dec 30 02:05:19 2015 +0200

    Fix create snapshot failure on VMs with SRIOV
    
    One use-case of guest VM network protection using SRIOV ports is having
    two direct ports connected to a VM, each one is related to a network that
    is connected to a different physical NIC on the host (e.g. phyNet1 on
    eth0 and phyNet2 on eth1). In this use-case, due to some physical NICs
    limitations it's advised to configure both ports with the same MAC
    address (or else outgoing or incoming traffic will not reach its
    destination in case of fail-over).
    
    Snapshot creation on such VMs fails since libvirt's detach-interface
    doesn't know which interface of the two to detach and fails. This is why
    I've changed the call inside _detach_sriov_ports from (the equivalent
    of) detach-interface to _detach_pci_devices with the source-device
    pci address of the SRIOV VF (always present in the context of SRIOV
    of course).
    
    This fix was tested on a real environment containing the above type of
    VMs. test_driver.test_detach_sriov_ports was slightly modified so
    that the VIF from which data is sent to _detach_pci_devices will
    contain the correct SRIOV values (pci_slot, vlan and hw_veb VIF type)
    
    Change-Id: If3edc1965c01a077eb61984a442e0d778d870d75
    Closes-Bug: #1529785


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1529785

Title:
  Create snapshot fails on VM containing multiple SRIOV vNICs configured
  with the same MAC

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Use case: 
  SRIOV protection on the guest VM with Intel NICs on the host requires (at one scenario) 2 SRIOV ports attached to different physical NICs (e.g. phyNet1 & phyNet2) configured with the same MAC address (and as a result, same IP).

  Problem: 
  Create snapshot on such VM fails

  Reason: 
  (from nova-compute.log)
  2015-12-28 12:03:37.594 4616 ERROR oslo_messaging.rpc.dispatcher [req-57a8c147-b945-43e8-9915-5a52d3b7deb9 18800eca7d674a60995039349089e8da 0b5c257568424704854f8d10342edf80 - - -] Exception during message handling: operation failed: multiple devices matching mac address fa:16:3e:e1:7f:3f found
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6917, in snapshot_instance
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     return self.manager.snapshot_instance(ctxt, image_id, instance)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     payload)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 341, in decorated_function
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 312, in decorated_function
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in decorated_function
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 357, in decorated_function
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 417, in decorated_function
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     instance=instance)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 407, in decorated_function
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     *args, **kwargs)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3297, in snapshot_instance
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     task_states.IMAGE_SNAPSHOT)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3327, in _snapshot_instance
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     update_task_state)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1422, in snapshot
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     self._detach_sriov_ports(context, instance, virt_dom)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3121, in _detach_sriov_ports
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     libvirt.VIR_DOMAIN_AFFECT_LIVE)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     rv = execute(f, *args, **kwargs)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     six.reraise(c, e, tb)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     rv = meth(*args, **kwargs)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1193, in detachDeviceFlags
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher libvirtError: operation failed: multiple devices matching mac address fa:16:3e:e1:7f:3f found
  2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher

  Observation: 
  When trying to manually call "virsh detach-interface instance-0000005b hostdev --mac fa:16:3e:e1:7f:3f" (MAC addr being the MAC addr of both SRIOV ports) the result is:
  error: Domain has multiple interfaces matching MAC address fa:16:3e:e1:7f:3f. You must use detach-device and specify the device pci address to remove it.

  Solution: 
  In driver.py call the equivalent of "virsh detach-device <pci addr>" instead of: guest.detach_device(cfg, live=True) - cfg being the <interface>...</interface> XML 
  (e.g.:
  <interface type="hostdev" managed="yes">
    <mac address="fa:16:3e:e1:7f:3f"/>
    <source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x1e" function="0x6"/>
    </source>
    <vlan>
      <tag id="2145"/>
    </vlan>
  </interface>)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1529785/+subscriptions
References

[Bug 1529785] [NEW] Create snapshot on VM containing multiple SRIOV vNICs configured with the same MAC fails
From: David Edery, 2015-12-29