yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1352732] Re: live migration fails:CPU feature `erms' specified more than once

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Fri, 05 Sep 2014 09:56:00 -0000
Reply-to: Bug 1352732 <1352732@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => juno-3

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1352732

Title:
  live migration fails:CPU feature `erms' specified more than once

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  When I live migrate VMs from one node to another(both are the same
  species),  nova service at the destination node throws exception
  below, my physical compute nodes are both ubuntu 14.04lts and
  openstack git stable/icehouse

  (I notice there are similar bug reports like
  https://bugs.launchpad.net/nova/+bug/1303536, but the codes throwing
  exception seem not the same )

  /usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py
  4229         # Compare CPU
  4230         source_cpu_info = src_compute_info['cpu_info']
  4231         self._compare_cpu(source_cpu_info)

  Refer to http://libvirt.org/html/libvirt-libvirt.html#virCPUCompareResult
  2014-08-05 14:20:11.027 914 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: XML error: CPU feature `erms' specified more than once
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     payload)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw) 
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 4440, in check_can_live_migrate_destination
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     block_migration, disk_over_commit)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4231, in check_can_live_migrate_destination
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     self._compare_cpu(source_cpu_info)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4373, in _compare_cpu
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     LOG.error(m, {'ret': ret, 'u': u})
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4369, in _compare_cpu
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     ret = self._conn.compareCPU(cpu.to_xml(), 0)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     rv = execute(f,*args,**kwargs)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     rv = meth(*args,**kwargs)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3191, in compareCPU
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virConnectCompareCPU() failed', conn=self)
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher libvirtError: XML error: CPU feature `erms' specified more than once
  2014-08-05 14:20:11.027 914 TRACE oslo.messaging.rpc.dispatcher 
  2014-08-05 14:20:11.031 914 ERROR oslo.messaging._drivers.common [-] Returning exception XML error: CPU feature `erms' specified more than once to caller
  2014-08-05 14:20:11.031 914 ERROR oslo.messaging._drivers.common [-] Returning exception XML error: CPU feature `erms' specified more than once to caller
  2014-08-05 14:20:11.031 914 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped\n    payload)\n', '  File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped\n    return f(self, context, *args, **kw)\n', '  File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function\n    e, sys.exc_info())\n', '  File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 4440, in check_can_live_migrate_destination\n    block_migration, disk_over_commit)\n', '  File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4231, in check_can_live_migrate_destination\n    self._compare_cpu(source_cpu_info)\n', '  File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4373, in _compare_cpu\n    LOG.error(m, {\'ret\': ret, \'u\': u})\n', '  File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/local/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4369, in _compare_cpu\n    ret = self._conn.compareCPU(cpu.to_xml(), 0)\n', '  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit\n    result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker\n    rv = meth(*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3191, in compareCPU\n    if ret == -1: raise libvirtError (\'virConnectCompareCPU() failed\', conn=self)\n', "libvirtError: XML error: CPU feature `erms' specified more than once\n"]
  2014-08-05 14:20:57.206 914 DEBUG nova.openstack.common.periodic_task [-] Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /usr/local/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py:178
  2014-08-05 14:20:57.207 914 DEBUG nova.openstack.common.periodic_task [-] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /usr/local/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py:178
  2014-08-05 14:20:57.207 914 DEBUG nova.openstack.common.periodic_task [-] Running periodic task ComputeManager.update_available_resource run_periodic_tasks /usr/local/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py:178


  
  my cpu info is :

  .....
  processor	: 39
  vendor_id	: GenuineIntel
  cpu family	: 6
  model		: 62
  model name	: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
  stepping	: 4
  microcode	: 0x416
  cpu MHz		: 3000.011
  cache size	: 25600 KB
  physical id	: 1
  siblings	: 20
  core id		: 12
  cpu cores	: 10
  apicid		: 57
  initial apicid	: 57
  fpu		: yes
  fpu_exception	: yes
  cpuid level	: 13
  wp		: yes
  flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
  bogomips	: 6001.50
  clflush size	: 64
  cache_alignment	: 64
  address sizes	: 46 bits physical, 48 bits virtual
  power management:

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1352732/+subscriptions
References

[Bug 1352732] [NEW] live migration fails:CPU feature `erms' specified more than once
From: marvel, 2014-08-05