← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #69828

[Bug 1737700] [NEW] live-migration error when no exchange 'scheduler_fanout' in vhost

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
Live-migration operation failed when _post_live_migration function executed on source compute node.
The failed reason is "no exchange 'scheduler_fanout' in vhost" when _update_scheduler_instance_info send message to nova-scheduler but nova-scheduler is restarting.

Steps to reproduce
==================
* I did live-migrate an instance.
* then I did "systemctl restart openstack-nova-scheduler".
* Instance's vm_state is error.

Expected result
===============
Instance's vm_state is active, this migration is successful.

Actual result
=============
Instance's vm_state is error

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

  Pike version

Logs & Configs
==============
2017-12-06 15:13:01.806 28207 WARNING nova.virt.libvirt.driver [req-dc0c38e4-cd57-496b-8d86-38b839af2ffb 2273fb5ce9e448d4a552665a230a18bb c6817e1696e84e69ab428b685b4d9fde - default default] [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] Error monitoring migration: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/': NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/'
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] Traceback (most recent call last):
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6811, in _live_migration
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     finish_event, disk_paths)
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6722, in _live_migration_monitor
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     migrate_data)
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     function_name, call_dict, binary)
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     self.force_reraise()
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     six.reraise(self.type_, self.value, self.tb)
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     return f(self, context, *args, **kw)
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 220, in decorated_function
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     kwargs['instance'], e, sys.exc_info())
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     self.force_reraise()
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     six.reraise(self.type_, self.value, self.tb)
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 208, in decorated_function
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     return function(self, context, *args, **kwargs)
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5822, in _post_live_migration
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     self._update_scheduler_instance_info(ctxt, instance)
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1650, in _update_scheduler_instance_info
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     instance)
2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]
233   instance-00000173              running

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  Description
  ===========
- Live-migration operation failed when _post_live_migration function executed on source compute node. 
+ Live-migration operation failed when _post_live_migration function executed on source compute node.
  The failed reason is "no exchange 'scheduler_fanout' in vhost" when _update_scheduler_instance_info send message to nova-scheduler but nova-scheduler is restarting.
-  
+ 
  Steps to reproduce
  ==================
- * I did live-migration an instance.
- * then I did "systemctl restart openstack-nova-scheduler"
- * Instance's vm_state is error
+ * I did live-migrate an instance.
+ * then I did "systemctl restart openstack-nova-scheduler".
+ * Instance's vm_state is error.
  
  Expected result
  ===============
  Instance's vm_state is active, this migration is successful.
  
  Actual result
  =============
  Instance's vm_state is error
  
  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
-   list for all releases: http://docs.openstack.org/releases/
+   list for all releases: http://docs.openstack.org/releases/
  
-   Pike version
+   Pike version
  
  Logs & Configs
  ==============
  2017-12-06 15:13:01.806 28207 WARNING nova.virt.libvirt.driver [req-dc0c38e4-cd57-496b-8d86-38b839af2ffb 2273fb5ce9e448d4a552665a230a18bb c6817e1696e84e69ab428b685b4d9fde - default default] [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] Error monitoring migration: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/': NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/'
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] Traceback (most recent call last):
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6811, in _live_migration
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     finish_event, disk_paths)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6722, in _live_migration_monitor
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     migrate_data)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     function_name, call_dict, binary)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     self.force_reraise()
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     six.reraise(self.type_, self.value, self.tb)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     return f(self, context, *args, **kw)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 220, in decorated_function
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     kwargs['instance'], e, sys.exc_info())
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     self.force_reraise()
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     six.reraise(self.type_, self.value, self.tb)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 208, in decorated_function
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     return function(self, context, *args, **kwargs)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5822, in _post_live_migration
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     self._update_scheduler_instance_info(ctxt, instance)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1650, in _update_scheduler_instance_info
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     instance)
- 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]  
+ 2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]
  233   instance-00000173              running

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1737700

Title:
  live-migration error when  no exchange 'scheduler_fanout' in vhost

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Live-migration operation failed when _post_live_migration function executed on source compute node.
  The failed reason is "no exchange 'scheduler_fanout' in vhost" when _update_scheduler_instance_info send message to nova-scheduler but nova-scheduler is restarting.

  Steps to reproduce
  ==================
  * I did live-migrate an instance.
  * then I did "systemctl restart openstack-nova-scheduler".
  * Instance's vm_state is error.

  Expected result
  ===============
  Instance's vm_state is active, this migration is successful.

  Actual result
  =============
  Instance's vm_state is error

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

    Pike version

  Logs & Configs
  ==============
  2017-12-06 15:13:01.806 28207 WARNING nova.virt.libvirt.driver [req-dc0c38e4-cd57-496b-8d86-38b839af2ffb 2273fb5ce9e448d4a552665a230a18bb c6817e1696e84e69ab428b685b4d9fde - default default] [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] Error monitoring migration: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/': NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'scheduler_fanout' in vhost '/'
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb] Traceback (most recent call last):
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6811, in _live_migration
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     finish_event, disk_paths)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6722, in _live_migration_monitor
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     migrate_data)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     function_name, call_dict, binary)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     self.force_reraise()
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     six.reraise(self.type_, self.value, self.tb)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     return f(self, context, *args, **kw)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 220, in decorated_function
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     kwargs['instance'], e, sys.exc_info())
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     self.force_reraise()
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     six.reraise(self.type_, self.value, self.tb)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 208, in decorated_function
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     return function(self, context, *args, **kwargs)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5822, in _post_live_migration
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     self._update_scheduler_instance_info(ctxt, instance)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1650, in _update_scheduler_instance_info
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]     instance)
  2017-12-06 15:13:01.806 28207 ERROR nova.virt.libvirt.driver [instance: 14a9e429-00b8-4093-93f4-dd9445be5dfb]
  233   instance-00000173              running

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1737700/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next