← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #74877

[Bug 1794302] [NEW] Live migration : service catalog is empty

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
When trying to live-migrate an instance, the instance start on the second node but then nova throw a "service catalog is empty".


Steps to reproduce
==================

* openstack server migrate 2a58b49e-08df-471b-9560-a9281621b2d1 --live
ruby.makz.me

Expected result
===============
The instance is migrated on the ruby.makz.me node

Actual result
=============
Nova start the instance on the second node for the live migration, nova controller throw a "Migration error: The service catalog is empty"

Environment
===========

Controller

openstack-nova-scheduler-18.0.0-1.el7.noarch
python-nova-18.0.0-1.el7.noarch
openstack-nova-console-18.0.0-1.el7.noarch
openstack-nova-common-18.0.0-1.el7.noarch
openstack-nova-conductor-18.0.0-1.el7.noarch
openstack-nova-spicehtml5proxy-18.0.0-1.el7.noarch
python2-novaclient-11.0.0-1.el7.noarch
openstack-nova-api-18.0.0-1.el7.noarch
openstack-nova-placement-api-18.0.0-1.el7.noarch
openstack-nova-novncproxy-18.0.0-1.el7.noarch

Compute node

openstack-nova-compute-18.0.0-1.el7.noarch
openstack-nova-common-18.0.0-1.el7.noarch
python-nova-18.0.0-1.el7.noarch
python2-novaclient-11.0.0-1.el7.noarch

2. Which hypervisor did you use?
Libvirt + KVM

libvirt-daemon-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-secret-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-mpath-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-network-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-logical-3.9.0-14.el7_5.7.x86_64
libvirt-libs-3.9.0-14.el7_5.7.x86_64
libvirt-client-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-core-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-iscsi-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-nwfilter-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-rbd-3.9.0-14.el7_5.7.x86_64
libvirt-python-3.9.0-1.el7.x86_64
libvirt-daemon-driver-storage-disk-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-interface-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-scsi-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-kvm-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-nodedev-3.9.0-14.el7_5.7.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.7.x86_64

qemu-kvm-ev-2.10.0-21.el7_5.4.1.x86_64
qemu-img-ev-2.10.0-21.el7_5.4.1.x86_64
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.7.x86_64
qemu-kvm-common-ev-2.10.0-21.el7_5.4.1.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
centos-release-qemu-ev-1.0-3.el7.centos.noarch


2. Which storage type did you use?

Cinder + LVM

python-cinder-13.0.0-1.el7.noarch
openstack-cinder-13.0.0-1.el7.noarch
python2-cinderclient-4.0.1-1.el7.noarch

3. Which networking type did you use?

openstack-neutron-13.0.1-1.el7.noarch
openstack-neutron-common-13.0.1-1.el7.noarch
openstack-neutron-ml2-13.0.1-1.el7.noarch
python2-neutron-lib-1.18.0-1.el7.noarch
python2-neutronclient-6.9.0-1.el7.noarch
openstack-neutron-linuxbridge-13.0.1-1.el7.noarch
python-neutron-13.0.1-1.el7.noarch
python2-openvswitch-2.9.0-3.el7.noarch

Logs & Configs
==============

Controller
==========

==> /var/log/neutron/server.log <==
2018-09-25 15:48:40.734 6677 INFO neutron.wsgi [req-b5ffce01-5f45-4099-9f4e-dca8d66bd04f 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET /v2.0/ports?device_id=2a58b49e-08df-471b-9560-a9281621b2d1 HTTP/1.1" status: 200  len: 1189 time: 0.5954452
2018-09-25 15:48:40.830 6677 INFO neutron.wsgi [req-d524e8ea-7201-4f31-a774-dd9b88e3cd1f 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET /v2.0/security-groups?id=2b2618f8-8c86-4ff0-8beb-52a9e8e5edad&id=1f47e4f6-62d5-4e91-bb91-f27c525bd0be&id=4e0d94e2-e915-41ea-8103-636af0d5c82f&id=9cfc779e-dd64-44f1-85b2-a9c6c852cfba HTTP/1.1" status: 200  len: 11665 time: 0.0915859

==> /var/log/nova/nova-api.log <==
2018-09-25 15:48:40.837 24668 INFO nova.osapi_compute.wsgi.server [req-c2c432cc-4911-4f8e-a8fc-9b2a732ed93c 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET /v2.1/servers/2a58b49e-08df-471b-9560-a9281621b2d1 HTTP/1.1" status: 200 len: 2108 time: 1.3573151

==> /var/log/nova/nova-placement-api.log <==
2018-09-25 15:48:42.552 8830 INFO nova.api.openstack.placement.requestlog [req-a2384cab-5e52-4c99-a4db-56c96475f4ee 0412e7701b674098b81afc2424781afb a03be5acfab64945a9259b798130d1eb - default default] 192.168.58.1 "GET /allocations/2a58b49e-08df-471b-9560-a9281621b2d1" status: 200 len: 245 microversion: 1.28\x1b[00m
2018-09-25 15:48:42.746 8831 INFO nova.api.openstack.placement.requestlog [req-95abd683-8e2f-4955-8c2d-25b6c88db49b 0412e7701b674098b81afc2424781afb a03be5acfab64945a9259b798130d1eb - default default] 192.168.58.1 "POST /allocations" status: 204 len: 0 microversion: 1.13\x1b[00m

==> /var/log/nova/nova-conductor.log <==
2018-09-25 15:48:43.132 24594 WARNING oslo_config.cfg [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] Deprecated: Option "url" from group "neutron" is deprecated for removal (Endpoint lookup uses the service catalog via common keystoneauth1 Adapter configuration options. In the current release, "url" will override this behavior, but will be ignored and/or removed in a future release. To achieve the same result, use the endpoint_override option instead.).  Its value may be silently ignored in the future.

==> /var/log/neutron/server.log <==
2018-09-25 15:48:43.151 6674 INFO neutron.wsgi [req-0751549f-ce04-4fd5-a590-e0fcbf4fc39d 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET /v2.0/extensions HTTP/1.1" status: 200  len: 9702 time: 0.0112290

==> /var/log/nova/nova-placement-api.log <==
2018-09-25 15:48:45.234 8829 INFO nova.api.openstack.placement.requestlog [req-b61f462c-1f34-4855-aa4c-83042f503897 0412e7701b674098b81afc2424781afb a03be5acfab64945a9259b798130d1eb - default default] 192.168.58.1 "POST /allocations" status: 204 len: 0 microversion: 1.13\x1b[00m

==> /var/log/nova/nova-conductor.log <==
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] Migration of instance 2a58b49e-08df-471b-9560-a9281621b2d1 to host ruby.makz.me unexpectedly failed.: EmptyCatalog: The service catalog is empty.
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager Traceback (most recent call last):
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 424, in _live_migrate
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     task.execute()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 27, in wrap
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     self.rollback()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     self.force_reraise()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     six.reraise(self.type_, self.value, self.tb)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 24, in wrap
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     return original(self)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 42, in execute
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     return self._execute()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 97, in _execute
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     source_node, dest_node = self._check_requested_destination()
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 171, in _check_requested_destination
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     self._call_livem_checks_on_host(self.destination)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 252, in _call_livem_checks_on_host
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     self._bind_ports_on_destination(destination))
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 263, in _bind_ports_on_destination
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     self.context, self.instance, destination)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1277, in bind_ports_to_host
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     json=data, raise_exc=False)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     return self.request(url, 'POST', **kwargs)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 213, in request
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     return self.session.request(url, method, **kwargs)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 706, in request
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     **endpoint_filter)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     return auth.get_endpoint(self, **kwargs)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     allow_version_hack=allow_version_hack, **kwargs)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 279, in get_endpoint_data
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     service_name=service_name)
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 425, in endpoint_data_for
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     raise exceptions.EmptyCatalog('The service catalog is empty.')
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager EmptyCatalog: The service catalog is empty.
2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager 
2018-09-25 15:48:45.269 24594 WARNING nova.scheduler.utils [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] Failed to compute_task_migrate_server: The service catalog is empty.: EmptyCatalog: The service catalog is empty.
2018-09-25 15:48:45.270 24594 WARNING nova.scheduler.utils [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] [instance: 2a58b49e-08df-471b-9560-a9281621b2d1] Setting instance to ERROR state.: EmptyCatalog: The service catalog is empty.
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] Exception during message handling: MigrationError: Migration error: The service catalog is empty.
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 86, in wrapper
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     return fn(self, context, *args, **kwargs)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in decorated_function
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 281, in migrate_server
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     block_migration, disk_over_commit, request_spec)
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 454, in _live_migrate
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     raise exception.MigrationError(reason=six.text_type(ex))
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server MigrationError: Migration error: The service catalog is empty.
2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server 

==> /var/log/nova/nova-api.log <==
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] Unexpected exception in API method: MigrationError_Remote: Migration error: The service catalog is empty.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
    res = self.dispatcher.dispatch(message)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
    result = func(ctxt, **new_args)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 86, in wrapper
    return fn(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 281, in migrate_server
    block_migration, disk_over_commit, request_spec)

  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 454, in _live_migrate
    raise exception.MigrationError(reason=six.text_type(ex))

MigrationError: Migration error: The service catalog is empty.
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/migrate_server.py", line 107, in _migrate_live
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     async_)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 205, in inner
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return function(self, context, instance, *args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 213, in _wrapped
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return fn(self, context, instance, *args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 153, in inner
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 4396, in live_migrate
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     request_spec=request_spec, async_=async_)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 112, in live_migrate_instance
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     request_spec=request_spec)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 338, in migrate_server
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return cctxt.call(context, 'migrate_server', **kw)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     retry=self.retry)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     retry=retry)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     call_monitor_timeout, retry=retry)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     raise result
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi MigrationError_Remote: Migration error: The service catalog is empty.
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     res = self.dispatcher.dispatch(message)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return self._do_dispatch(endpoint, method, ctxt, args)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     result = func(ctxt, **new_args)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 86, in wrapper
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return fn(self, context, *args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in decorated_function
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return function(self, context, *args, **kwargs)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 281, in migrate_server
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     block_migration, disk_over_commit, request_spec)
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 454, in _live_migrate
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     raise exception.MigrationError(reason=six.text_type(ex))
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi MigrationError: Migration error: The service catalog is empty.
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
2018-09-25 15:48:45.481 24668 INFO nova.api.openstack.wsgi [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'MigrationError_Remote'>
2018-09-25 15:48:45.483 24668 INFO nova.osapi_compute.wsgi.server [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "POST /v2.1/servers/2a58b49e-08df-471b-9560-a9281621b2d1/action HTTP/1.1" status: 500 len: 628 time: 4.6382320


Compute node
============

2018-09-25 15:48:42.961 1741 INFO nova.virt.libvirt.driver [req-
d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
f2b81bf3d8ec40c191ae06bc2682e606 - default default] Instance launched
has CPU info: {"vendor": "Intel", "model": "Westmere-IBRS", "arch":
"x86_64", "features": ["pge", "clflush", "sep", "syscall", "vme",
"dtes64", "stibp", "msr", "vmx", "xtpr", "cmov", "ssse3", "est", "pat",
"monitor", "smx", "pbe", "lm", "tsc", "nx", "fxsr", "tm", "sse4.1",
"pae", "sse4.2", "pclmuldq", "cx16", "pcid", "mmx", "arat", "cx8",
"mce", "de", "rdtscp", "ht", "dca", "lahf_lm", "pdcm", "mca", "pdpe1gb",
"apic", "sse", "pse", "ds", "invtsc", "pni", "tm2", "aes", "sse2", "ss",
"ds_cpl", "acpi", "spec-ctrl", "fpu", "ssbd", "pse36", "mtrr",
"popcnt"], "topology": {"cores": 6, "cells": 2, "threads": 2, "sockets":
1}}

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1794302

Title:
  Live migration : service catalog is empty

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  When trying to live-migrate an instance, the instance start on the second node but then nova throw a "service catalog is empty".

  
  Steps to reproduce
  ==================

  * openstack server migrate 2a58b49e-08df-471b-9560-a9281621b2d1 --live
  ruby.makz.me

  Expected result
  ===============
  The instance is migrated on the ruby.makz.me node

  Actual result
  =============
  Nova start the instance on the second node for the live migration, nova controller throw a "Migration error: The service catalog is empty"

  Environment
  ===========

  Controller

  openstack-nova-scheduler-18.0.0-1.el7.noarch
  python-nova-18.0.0-1.el7.noarch
  openstack-nova-console-18.0.0-1.el7.noarch
  openstack-nova-common-18.0.0-1.el7.noarch
  openstack-nova-conductor-18.0.0-1.el7.noarch
  openstack-nova-spicehtml5proxy-18.0.0-1.el7.noarch
  python2-novaclient-11.0.0-1.el7.noarch
  openstack-nova-api-18.0.0-1.el7.noarch
  openstack-nova-placement-api-18.0.0-1.el7.noarch
  openstack-nova-novncproxy-18.0.0-1.el7.noarch

  Compute node

  openstack-nova-compute-18.0.0-1.el7.noarch
  openstack-nova-common-18.0.0-1.el7.noarch
  python-nova-18.0.0-1.el7.noarch
  python2-novaclient-11.0.0-1.el7.noarch

  2. Which hypervisor did you use?
  Libvirt + KVM

  libvirt-daemon-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-secret-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-storage-mpath-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-network-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-storage-logical-3.9.0-14.el7_5.7.x86_64
  libvirt-libs-3.9.0-14.el7_5.7.x86_64
  libvirt-client-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-storage-core-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-storage-iscsi-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-nwfilter-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-qemu-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-storage-rbd-3.9.0-14.el7_5.7.x86_64
  libvirt-python-3.9.0-1.el7.x86_64
  libvirt-daemon-driver-storage-disk-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-interface-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-storage-scsi-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-storage-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-kvm-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-nodedev-3.9.0-14.el7_5.7.x86_64
  libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.7.x86_64

  qemu-kvm-ev-2.10.0-21.el7_5.4.1.x86_64
  qemu-img-ev-2.10.0-21.el7_5.4.1.x86_64
  libvirt-daemon-driver-qemu-3.9.0-14.el7_5.7.x86_64
  qemu-kvm-common-ev-2.10.0-21.el7_5.4.1.x86_64
  ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
  centos-release-qemu-ev-1.0-3.el7.centos.noarch

  
  2. Which storage type did you use?

  Cinder + LVM

  python-cinder-13.0.0-1.el7.noarch
  openstack-cinder-13.0.0-1.el7.noarch
  python2-cinderclient-4.0.1-1.el7.noarch

  3. Which networking type did you use?

  openstack-neutron-13.0.1-1.el7.noarch
  openstack-neutron-common-13.0.1-1.el7.noarch
  openstack-neutron-ml2-13.0.1-1.el7.noarch
  python2-neutron-lib-1.18.0-1.el7.noarch
  python2-neutronclient-6.9.0-1.el7.noarch
  openstack-neutron-linuxbridge-13.0.1-1.el7.noarch
  python-neutron-13.0.1-1.el7.noarch
  python2-openvswitch-2.9.0-3.el7.noarch

  Logs & Configs
  ==============

  Controller
  ==========

  ==> /var/log/neutron/server.log <==
  2018-09-25 15:48:40.734 6677 INFO neutron.wsgi [req-b5ffce01-5f45-4099-9f4e-dca8d66bd04f 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET /v2.0/ports?device_id=2a58b49e-08df-471b-9560-a9281621b2d1 HTTP/1.1" status: 200  len: 1189 time: 0.5954452
  2018-09-25 15:48:40.830 6677 INFO neutron.wsgi [req-d524e8ea-7201-4f31-a774-dd9b88e3cd1f 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET /v2.0/security-groups?id=2b2618f8-8c86-4ff0-8beb-52a9e8e5edad&id=1f47e4f6-62d5-4e91-bb91-f27c525bd0be&id=4e0d94e2-e915-41ea-8103-636af0d5c82f&id=9cfc779e-dd64-44f1-85b2-a9c6c852cfba HTTP/1.1" status: 200  len: 11665 time: 0.0915859

  ==> /var/log/nova/nova-api.log <==
  2018-09-25 15:48:40.837 24668 INFO nova.osapi_compute.wsgi.server [req-c2c432cc-4911-4f8e-a8fc-9b2a732ed93c 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET /v2.1/servers/2a58b49e-08df-471b-9560-a9281621b2d1 HTTP/1.1" status: 200 len: 2108 time: 1.3573151

  ==> /var/log/nova/nova-placement-api.log <==
  2018-09-25 15:48:42.552 8830 INFO nova.api.openstack.placement.requestlog [req-a2384cab-5e52-4c99-a4db-56c96475f4ee 0412e7701b674098b81afc2424781afb a03be5acfab64945a9259b798130d1eb - default default] 192.168.58.1 "GET /allocations/2a58b49e-08df-471b-9560-a9281621b2d1" status: 200 len: 245 microversion: 1.28\x1b[00m
  2018-09-25 15:48:42.746 8831 INFO nova.api.openstack.placement.requestlog [req-95abd683-8e2f-4955-8c2d-25b6c88db49b 0412e7701b674098b81afc2424781afb a03be5acfab64945a9259b798130d1eb - default default] 192.168.58.1 "POST /allocations" status: 204 len: 0 microversion: 1.13\x1b[00m

  ==> /var/log/nova/nova-conductor.log <==
  2018-09-25 15:48:43.132 24594 WARNING oslo_config.cfg [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] Deprecated: Option "url" from group "neutron" is deprecated for removal (Endpoint lookup uses the service catalog via common keystoneauth1 Adapter configuration options. In the current release, "url" will override this behavior, but will be ignored and/or removed in a future release. To achieve the same result, use the endpoint_override option instead.).  Its value may be silently ignored in the future.

  ==> /var/log/neutron/server.log <==
  2018-09-25 15:48:43.151 6674 INFO neutron.wsgi [req-0751549f-ce04-4fd5-a590-e0fcbf4fc39d 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "GET /v2.0/extensions HTTP/1.1" status: 200  len: 9702 time: 0.0112290

  ==> /var/log/nova/nova-placement-api.log <==
  2018-09-25 15:48:45.234 8829 INFO nova.api.openstack.placement.requestlog [req-b61f462c-1f34-4855-aa4c-83042f503897 0412e7701b674098b81afc2424781afb a03be5acfab64945a9259b798130d1eb - default default] 192.168.58.1 "POST /allocations" status: 204 len: 0 microversion: 1.13\x1b[00m

  ==> /var/log/nova/nova-conductor.log <==
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] Migration of instance 2a58b49e-08df-471b-9560-a9281621b2d1 to host ruby.makz.me unexpectedly failed.: EmptyCatalog: The service catalog is empty.
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager Traceback (most recent call last):
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 424, in _live_migrate
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     task.execute()
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 27, in wrap
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     self.rollback()
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     self.force_reraise()
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     six.reraise(self.type_, self.value, self.tb)
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 24, in wrap
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     return original(self)
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 42, in execute
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     return self._execute()
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 97, in _execute
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     source_node, dest_node = self._check_requested_destination()
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 171, in _check_requested_destination
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     self._call_livem_checks_on_host(self.destination)
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 252, in _call_livem_checks_on_host
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     self._bind_ports_on_destination(destination))
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 263, in _bind_ports_on_destination
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     self.context, self.instance, destination)
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1277, in bind_ports_to_host
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     json=data, raise_exc=False)
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     return self.request(url, 'POST', **kwargs)
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 213, in request
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     return self.session.request(url, method, **kwargs)
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 706, in request
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     **endpoint_filter)
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     return auth.get_endpoint(self, **kwargs)
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     allow_version_hack=allow_version_hack, **kwargs)
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 279, in get_endpoint_data
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     service_name=service_name)
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager   File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 425, in endpoint_data_for
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager     raise exceptions.EmptyCatalog('The service catalog is empty.')
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager EmptyCatalog: The service catalog is empty.
  2018-09-25 15:48:45.237 24594 ERROR nova.conductor.manager 
  2018-09-25 15:48:45.269 24594 WARNING nova.scheduler.utils [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] Failed to compute_task_migrate_server: The service catalog is empty.: EmptyCatalog: The service catalog is empty.
  2018-09-25 15:48:45.270 24594 WARNING nova.scheduler.utils [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] [instance: 2a58b49e-08df-471b-9560-a9281621b2d1] Setting instance to ERROR state.: EmptyCatalog: The service catalog is empty.
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] Exception during message handling: MigrationError: Migration error: The service catalog is empty.
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 86, in wrapper
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     return fn(self, context, *args, **kwargs)
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in decorated_function
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 281, in migrate_server
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     block_migration, disk_over_commit, request_spec)
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 454, in _live_migrate
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server     raise exception.MigrationError(reason=six.text_type(ex))
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server MigrationError: Migration error: The service catalog is empty.
  2018-09-25 15:48:45.424 24594 ERROR oslo_messaging.rpc.server 

  ==> /var/log/nova/nova-api.log <==
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] Unexpected exception in API method: MigrationError_Remote: Migration error: The service catalog is empty.
  Traceback (most recent call last):

    File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
      res = self.dispatcher.dispatch(message)

    File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)

    File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
      result = func(ctxt, **new_args)

    File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
      return func(*args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 86, in wrapper
      return fn(self, context, *args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in decorated_function
      return function(self, context, *args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 281, in migrate_server
      block_migration, disk_over_commit, request_spec)

    File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 454, in _live_migrate
      raise exception.MigrationError(reason=six.text_type(ex))

  MigrationError: Migration error: The service catalog is empty.
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/migrate_server.py", line 107, in _migrate_live
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     async_)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 205, in inner
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return function(self, context, instance, *args, **kwargs)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 213, in _wrapped
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return fn(self, context, instance, *args, **kwargs)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 153, in inner
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 4396, in live_migrate
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     request_spec=request_spec, async_=async_)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 112, in live_migrate_instance
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     request_spec=request_spec)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 338, in migrate_server
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return cctxt.call(context, 'migrate_server', **kw)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     retry=self.retry)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     retry=retry)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     call_monitor_timeout, retry=retry)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     raise result
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi MigrationError_Remote: Migration error: The service catalog is empty.
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     res = self.dispatcher.dispatch(message)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return self._do_dispatch(endpoint, method, ctxt, args)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     result = func(ctxt, **new_args)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 86, in wrapper
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return fn(self, context, *args, **kwargs)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in decorated_function
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     return function(self, context, *args, **kwargs)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 281, in migrate_server
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     block_migration, disk_over_commit, request_spec)
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 454, in _live_migrate
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi     raise exception.MigrationError(reason=six.text_type(ex))
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi MigrationError: Migration error: The service catalog is empty.
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
  2018-09-25 15:48:45.446 24668 ERROR nova.api.openstack.wsgi 
  2018-09-25 15:48:45.481 24668 INFO nova.api.openstack.wsgi [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'MigrationError_Remote'>
  2018-09-25 15:48:45.483 24668 INFO nova.osapi_compute.wsgi.server [req-d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b f2b81bf3d8ec40c191ae06bc2682e606 - default default] 192.168.58.1 "POST /v2.1/servers/2a58b49e-08df-471b-9560-a9281621b2d1/action HTTP/1.1" status: 500 len: 628 time: 4.6382320

  
  Compute node
  ============

  2018-09-25 15:48:42.961 1741 INFO nova.virt.libvirt.driver [req-
  d26db4fb-79f2-47c3-b779-4c49e76791e7 3829056496a04d929af44e29efc7568b
  f2b81bf3d8ec40c191ae06bc2682e606 - default default] Instance launched
  has CPU info: {"vendor": "Intel", "model": "Westmere-IBRS", "arch":
  "x86_64", "features": ["pge", "clflush", "sep", "syscall", "vme",
  "dtes64", "stibp", "msr", "vmx", "xtpr", "cmov", "ssse3", "est",
  "pat", "monitor", "smx", "pbe", "lm", "tsc", "nx", "fxsr", "tm",
  "sse4.1", "pae", "sse4.2", "pclmuldq", "cx16", "pcid", "mmx", "arat",
  "cx8", "mce", "de", "rdtscp", "ht", "dca", "lahf_lm", "pdcm", "mca",
  "pdpe1gb", "apic", "sse", "pse", "ds", "invtsc", "pni", "tm2", "aes",
  "sse2", "ss", "ds_cpl", "acpi", "spec-ctrl", "fpu", "ssbd", "pse36",
  "mtrr", "popcnt"], "topology": {"cores": 6, "cells": 2, "threads": 2,
  "sockets": 1}}

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1794302/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next