yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #64279
[Bug 1693721] [NEW] Error handling for HostMappingNotFound in conductor manager does not work properly
Public bug reported:
Description
===========
When creating a server,
if a compute node doesn't belong any cell and nova-scheduler selects the host,
creating a server fails (HostMappingNotFound) and a server status become stuck in 'BUILD'.
The server cannot be deleted by "Delete server" API.
This issue can be avoided by executing the following command before
creating a server.
nova-manage cell_v2 map_cell_and_hosts
But an error handling(*1) for HostMappingNotFound exists.
So it should be executed successfully.
*1:
https://github.com/openstack/nova/blob/8ccf47cf37b7181431f3de0af1f39b7c18f4a6ac/nova/conductor/manager.py#L943-L945
Steps to reproduce
==================
1. start nova processes (except nova-compute) and
other components processes (galnce, neutorn, keystone, cinder, etc)
in devstack
2. start nova-compute process
3. create a server (POST /servers)
Expected result
===============
The status of the server becomes ERROR.
The server can be deleted.
Actual result
=============
The server becomes stuck in 'BUILD' status.
The server cannot be deleted.
Environment
===========
nova:
master (commit 8ccf47cf37b7181431f3de0af1f39b7c18f4a6ac)
devstack:
master (commit 9b2a2fa55dbba724a781d2720546611a8add893)
OS:
Ubuntu 16.04.2 LTS
Logs & Configs
==============
stack@devstack-master:/opt/devstack$ nova boot --flavor 1 --image 91bd1f63-77ff-41d9-abe3-367163ef9673 server1
+--------------------------------------+-----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | server1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-8lxtcp9q |
| OS-EXT-SRV-ATTR:root_device_name | - |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | V3FLw3BDJUQu |
| config_drive | |
| created | 2017-05-26T07:35:12Z |
| description | - |
| flavor | m1.tiny (1) |
| hostId | |
| host_status | |
| id | 421bb801-005a-4f73-9fe1-0916fd691b2b |
| image | cirros-0.3.5-x86_64-disk (91bd1f63-77ff-41d9-abe3-367163ef9673) |
| key_name | - |
| locked | False |
| metadata | {} |
| name | server1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tags | [] |
| tenant_id | 56eb461553af4c28a09934bd3f1d6769 |
| updated | 2017-05-26T07:35:12Z |
| user_id | c001d3de4c1d49e18d3b23faf1bcebc2 |
+--------------------------------------+-----------------------------------------------------------------+
stack@devstack-master:/opt/devstack$ nova list
+--------------------------------------+---------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+----------+
| 421bb801-005a-4f73-9fe1-0916fd691b2b | server1 | BUILD | scheduling | NOSTATE | |
+--------------------------------------+---------+--------+------------+-------------+----------+
stack@devstack-master:/opt/devstack$ nova delete server1
Request to delete server server1 has been accepted.
stack@devstack-master:/opt/devstack$ nova list
+--------------------------------------+---------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+----------+
| 421bb801-005a-4f73-9fe1-0916fd691b2b | server1 | BUILD | scheduling | NOSTATE | |
+--------------------------------------+---------+--------+------------+-------------+----------+
nova-conductor log:
****************************************************** {{(pid=9578) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2797}}
5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-f440d128-3ad2-4362-b06e-3b8d1924b82a None None] Creating RPC server for service conductor {{(pid=9955) start /opt/stack/nova/nova/service.py:166}}
5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-0bb407a0-bb66-4306-a69b-f7a41918608c None None] Creating RPC server for service conductor {{(pid=9954) start /opt/stack/nova/nova/service.py:166}}
5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-f440d128-3ad2-4362-b06e-3b8d1924b82a None None] Join ServiceGroup membership for this service conductor {{(pid=9955) start /opt/stack/nova/nova/service.py:184}}
5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.servicegroup.drivers.db [req-f440d128-3ad2-4362-b06e-3b8d1924b82a None None] DB_Driver: join new ServiceGroup member devstack-master to the conductor group, service = <Service: host=devstack-master, binary=nova-conductor, manager_class_name=nova.conductor.manager.ConductorManager> {{(pid=9955) join /opt/stack/nova/nova/servicegroup/drivers/db.py:47}}
5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-0bb407a0-bb66-4306-a69b-f7a41918608c None None] Join ServiceGroup membership for this service conductor {{(pid=9954) start /opt/stack/nova/nova/service.py:184}}
5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.servicegroup.drivers.db [req-0bb407a0-bb66-4306-a69b-f7a41918608c None None] DB_Driver: join new ServiceGroup member devstack-master to the conductor group, service = <Service: host=devstack-master, binary=nova-conductor, manager_class_name=nova.conductor.manager.ConductorManager> {{(pid=9954) join /opt/stack/nova/nova/servicegroup/drivers/db.py:47}}
5月 26 16:35:12 devstack-master nova-conductor[9578]: WARNING oslo_config.cfg [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Option "scheduler_default_filters" from group "DEFAULT" is deprecated. Use option "enabled_filters" from group "filter_scheduler".
5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_db.sqlalchemy.engines [req-40896def-378f-4ba6-895c-023bab013faf admin admin] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=9955) _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:260}}
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Failed to schedule instances
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager Traceback (most recent call last):
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/conductor/manager.py", line 916, in schedule_and_build_instances
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/conductor/manager.py", line 637, in _schedule_instances
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/utils.py", line 371, in wrapped
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return func(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/client/__init__.py", line 51, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/client/__init__.py", line 37, in __run_method
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/client/query.py", line 32, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/rpcapi.py", line 130, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager retry=self.retry)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager timeout=timeout, retry=retry)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager retry=retry)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 496, in _send
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager raise result
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager HostMappingNotFound_Remote: Host 'devstack-master' is not mapped to any cell
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager Traceback (most recent call last):
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager res = self.dispatcher.dispatch(message)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self._do_dispatch(endpoint, method, ctxt, args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager result = func(ctxt, **new_args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 229, in inner
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return func(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/manager.py", line 98, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 57, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager selected_hosts = self._schedule(context, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 102, in _schedule
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager hosts = self._get_all_host_states(elevated, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 180, in _get_all_host_states
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 646, in get_host_states_by_uuids
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self._get_host_states(context, compute_nodes, services)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 690, in _get_host_states
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager self._get_instance_info(context, compute))
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 730, in _get_instance_info
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager inst_dict = self._get_instances_by_host(context, host_name)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 709, in _get_instances_by_host
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager hm = objects.HostMapping.get_by_host(context, host_name)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager result = fn(cls, context, *args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/objects/host_mapping.py", line 100, in get_by_host
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager db_mapping = cls._get_by_host_from_db(context, host)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 979, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return fn(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/objects/host_mapping.py", line 95, in _get_by_host_from_db
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager raise exception.HostMappingNotFound(name=host)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager HostMappingNotFound: Host 'devstack-master' is not mapped to any cell
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_concurrency.lockutils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=9955) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:270}}
5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_concurrency.lockutils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=9955) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282}}
5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_db.sqlalchemy.engines [req-40896def-378f-4ba6-895c-023bab013faf admin admin] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=9955) _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:260}}
5月 26 16:35:12 devstack-master nova-conductor[9578]: WARNING nova.scheduler.utils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Failed to compute_task_build_instances: Host 'devstack-master' is not mapped to any cell
5月 26 16:35:12 devstack-master nova-conductor[9578]: Traceback (most recent call last):
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
5月 26 16:35:12 devstack-master nova-conductor[9578]: res = self.dispatcher.dispatch(message)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
5月 26 16:35:12 devstack-master nova-conductor[9578]: return self._do_dispatch(endpoint, method, ctxt, args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
5月 26 16:35:12 devstack-master nova-conductor[9578]: result = func(ctxt, **new_args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 229, in inner
5月 26 16:35:12 devstack-master nova-conductor[9578]: return func(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/manager.py", line 98, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: dests = self.driver.select_destinations(ctxt, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 57, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: selected_hosts = self._schedule(context, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 102, in _schedule
5月 26 16:35:12 devstack-master nova-conductor[9578]: hosts = self._get_all_host_states(elevated, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 180, in _get_all_host_states
5月 26 16:35:12 devstack-master nova-conductor[9578]: spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 646, in get_host_states_by_uuids
5月 26 16:35:12 devstack-master nova-conductor[9578]: return self._get_host_states(context, compute_nodes, services)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 690, in _get_host_states
5月 26 16:35:12 devstack-master nova-conductor[9578]: self._get_instance_info(context, compute))
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 730, in _get_instance_info
5月 26 16:35:12 devstack-master nova-conductor[9578]: inst_dict = self._get_instances_by_host(context, host_name)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 709, in _get_instances_by_host
5月 26 16:35:12 devstack-master nova-conductor[9578]: hm = objects.HostMapping.get_by_host(context, host_name)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: result = fn(cls, context, *args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/objects/host_mapping.py", line 100, in get_by_host
5月 26 16:35:12 devstack-master nova-conductor[9578]: db_mapping = cls._get_by_host_from_db(context, host)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 979, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: return fn(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/objects/host_mapping.py", line 95, in _get_by_host_from_db
5月 26 16:35:12 devstack-master nova-conductor[9578]: raise exception.HostMappingNotFound(name=host)
5月 26 16:35:12 devstack-master nova-conductor[9578]: HostMappingNotFound: Host 'devstack-master' is not mapped to any cell
5月 26 16:35:12 devstack-master nova-conductor[9578]:
5月 26 16:35:12 devstack-master nova-conductor[9578]: WARNING nova.scheduler.utils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] [instance: 421bb801-005a-4f73-9fe1-0916fd691b2b] Setting instance to ERROR state.
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Exception during message handling
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 920, in schedule_and_build_instances
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server build_requests=build_requests)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 890, in _bury_in_cell0
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server exc, legacy_spec)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 355, in _set_vm_state_and_notify
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server ex, request_spec)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/utils.py", line 103, in set_vm_state_and_notify
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server instance.save()
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return fn(self, *args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/objects/instance.py", line 781, in save
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server columns_to_join=_expected_cols(expected_attrs))
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/api.py", line 860, in instance_update_and_get_original
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server expected=expected)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server self.force_reraise()
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 251, in wrapped
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return f(context, *args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2673, in instance_update_and_get_original
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server columns_to_join=columns_to_join)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1929, in _instance_get_by_uuid
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server raise exception.InstanceNotFound(instance_id=uuid)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server InstanceNotFound: Instance 421bb801-005a-4f73-9fe1-0916fd691b2b could not be found.
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server
** Affects: nova
Importance: Undecided
Assignee: Takashi NATSUME (natsume-takashi)
Status: New
** Tags: cells conductor
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1693721
Title:
Error handling for HostMappingNotFound in conductor manager does not
work properly
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
When creating a server,
if a compute node doesn't belong any cell and nova-scheduler selects the host,
creating a server fails (HostMappingNotFound) and a server status become stuck in 'BUILD'.
The server cannot be deleted by "Delete server" API.
This issue can be avoided by executing the following command before
creating a server.
nova-manage cell_v2 map_cell_and_hosts
But an error handling(*1) for HostMappingNotFound exists.
So it should be executed successfully.
*1:
https://github.com/openstack/nova/blob/8ccf47cf37b7181431f3de0af1f39b7c18f4a6ac/nova/conductor/manager.py#L943-L945
Steps to reproduce
==================
1. start nova processes (except nova-compute) and
other components processes (galnce, neutorn, keystone, cinder, etc)
in devstack
2. start nova-compute process
3. create a server (POST /servers)
Expected result
===============
The status of the server becomes ERROR.
The server can be deleted.
Actual result
=============
The server becomes stuck in 'BUILD' status.
The server cannot be deleted.
Environment
===========
nova:
master (commit 8ccf47cf37b7181431f3de0af1f39b7c18f4a6ac)
devstack:
master (commit 9b2a2fa55dbba724a781d2720546611a8add893)
OS:
Ubuntu 16.04.2 LTS
Logs & Configs
==============
stack@devstack-master:/opt/devstack$ nova boot --flavor 1 --image 91bd1f63-77ff-41d9-abe3-367163ef9673 server1
+--------------------------------------+-----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | server1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-8lxtcp9q |
| OS-EXT-SRV-ATTR:root_device_name | - |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | V3FLw3BDJUQu |
| config_drive | |
| created | 2017-05-26T07:35:12Z |
| description | - |
| flavor | m1.tiny (1) |
| hostId | |
| host_status | |
| id | 421bb801-005a-4f73-9fe1-0916fd691b2b |
| image | cirros-0.3.5-x86_64-disk (91bd1f63-77ff-41d9-abe3-367163ef9673) |
| key_name | - |
| locked | False |
| metadata | {} |
| name | server1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tags | [] |
| tenant_id | 56eb461553af4c28a09934bd3f1d6769 |
| updated | 2017-05-26T07:35:12Z |
| user_id | c001d3de4c1d49e18d3b23faf1bcebc2 |
+--------------------------------------+-----------------------------------------------------------------+
stack@devstack-master:/opt/devstack$ nova list
+--------------------------------------+---------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+----------+
| 421bb801-005a-4f73-9fe1-0916fd691b2b | server1 | BUILD | scheduling | NOSTATE | |
+--------------------------------------+---------+--------+------------+-------------+----------+
stack@devstack-master:/opt/devstack$ nova delete server1
Request to delete server server1 has been accepted.
stack@devstack-master:/opt/devstack$ nova list
+--------------------------------------+---------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+----------+
| 421bb801-005a-4f73-9fe1-0916fd691b2b | server1 | BUILD | scheduling | NOSTATE | |
+--------------------------------------+---------+--------+------------+-------------+----------+
nova-conductor log:
****************************************************** {{(pid=9578) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2797}}
5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-f440d128-3ad2-4362-b06e-3b8d1924b82a None None] Creating RPC server for service conductor {{(pid=9955) start /opt/stack/nova/nova/service.py:166}}
5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-0bb407a0-bb66-4306-a69b-f7a41918608c None None] Creating RPC server for service conductor {{(pid=9954) start /opt/stack/nova/nova/service.py:166}}
5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-f440d128-3ad2-4362-b06e-3b8d1924b82a None None] Join ServiceGroup membership for this service conductor {{(pid=9955) start /opt/stack/nova/nova/service.py:184}}
5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.servicegroup.drivers.db [req-f440d128-3ad2-4362-b06e-3b8d1924b82a None None] DB_Driver: join new ServiceGroup member devstack-master to the conductor group, service = <Service: host=devstack-master, binary=nova-conductor, manager_class_name=nova.conductor.manager.ConductorManager> {{(pid=9955) join /opt/stack/nova/nova/servicegroup/drivers/db.py:47}}
5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.service [req-0bb407a0-bb66-4306-a69b-f7a41918608c None None] Join ServiceGroup membership for this service conductor {{(pid=9954) start /opt/stack/nova/nova/service.py:184}}
5月 26 16:21:23 devstack-master nova-conductor[9578]: DEBUG nova.servicegroup.drivers.db [req-0bb407a0-bb66-4306-a69b-f7a41918608c None None] DB_Driver: join new ServiceGroup member devstack-master to the conductor group, service = <Service: host=devstack-master, binary=nova-conductor, manager_class_name=nova.conductor.manager.ConductorManager> {{(pid=9954) join /opt/stack/nova/nova/servicegroup/drivers/db.py:47}}
5月 26 16:35:12 devstack-master nova-conductor[9578]: WARNING oslo_config.cfg [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Option "scheduler_default_filters" from group "DEFAULT" is deprecated. Use option "enabled_filters" from group "filter_scheduler".
5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_db.sqlalchemy.engines [req-40896def-378f-4ba6-895c-023bab013faf admin admin] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=9955) _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:260}}
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Failed to schedule instances
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager Traceback (most recent call last):
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/conductor/manager.py", line 916, in schedule_and_build_instances
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/conductor/manager.py", line 637, in _schedule_instances
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/utils.py", line 371, in wrapped
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return func(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/client/__init__.py", line 51, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/client/__init__.py", line 37, in __run_method
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/client/query.py", line 32, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/rpcapi.py", line 130, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager retry=self.retry)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager timeout=timeout, retry=retry)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager retry=retry)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 496, in _send
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager raise result
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager HostMappingNotFound_Remote: Host 'devstack-master' is not mapped to any cell
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager Traceback (most recent call last):
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager res = self.dispatcher.dispatch(message)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self._do_dispatch(endpoint, method, ctxt, args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager result = func(ctxt, **new_args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 229, in inner
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return func(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/manager.py", line 98, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 57, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager selected_hosts = self._schedule(context, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 102, in _schedule
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager hosts = self._get_all_host_states(elevated, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 180, in _get_all_host_states
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 646, in get_host_states_by_uuids
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return self._get_host_states(context, compute_nodes, services)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 690, in _get_host_states
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager self._get_instance_info(context, compute))
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 730, in _get_instance_info
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager inst_dict = self._get_instances_by_host(context, host_name)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/scheduler/host_manager.py", line 709, in _get_instances_by_host
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager hm = objects.HostMapping.get_by_host(context, host_name)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager result = fn(cls, context, *args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/objects/host_mapping.py", line 100, in get_by_host
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager db_mapping = cls._get_by_host_from_db(context, host)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 979, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager return fn(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager File "/opt/stack/nova/nova/objects/host_mapping.py", line 95, in _get_by_host_from_db
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager raise exception.HostMappingNotFound(name=host)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager HostMappingNotFound: Host 'devstack-master' is not mapped to any cell
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR nova.conductor.manager
5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_concurrency.lockutils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=9955) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:270}}
5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_concurrency.lockutils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s {{(pid=9955) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282}}
5月 26 16:35:12 devstack-master nova-conductor[9578]: DEBUG oslo_db.sqlalchemy.engines [req-40896def-378f-4ba6-895c-023bab013faf admin admin] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=9955) _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:260}}
5月 26 16:35:12 devstack-master nova-conductor[9578]: WARNING nova.scheduler.utils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Failed to compute_task_build_instances: Host 'devstack-master' is not mapped to any cell
5月 26 16:35:12 devstack-master nova-conductor[9578]: Traceback (most recent call last):
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
5月 26 16:35:12 devstack-master nova-conductor[9578]: res = self.dispatcher.dispatch(message)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
5月 26 16:35:12 devstack-master nova-conductor[9578]: return self._do_dispatch(endpoint, method, ctxt, args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
5月 26 16:35:12 devstack-master nova-conductor[9578]: result = func(ctxt, **new_args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 229, in inner
5月 26 16:35:12 devstack-master nova-conductor[9578]: return func(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/manager.py", line 98, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: dests = self.driver.select_destinations(ctxt, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 57, in select_destinations
5月 26 16:35:12 devstack-master nova-conductor[9578]: selected_hosts = self._schedule(context, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 102, in _schedule
5月 26 16:35:12 devstack-master nova-conductor[9578]: hosts = self._get_all_host_states(elevated, spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 180, in _get_all_host_states
5月 26 16:35:12 devstack-master nova-conductor[9578]: spec_obj)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 646, in get_host_states_by_uuids
5月 26 16:35:12 devstack-master nova-conductor[9578]: return self._get_host_states(context, compute_nodes, services)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 690, in _get_host_states
5月 26 16:35:12 devstack-master nova-conductor[9578]: self._get_instance_info(context, compute))
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 730, in _get_instance_info
5月 26 16:35:12 devstack-master nova-conductor[9578]: inst_dict = self._get_instances_by_host(context, host_name)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/scheduler/host_manager.py", line 709, in _get_instances_by_host
5月 26 16:35:12 devstack-master nova-conductor[9578]: hm = objects.HostMapping.get_by_host(context, host_name)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: result = fn(cls, context, *args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/objects/host_mapping.py", line 100, in get_by_host
5月 26 16:35:12 devstack-master nova-conductor[9578]: db_mapping = cls._get_by_host_from_db(context, host)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 979, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: return fn(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: File "/opt/stack/nova/nova/objects/host_mapping.py", line 95, in _get_by_host_from_db
5月 26 16:35:12 devstack-master nova-conductor[9578]: raise exception.HostMappingNotFound(name=host)
5月 26 16:35:12 devstack-master nova-conductor[9578]: HostMappingNotFound: Host 'devstack-master' is not mapped to any cell
5月 26 16:35:12 devstack-master nova-conductor[9578]:
5月 26 16:35:12 devstack-master nova-conductor[9578]: WARNING nova.scheduler.utils [req-40896def-378f-4ba6-895c-023bab013faf admin admin] [instance: 421bb801-005a-4f73-9fe1-0916fd691b2b] Setting instance to ERROR state.
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server [req-40896def-378f-4ba6-895c-023bab013faf admin admin] Exception during message handling
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 920, in schedule_and_build_instances
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server build_requests=build_requests)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 890, in _bury_in_cell0
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server exc, legacy_spec)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 355, in _set_vm_state_and_notify
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server ex, request_spec)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/utils.py", line 103, in set_vm_state_and_notify
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server instance.save()
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return fn(self, *args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/objects/instance.py", line 781, in save
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server columns_to_join=_expected_cols(expected_attrs))
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/api.py", line 860, in instance_update_and_get_original
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server expected=expected)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server self.force_reraise()
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 251, in wrapped
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server return f(context, *args, **kwargs)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2673, in instance_update_and_get_original
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server columns_to_join=columns_to_join)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1929, in _instance_get_by_uuid
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server raise exception.InstanceNotFound(instance_id=uuid)
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server InstanceNotFound: Instance 421bb801-005a-4f73-9fe1-0916fd691b2b could not be found.
5月 26 16:35:12 devstack-master nova-conductor[9578]: ERROR oslo_messaging.rpc.server
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1693721/+subscriptions
