yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1515870] Re: server can not launch while there's a new nova compute node registed failed

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Wed, 04 Jan 2017 23:49:31 -0000
Reply-to: Bug 1515870 <1515870@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
*** This bug is a duplicate of bug 1610679 ***
    https://bugs.launchpad.net/bugs/1610679

For the non-Ironic case that this bug was originally reported against,
it might be a duplicate of bug 1610679 which was fixed in Ocata and
backported to Newton:

https://review.openstack.org/#/q/Ia68298a3f01d89bbf302ac734389f7282176c553,n,z

** This bug has been marked a duplicate of bug 1610679
   race conditions between compute and schedule disk report

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1515870

Title:
  server can not launch while there's a new nova compute node registed
  failed

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  1. Exact version of Nova/OpenStack you are running:
  kilo 2015.1.0
  2.Relevant log files:
  2015-11-03 16:00:29.990 3568 ERROR oslo_messaging.rpc.dispatcher [req-ce8d5d3d-6a79-4827-b472-02940be546bc 60ca5cf0e1bf44b985ee5ceae440fcfc b2a5638f40fd43a59a9be1e9c12f7d89 - - -] Exception during message handling: unsupported operand type(s) for *: 'NoneType' and 'int'
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 86, in select_destinations
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher     filter_properties)
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 67, in select_destinations
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher     filter_properties)
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 131, in _schedule
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher     hosts = self._get_all_host_states(elevated)
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 176, in _get_all_host_states
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher     return self.host_manager.get_all_host_states(context)
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 561, in get_all_host_states
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher     host_state = self.host_state_cls(host, node, compute=compute)
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 318, in host_state_cls
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher     return HostState(host, node, **kwargs)
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 163, in __init__
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher     self.update_from_compute_node(compute)
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 208, in update_from_compute_node
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher     free_disk_mb = free_gb * 1024
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
  2015-11-03 16:00:29.990 3568 TRACE oslo_messaging.rpc.dispatcher
  2015-11-03 16:00:29.991 3568 ERROR oslo_messaging._drivers.common [req-ce8d5d3d-6a79-4827-b472-02940be546bc 60ca5cf0e1bf44b985ee5ceae440fcfc b2a5638f40fd43a59a9be1e9c12f7d89 - - -] Returning exception unsupported operand type(s) for *: 'NoneType' and 'int' to caller
  2015-11-03 16:00:29.991 3568 ERROR oslo_messaging._drivers.common [req-ce8d5d3d-6a79-4827-b472-02940be546bc 60ca5cf0e1bf44b985ee5ceae440fcfc b2a5638f40fd43a59a9be1e9c12f7d89 - - -] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner\n    return func(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 86, in select_destinations\n    filter_properties)\n', '  File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 67, in select_destinations\n    filter_properties)\n', '  File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 131, in _schedule\n    hosts = self._get_all_host_states(elevated)\n', '  File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 176, in _get_all_host_states\n    return self.host_manager.get_all_host_states(context)\n', '  File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 561, in get_all_host_states\n    host_state = self.host_state_cls(host, node, compute=compute)\n', '  File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 318, in host_state_cls\n    return HostState(host, node, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 163, in __init__\n    self.update_from_compute_node(compute)\n', '  File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 208, in update_from_compute_node\n    free_disk_mb = free_gb * 1024\n', "TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'\n"]

  3.Reproduce steps:
  I can launch VM success,then add a nova compute.

  But the info of  nova compute registed to mariadb is not enough.

  eg.  the fram_ram_mb in the table  compute_nodes is null.

  MariaDB [nova]> select created_at ,updated_at ,deleted_at ,service_id ,free_ram_mb , free_disk_gb  ,id ,current_workload , hypervisor_hostname  from compute_nodes where host='NailSBCJslot3rackN9';
  +---------------------+------------+------------+------------+-------------+--------------+----+------------------+---------------------+
  | created_at          | updated_at | deleted_at | service_id | free_ram_mb | free_disk_gb | id | current_workload | hypervisor_hostname |
  +---------------------+------------+------------+------------+-------------+--------------+----+------------------+---------------------+
  | 2015-11-03 07:31:57 | NULL       | NULL       |          8 |        NULL |         NULL |  2 |             NULL | NailSBCJslot3rackN9 |
  +---------------------+------------+------------+------------+-------------+--------------+----+------------------+---------------------+

  Expected result:
  * the VM can launch success.
  There are other nova computes in the system, where can launch VM success.
  The "bad" nova compute should not cause the new VM launched failed.

  Actual result:
  * VM launch failed.

  4. Bonus points for reproducing shell script / test:
  When launch VM, check whether the items in the table of compute_nodes is null.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1515870/+subscriptions
References

[Bug 1515870] [NEW] server can not launch while there's a nova compute node registed failed
From: lingchen, 2015-11-13