yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

*** This bug is a duplicate of bug 1610679 ***
    https://bugs.launchpad.net/bugs/1610679

** This bug is no longer a duplicate of bug 1515870
   server can not launch while there's a new nova compute node registed failed
** This bug has been marked a duplicate of bug 1610679
   race conditions between compute and schedule disk report

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1517770

Title:
  NULL free_disk_gb causes scheduler failure

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  It appears a race exists between nova-scheduler and the compute
  manager when a ComputeNode entry is created for the first time.

  The following log messages were noticed after multiple transient
  failures to create VM on a newly deployed single node system.

  2015-11-03 18:41:27.886 13735 WARNING nova.scheduler.host_manager [req-dd2b0758-78a4-4a67-90c8-9586d4d55489 db30a70a389548ed916f52d2f5c25544 617c3194750f44cfa1e9a747b2ac36f5 - - -] Host zs-zhost1 has more disk space than database expected (13119gb > Nonegb)
  2015-11-03 18:41:27.904 13783 WARNING nova.scheduler.utils [req-dd2b0758-78a4-4a67-90c8-9586d4d55489 db30a70a389548ed916f52d2f5c25544 617c3194750f44cfa1e9a747b2ac36f5 - - -] Failed to compute_task_build_instances: unsupported operand type(s) for *: 'NoneType' and 'int'
  Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  executor_callback))
  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  executor_callback)
  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  result = func(ctxt, **new_args)
  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner
  return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 86, in select_destinations
  filter_properties)
  File "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", line 67, in select_destinations
  filter_properties)
  File "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", line 131, in _schedule
  hosts = self._get_all_host_states(elevated)
  File "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", line 176, in _get_all_host_states
  return self.host_manager.get_all_host_states(context)
  File "/usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py", line 552, in get_all_host_states
  host_state = self.host_state_cls(host, node, compute=compute)
  File "/usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py", line 309, in host_state_cls
  return HostState(host, node, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py", line 157, in _init_
  self.update_from_compute_node(compute)
  File "/usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py", line 202, in update_from_compute_node
  free_disk_mb = free_gb * 1024
  TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
  2015-11-03 18:41:27.907 13783 WARNING nova.scheduler.utils [req-dd2b0758-78a4-4a67-90c8-9586d4d55489 db30a70a389548ed916f52d2f5c25544 617c3194750f44cfa1e9a747b2ac36f5 - - -] [instance: bd6bb6a7-e917-4ce7-b207-817144ac7853] Setting instance to ERROR state.

  I believe that during the execution of
  resource_tracker._update_available_resource() for a new node, the
  period between the initial insert of the ComputeNode entry in
  _init_compute_node() and the call to _update() leaves a ComputeNode
  with a NULL free_disk_gb for a small window of time.

  Commit 6aa36ab seems likely to have exposed this more widely.

  Versions (Kilo):
  ii  nova-common                          1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute - common files
  ii  nova-compute                         1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                     1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                 1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute - compute node libvirt support
  ii  python-nova                          1:2015.1.1-0ubuntu1~cloud2            all          OpenStack Compute Python libraries
  ii  python-novaclient                    1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1517770/+subscriptions