yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1781648] Re: heal_allocations test randomly failing with "ValueError: Field `compute_node_uuid' cannot be None"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Wed, 29 Aug 2018 17:13:56 -0000
Reply-to: Bug 1781648 <1781648@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/rocky
   Importance: Undecided
       Status: New

** Changed in: nova/rocky
       Status: New => Confirmed

** Changed in: nova/rocky
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1781648

Title:
  heal_allocations test randomly failing with "ValueError: Field
  `compute_node_uuid' cannot be None"

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) rocky series:
  Confirmed

Bug description:
  http://logs.openstack.org/44/564444/14/gate/nova-tox-
  functional/75cad04/job-output.txt.gz#_2018-07-13_16_27_07_833394

  Things are failing during scheduling:

  2018-07-13 16:27:07.846188 | ubuntu-xenial |     2018-07-13 16:27:02,302 INFO [nova.scheduler.host_manager] Host filter forcing available hosts to cell1
  2018-07-13 16:27:07.846244 | ubuntu-xenial |     2018-07-13 16:27:02,303 ERROR [oslo_messaging.rpc.server] Exception during message handling
  2018-07-13 16:27:07.846273 | ubuntu-xenial |     Traceback (most recent call last):
  2018-07-13 16:27:07.846367 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  2018-07-13 16:27:07.846400 | ubuntu-xenial |         res = self.dispatcher.dispatch(message)
  2018-07-13 16:27:07.846492 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2018-07-13 16:27:07.846532 | ubuntu-xenial |         return self._do_dispatch(endpoint, method, ctxt, args)
  2018-07-13 16:27:07.846626 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2018-07-13 16:27:07.846663 | ubuntu-xenial |         result = func(ctxt, **new_args)
  2018-07-13 16:27:07.846762 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
  2018-07-13 16:27:07.846791 | ubuntu-xenial |         return func(*args, **kwargs)
  2018-07-13 16:27:07.846840 | ubuntu-xenial |       File "nova/scheduler/manager.py", line 169, in select_destinations
  2018-07-13 16:27:07.846877 | ubuntu-xenial |         allocation_request_version, return_alternates)
  2018-07-13 16:27:07.846925 | ubuntu-xenial |       File "nova/scheduler/filter_scheduler.py", line 91, in select_destinations
  2018-07-13 16:27:07.846966 | ubuntu-xenial |         allocation_request_version, return_alternates)
  2018-07-13 16:27:07.847010 | ubuntu-xenial |       File "nova/scheduler/filter_scheduler.py", line 177, in _schedule
  2018-07-13 16:27:07.847031 | ubuntu-xenial |         hosts, num_alts)
  2018-07-13 16:27:07.847080 | ubuntu-xenial |       File "nova/scheduler/filter_scheduler.py", line 320, in _legacy_find_hosts
  2018-07-13 16:27:07.847108 | ubuntu-xenial |         spec_obj, hosts, num, num_alts)
  2018-07-13 16:27:07.847157 | ubuntu-xenial |       File "nova/scheduler/filter_scheduler.py", line 356, in _get_alternate_hosts
  2018-07-13 16:27:07.847197 | ubuntu-xenial |         allocation_request_version=allocation_request_version)
  2018-07-13 16:27:07.847239 | ubuntu-xenial |       File "nova/objects/selection.py", line 60, in from_host_state
  2018-07-13 16:27:07.847279 | ubuntu-xenial |         allocation_request_version=allocation_request_version)
  2018-07-13 16:27:07.847369 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 307, in __init__
  2018-07-13 16:27:07.847398 | ubuntu-xenial |         setattr(self, key, kwargs[key])
  2018-07-13 16:27:07.847494 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 72, in setter
  2018-07-13 16:27:07.847536 | ubuntu-xenial |         field_value = field.coerce(self, name, value)
  2018-07-13 16:27:07.847627 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 199, in coerce
  2018-07-13 16:27:07.847662 | ubuntu-xenial |         return self._null(obj, attr)
  2018-07-13 16:27:07.847757 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 177, in _null
  2018-07-13 16:27:07.847800 | ubuntu-xenial |         raise ValueError(_("Field `%s' cannot be None") % attr)
  2018-07-13 16:27:07.847837 | ubuntu-xenial |     ValueError: Field `compute_node_uuid' cannot be None
  2018-07-13 16:27:07.847889 | ubuntu-xenial |     2018-07-13 16:27:02,305 ERROR [nova.conductor.manager] Failed to schedule instances
  2018-07-13 16:27:07.847917 | ubuntu-xenial |     Traceback (most recent call last):
  2018-07-13 16:27:07.847967 | ubuntu-xenial |       File "nova/conductor/manager.py", line 1189, in schedule_and_build_instances
  2018-07-13 16:27:07.848000 | ubuntu-xenial |         instance_uuids, return_alternates=True)
  2018-07-13 16:27:07.848044 | ubuntu-xenial |       File "nova/conductor/manager.py", line 714, in _schedule_instances
  2018-07-13 16:27:07.848075 | ubuntu-xenial |         return_alternates=return_alternates)
  2018-07-13 16:27:07.848113 | ubuntu-xenial |       File "nova/scheduler/utils.py", line 840, in wrapped
  2018-07-13 16:27:07.848139 | ubuntu-xenial |         return func(*args, **kwargs)
  2018-07-13 16:27:07.848187 | ubuntu-xenial |       File "nova/scheduler/client/__init__.py", line 53, in select_destinations
  2018-07-13 16:27:07.848225 | ubuntu-xenial |         instance_uuids, return_objects, return_alternates)
  2018-07-13 16:27:07.848270 | ubuntu-xenial |       File "nova/scheduler/client/__init__.py", line 37, in __run_method
  2018-07-13 16:27:07.848318 | ubuntu-xenial |         return getattr(self.instance, __name)(*args, **kwargs)
  2018-07-13 16:27:07.848368 | ubuntu-xenial |       File "nova/scheduler/client/query.py", line 42, in select_destinations
  2018-07-13 16:27:07.848406 | ubuntu-xenial |         instance_uuids, return_objects, return_alternates)
  2018-07-13 16:27:07.848449 | ubuntu-xenial |       File "nova/scheduler/rpcapi.py", line 158, in select_destinations
  2018-07-13 16:27:07.848491 | ubuntu-xenial |         return cctxt.call(ctxt, 'select_destinations', **msg_args)
  2018-07-13 16:27:07.848579 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2018-07-13 16:27:07.848600 | ubuntu-xenial |         retry=self.retry)
  2018-07-13 16:27:07.848688 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2018-07-13 16:27:07.864805 | ubuntu-xenial |         retry=retry)
  2018-07-13 16:27:07.864917 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send
  2018-07-13 16:27:07.864965 | ubuntu-xenial |         return self._send(target, ctxt, message, wait_for_reply, timeout)
  2018-07-13 16:27:07.865058 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 209, in _send
  2018-07-13 16:27:07.865078 | ubuntu-xenial |         raise failure
  2018-07-13 16:27:07.865115 | ubuntu-xenial |     ValueError: Field `compute_node_uuid' cannot be None

  Must have some weird issues with multi-cell setup in the test or maybe
  something with the way the compute services are started?

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20compute_task_build_instances%3A%20Field%20%60compute_node_uuid'%20cannot%20be%20None%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1781648/+subscriptions
References

[Bug 1781648] [NEW] heal_allocations test randomly failing with "ValueError: Field `compute_node_uuid' cannot be None"
From: Matt Riedemann, 2018-07-13