yahoo-eng-team team mailing list archive

[OpenStack Infra <1781648@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/597571
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4fe4fbe7a43ac7e971b0aabc8a9079c7701cae55
Submitter: Zuul
Branch:    master

commit 4fe4fbe7a43ac7e971b0aabc8a9079c7701cae55
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Wed Aug 29 11:58:12 2018 -0400

    Restart scheduler in TestNovaManagePlacementHealAllocations
    
    TestNovaManagePlacementHealAllocations tests rely on the
    CachingScheduler specifically because it does not use
    placement and therefore does not create allocations during
    scheduling, which gives us instances that need to heal
    allocations.
    
    However, we have a race in the test setup where the scheduler
    is started before the compute services. The CachingScheduler
    runs a periodic task on startup to fetch the hosts from the
    DB to initialize it's cache, and then uses that during scheduling
    until the periodic runs again. So what we can hit is the
    scheduler starts, loads an empty cache, then we start the computes
    and try to create an instance but because of the empty cache we
    fail with a NoValidHost error.
    
    This restarts and resets the CachingScheduler cache *after* we have
    started the computes and asserted they are available in the API.
    
    Change-Id: I32f607a436e9851a96877123ae3d1fe51f444f73
    Closes-Bug: #1781648


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1781648

Title:
  heal_allocations test randomly failing with "ValueError: Field
  `compute_node_uuid' cannot be None"

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) rocky series:
  Confirmed

Bug description:
  http://logs.openstack.org/44/564444/14/gate/nova-tox-
  functional/75cad04/job-output.txt.gz#_2018-07-13_16_27_07_833394

  Things are failing during scheduling:

  2018-07-13 16:27:07.846188 | ubuntu-xenial |     2018-07-13 16:27:02,302 INFO [nova.scheduler.host_manager] Host filter forcing available hosts to cell1
  2018-07-13 16:27:07.846244 | ubuntu-xenial |     2018-07-13 16:27:02,303 ERROR [oslo_messaging.rpc.server] Exception during message handling
  2018-07-13 16:27:07.846273 | ubuntu-xenial |     Traceback (most recent call last):
  2018-07-13 16:27:07.846367 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
  2018-07-13 16:27:07.846400 | ubuntu-xenial |         res = self.dispatcher.dispatch(message)
  2018-07-13 16:27:07.846492 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2018-07-13 16:27:07.846532 | ubuntu-xenial |         return self._do_dispatch(endpoint, method, ctxt, args)
  2018-07-13 16:27:07.846626 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2018-07-13 16:27:07.846663 | ubuntu-xenial |         result = func(ctxt, **new_args)
  2018-07-13 16:27:07.846762 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
  2018-07-13 16:27:07.846791 | ubuntu-xenial |         return func(*args, **kwargs)
  2018-07-13 16:27:07.846840 | ubuntu-xenial |       File "nova/scheduler/manager.py", line 169, in select_destinations
  2018-07-13 16:27:07.846877 | ubuntu-xenial |         allocation_request_version, return_alternates)
  2018-07-13 16:27:07.846925 | ubuntu-xenial |       File "nova/scheduler/filter_scheduler.py", line 91, in select_destinations
  2018-07-13 16:27:07.846966 | ubuntu-xenial |         allocation_request_version, return_alternates)
  2018-07-13 16:27:07.847010 | ubuntu-xenial |       File "nova/scheduler/filter_scheduler.py", line 177, in _schedule
  2018-07-13 16:27:07.847031 | ubuntu-xenial |         hosts, num_alts)
  2018-07-13 16:27:07.847080 | ubuntu-xenial |       File "nova/scheduler/filter_scheduler.py", line 320, in _legacy_find_hosts
  2018-07-13 16:27:07.847108 | ubuntu-xenial |         spec_obj, hosts, num, num_alts)
  2018-07-13 16:27:07.847157 | ubuntu-xenial |       File "nova/scheduler/filter_scheduler.py", line 356, in _get_alternate_hosts
  2018-07-13 16:27:07.847197 | ubuntu-xenial |         allocation_request_version=allocation_request_version)
  2018-07-13 16:27:07.847239 | ubuntu-xenial |       File "nova/objects/selection.py", line 60, in from_host_state
  2018-07-13 16:27:07.847279 | ubuntu-xenial |         allocation_request_version=allocation_request_version)
  2018-07-13 16:27:07.847369 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 307, in __init__
  2018-07-13 16:27:07.847398 | ubuntu-xenial |         setattr(self, key, kwargs[key])
  2018-07-13 16:27:07.847494 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 72, in setter
  2018-07-13 16:27:07.847536 | ubuntu-xenial |         field_value = field.coerce(self, name, value)
  2018-07-13 16:27:07.847627 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 199, in coerce
  2018-07-13 16:27:07.847662 | ubuntu-xenial |         return self._null(obj, attr)
  2018-07-13 16:27:07.847757 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 177, in _null
  2018-07-13 16:27:07.847800 | ubuntu-xenial |         raise ValueError(_("Field `%s' cannot be None") % attr)
  2018-07-13 16:27:07.847837 | ubuntu-xenial |     ValueError: Field `compute_node_uuid' cannot be None
  2018-07-13 16:27:07.847889 | ubuntu-xenial |     2018-07-13 16:27:02,305 ERROR [nova.conductor.manager] Failed to schedule instances
  2018-07-13 16:27:07.847917 | ubuntu-xenial |     Traceback (most recent call last):
  2018-07-13 16:27:07.847967 | ubuntu-xenial |       File "nova/conductor/manager.py", line 1189, in schedule_and_build_instances
  2018-07-13 16:27:07.848000 | ubuntu-xenial |         instance_uuids, return_alternates=True)
  2018-07-13 16:27:07.848044 | ubuntu-xenial |       File "nova/conductor/manager.py", line 714, in _schedule_instances
  2018-07-13 16:27:07.848075 | ubuntu-xenial |         return_alternates=return_alternates)
  2018-07-13 16:27:07.848113 | ubuntu-xenial |       File "nova/scheduler/utils.py", line 840, in wrapped
  2018-07-13 16:27:07.848139 | ubuntu-xenial |         return func(*args, **kwargs)
  2018-07-13 16:27:07.848187 | ubuntu-xenial |       File "nova/scheduler/client/__init__.py", line 53, in select_destinations
  2018-07-13 16:27:07.848225 | ubuntu-xenial |         instance_uuids, return_objects, return_alternates)
  2018-07-13 16:27:07.848270 | ubuntu-xenial |       File "nova/scheduler/client/__init__.py", line 37, in __run_method
  2018-07-13 16:27:07.848318 | ubuntu-xenial |         return getattr(self.instance, __name)(*args, **kwargs)
  2018-07-13 16:27:07.848368 | ubuntu-xenial |       File "nova/scheduler/client/query.py", line 42, in select_destinations
  2018-07-13 16:27:07.848406 | ubuntu-xenial |         instance_uuids, return_objects, return_alternates)
  2018-07-13 16:27:07.848449 | ubuntu-xenial |       File "nova/scheduler/rpcapi.py", line 158, in select_destinations
  2018-07-13 16:27:07.848491 | ubuntu-xenial |         return cctxt.call(ctxt, 'select_destinations', **msg_args)
  2018-07-13 16:27:07.848579 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
  2018-07-13 16:27:07.848600 | ubuntu-xenial |         retry=self.retry)
  2018-07-13 16:27:07.848688 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
  2018-07-13 16:27:07.864805 | ubuntu-xenial |         retry=retry)
  2018-07-13 16:27:07.864917 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send
  2018-07-13 16:27:07.864965 | ubuntu-xenial |         return self._send(target, ctxt, message, wait_for_reply, timeout)
  2018-07-13 16:27:07.865058 | ubuntu-xenial |       File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_fake.py", line 209, in _send
  2018-07-13 16:27:07.865078 | ubuntu-xenial |         raise failure
  2018-07-13 16:27:07.865115 | ubuntu-xenial |     ValueError: Field `compute_node_uuid' cannot be None

  Must have some weird issues with multi-cell setup in the test or maybe
  something with the way the compute services are started?

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20compute_task_build_instances%3A%20Field%20%60compute_node_uuid'%20cannot%20be%20None%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1781648/+subscriptions