yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem.os@xxxxxxxxx>]

The HostState.failed_builds change was introduced in Rocky and
backported through to Pike:

https://review.opendev.org/#/q/I71c56fe770f8c3f66db97fa542fdfdf2b9865fb8

** Tags added: scheduler

** Also affects: nova/queens
   Importance: Undecided
       Status: New

** Also affects: nova/stein
   Importance: Undecided
       Status: New

** Also affects: nova/pike
   Importance: Undecided
       Status: New

** Also affects: nova/rocky
   Importance: Undecided
       Status: New

** Changed in: nova/pike
       Status: New => Triaged

** Changed in: nova/stein
   Importance: Undecided => Medium

** Changed in: nova/queens
       Status: New => Triaged

** Changed in: nova/queens
   Importance: Undecided => Medium

** Changed in: nova/rocky
   Importance: Undecided => Medium

** Changed in: nova/pike
   Importance: Undecided => Medium

** Changed in: nova/rocky
       Status: New => Triaged

** Changed in: nova/stein
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1834691

Title:
  AttributeError: 'HostState' object has no attribute 'failed_builds'

Status in OpenStack Compute (nova):
  Triaged
Status in OpenStack Compute (nova) pike series:
  Triaged
Status in OpenStack Compute (nova) queens series:
  Triaged
Status in OpenStack Compute (nova) rocky series:
  Triaged
Status in OpenStack Compute (nova) stein series:
  Triaged

Bug description:
  I'm not sure how this happens, but I hit this in a devstack
  environment with 35 nova-compute services using the fake virt driver,
  one scheduler worker and then trying to schedule a server:

  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server [None req-91eb7095-ab0e-4c51-ba65-669863c08cbf admin admin] Exception during message handling
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 165, in _p
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, i
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, i
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 228, in in
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/manager.py", line 192, in select_destinations
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     allocation_request_version, return_alternates)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     allocation_request_version, return_alternates)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 210, in _schedule
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     hosts = self._get_sorted_hosts(spec_obj, hosts, num)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 449, in _get_sorted_hosts
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     spec_obj)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/host_manager.py", line 600, in get_weighed_hosts
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     hosts, spec_obj)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/weights.py", line 137, in get_weighed_objects
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     weights = weigher.weigh_objects(weighed_objs, weighing_properties)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/weights.py", line 107, in weigh_objects
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     weight = self._weigh_object(obj.obj, weight_properties)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/weights/compute.py", line 36, in _weigh_object
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     return host_state.failed_builds
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server AttributeError: 'HostState' object has no attribute 'failed_builds'
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server 

  The HostState.failed_builds attribute gets set outside of __init__ in
  _update_from_compute_node here:

  https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L261

  And I'd expect the _update_from_compute_node to get called from
  update() which is called from here:

  https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L787

  Ah bingo if the compute node hasn't had resource usage updated yet we
  don't set the failed_builds attribute:

  https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L191

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1834691/+subscriptions