← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #79085

[Bug 1834691] [NEW] AttributeError: 'HostState' object has no attribute 'failed_builds'

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I'm not sure how this happens, but I hit this in a devstack environment
with 35 nova-compute services using the fake virt driver, one scheduler
worker and then trying to schedule a server:

Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server [None req-91eb7095-ab0e-4c51-ba65-669863c08cbf admin admin] Exception during message handling
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 165, in _p
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, i
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, i
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 228, in in
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/manager.py", line 192, in select_destinations
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     allocation_request_version, return_alternates)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     allocation_request_version, return_alternates)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 210, in _schedule
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     hosts = self._get_sorted_hosts(spec_obj, hosts, num)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 449, in _get_sorted_hosts
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     spec_obj)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/host_manager.py", line 600, in get_weighed_hosts
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     hosts, spec_obj)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/weights.py", line 137, in get_weighed_objects
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     weights = weigher.weigh_objects(weighed_objs, weighing_properties)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/weights.py", line 107, in weigh_objects
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     weight = self._weigh_object(obj.obj, weight_properties)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/weights/compute.py", line 36, in _weigh_object
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     return host_state.failed_builds
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server AttributeError: 'HostState' object has no attribute 'failed_builds'
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server 

The HostState.failed_builds attribute gets set outside of __init__ in
_update_from_compute_node here:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L261

And I'd expect the _update_from_compute_node to get called from update()
which is called from here:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L787

Ah bingo if the compute node hasn't had resource usage updated yet we
don't set the failed_builds attribute:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L191

** Affects: nova
     Importance: Medium
     Assignee: Matt Riedemann (mriedem)
         Status: Triaged

** Affects: nova/pike
     Importance: Medium
         Status: Triaged

** Affects: nova/queens
     Importance: Medium
         Status: Triaged

** Affects: nova/rocky
     Importance: Medium
         Status: Triaged

** Affects: nova/stein
     Importance: Medium
         Status: Triaged


** Tags: scheduler

** Changed in: nova
       Status: New => Triaged

** Changed in: nova
   Importance: Undecided => Medium

** Changed in: nova
     Assignee: (unassigned) => Matt Riedemann (mriedem)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1834691

Title:
  AttributeError: 'HostState' object has no attribute 'failed_builds'

Status in OpenStack Compute (nova):
  Triaged
Status in OpenStack Compute (nova) pike series:
  Triaged
Status in OpenStack Compute (nova) queens series:
  Triaged
Status in OpenStack Compute (nova) rocky series:
  Triaged
Status in OpenStack Compute (nova) stein series:
  Triaged

Bug description:
  I'm not sure how this happens, but I hit this in a devstack
  environment with 35 nova-compute services using the fake virt driver,
  one scheduler worker and then trying to schedule a server:

  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server [None req-91eb7095-ab0e-4c51-ba65-669863c08cbf admin admin] Exception during message handling
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 165, in _p
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, i
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, i
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 228, in in
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/manager.py", line 192, in select_destinations
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     allocation_request_version, return_alternates)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     allocation_request_version, return_alternates)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 210, in _schedule
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     hosts = self._get_sorted_hosts(spec_obj, hosts, num)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 449, in _get_sorted_hosts
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     spec_obj)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/host_manager.py", line 600, in get_weighed_hosts
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     hosts, spec_obj)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/weights.py", line 137, in get_weighed_objects
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     weights = weigher.weigh_objects(weighed_objs, weighing_properties)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/weights.py", line 107, in weigh_objects
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     weight = self._weigh_object(obj.obj, weight_properties)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/weights/compute.py", line 36, in _weigh_object
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     return host_state.failed_builds
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server AttributeError: 'HostState' object has no attribute 'failed_builds'
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server 

  The HostState.failed_builds attribute gets set outside of __init__ in
  _update_from_compute_node here:

  https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L261

  And I'd expect the _update_from_compute_node to get called from
  update() which is called from here:

  https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L787

  Ah bingo if the compute node hasn't had resource usage updated yet we
  don't set the failed_builds attribute:

  https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L191

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1834691/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next