yahoo-eng-team team mailing list archive

[OpenStack Infra <1834691@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/668243
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d540903463aa9b0cf69cefac7cc60e5b70e40a1c
Submitter: Zuul
Branch:    master

commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c
Author: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date:   Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds
    
    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.
    
    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1834691

Title:
  AttributeError: 'HostState' object has no attribute 'failed_builds'

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) pike series:
  Triaged
Status in OpenStack Compute (nova) queens series:
  Triaged
Status in OpenStack Compute (nova) rocky series:
  Triaged
Status in OpenStack Compute (nova) stein series:
  Triaged

Bug description:
  I'm not sure how this happens, but I hit this in a devstack
  environment with 35 nova-compute services using the fake virt driver,
  one scheduler worker and then trying to schedule a server:

  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server [None req-91eb7095-ab0e-4c51-ba65-669863c08cbf admin admin] Exception during message handling
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 165, in _p
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, i
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, i
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 228, in in
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/manager.py", line 192, in select_destinations
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     allocation_request_version, return_alternates)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     allocation_request_version, return_alternates)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 210, in _schedule
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     hosts = self._get_sorted_hosts(spec_obj, hosts, num)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 449, in _get_sorted_hosts
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     spec_obj)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/host_manager.py", line 600, in get_weighed_hosts
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     hosts, spec_obj)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/weights.py", line 137, in get_weighed_objects
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     weights = weigher.weigh_objects(weighed_objs, weighing_properties)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/weights.py", line 107, in weigh_objects
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     weight = self._weigh_object(obj.obj, weight_properties)
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/scheduler/weights/compute.py", line 36, in _weigh_object
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server     return host_state.failed_builds
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server AttributeError: 'HostState' object has no attribute 'failed_builds'
  Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server 

  The HostState.failed_builds attribute gets set outside of __init__ in
  _update_from_compute_node here:

  https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L261

  And I'd expect the _update_from_compute_node to get called from
  update() which is called from here:

  https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L787

  Ah bingo if the compute node hasn't had resource usage updated yet we
  don't set the failed_builds attribute:

  https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L191

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1834691/+subscriptions