yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1765568] [NEW] AttributeError: 'NoneType' object has no attribute 'generation' in stable/ocata CI

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Fri, 20 Apr 2018 01:04:50 -0000
Reply-to: Bug 1765568 <1765568@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

We've been seeing this intermittently:

http://logs.openstack.org/33/559033/2/gate/legacy-tempest-dsvm-neutron-
full/ad0acef/logs/screen-n-cpu.txt.gz?level=TRACE#_2018-04-19_21_26_05_260

2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager [req-f2f68994-4031-465c-b1cc-0e995617b860 - -] Error updating resources for node ubuntu-xenial-inap-mtl01-0003624986.
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager Traceback (most recent call last):
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/manager.py", line 6582, in update_available_resource_for_node
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 551, in update_available_resource
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     self._update_available_resource(context, resources)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     return f(*args, **kwargs)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 575, in _update_available_resource
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     self._init_compute_node(context, resources)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 438, in _init_compute_node
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     self.scheduler_client.update_resource_stats(cn)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 60, in update_resource_stats
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     self.reportclient.update_resource_stats(compute_node)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 37, in __run_method
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 642, in update_resource_stats
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     self._update_inventory(compute_node.uuid, inv_data)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 55, in wrapper
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     return f(self, *a, **k)
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 563, in _update_inventory
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     if self._update_inventory_attempt(rp_uuid, inv_data):
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 471, in _update_inventory_attempt
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     cur_rp_gen = self._resource_providers[rp_uuid].generation
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager AttributeError: 'NoneType' object has no attribute 'generation'
2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager 

Since this change merged in stable/ocata:

https://review.openstack.org/#/c/525309/

We have a revert up for it: https://review.openstack.org/#/c/562739/

This bug is for tracking the revert and the failure in elastic-recheck.

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AttributeError%3A%20'NoneType'%20object%20has%20no%20attribute%20'generation'%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20build_branch%3A%5C%22stable%2Focata%5C%22&from=7d

** Affects: nova
     Importance: Undecided
         Status: Invalid

** Affects: nova/ocata
     Importance: Critical
     Assignee: Lee Yarwood (lyarwood)
         Status: In Progress

** Also affects: nova/ocata
   Importance: Undecided
       Status: New

** Changed in: nova
       Status: New => Invalid

** Changed in: nova/ocata
       Status: New => In Progress

** Changed in: nova/ocata
   Importance: Undecided => Critical

** Changed in: nova/ocata
     Assignee: (unassigned) => Lee Yarwood (lyarwood)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1765568

Title:
  AttributeError: 'NoneType' object has no attribute 'generation' in
  stable/ocata CI

Status in OpenStack Compute (nova):
  Invalid
Status in OpenStack Compute (nova) ocata series:
  In Progress

Bug description:
  We've been seeing this intermittently:

  http://logs.openstack.org/33/559033/2/gate/legacy-tempest-dsvm-
  neutron-
  full/ad0acef/logs/screen-n-cpu.txt.gz?level=TRACE#_2018-04-19_21_26_05_260

  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager [req-f2f68994-4031-465c-b1cc-0e995617b860 - -] Error updating resources for node ubuntu-xenial-inap-mtl01-0003624986.
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager Traceback (most recent call last):
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/manager.py", line 6582, in update_available_resource_for_node
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 551, in update_available_resource
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     self._update_available_resource(context, resources)
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     return f(*args, **kwargs)
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 575, in _update_available_resource
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     self._init_compute_node(context, resources)
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 438, in _init_compute_node
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     self.scheduler_client.update_resource_stats(cn)
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 60, in update_resource_stats
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     self.reportclient.update_resource_stats(compute_node)
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 37, in __run_method
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     return getattr(self.instance, __name)(*args, **kwargs)
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 642, in update_resource_stats
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     self._update_inventory(compute_node.uuid, inv_data)
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 55, in wrapper
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     return f(self, *a, **k)
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 563, in _update_inventory
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     if self._update_inventory_attempt(rp_uuid, inv_data):
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/scheduler/client/report.py", line 471, in _update_inventory_attempt
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager     cur_rp_gen = self._resource_providers[rp_uuid].generation
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager AttributeError: 'NoneType' object has no attribute 'generation'
  2018-04-19 21:26:05.260 12465 ERROR nova.compute.manager 

  Since this change merged in stable/ocata:

  https://review.openstack.org/#/c/525309/

  We have a revert up for it: https://review.openstack.org/#/c/562739/

  This bug is for tracking the revert and the failure in elastic-
  recheck.

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AttributeError%3A%20'NoneType'%20object%20has%20no%20attribute%20'generation'%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20build_branch%3A%5C%22stable%2Focata%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1765568/+subscriptions
Follow ups

[Bug 1765568] Re: AttributeError: 'NoneType' object has no attribute 'generation' in stable/ocata CI
From: Matt Riedemann, 2018-05-03