yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1460053] [NEW] nova-compute fails with "AttributeError: 'NoneType' object has no attribute 'get'" after kilo upgrade

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Francois Deppierraz <1460053@xxxxxxxxxxxxxxxxxx>
Date: Fri, 29 May 2015 12:42:49 -0000
Reply-to: Bug 1460053 <1460053@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

On compute node 'compute2', nova-compute fails to start with the
following exception:

2015-05-29 14:12:42.545 16355 ERROR nova.openstack.common.threadgroup [req-a1d0fd3b-e3ff-48af-a568-4198ca22e3bc - - - - -] 'NoneType' object has no attribute 'get'
Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 422, in _object_dispatch
    return getattr(target, method)(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in wrapper
    result = fn(cls, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1152, in get_by_host_and_node
    expected_attrs)

  File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1068, in _make_instance_list
    expected_attrs=expected_attrs)

  File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 501, in _from_db_object
    db_inst.get('extra').get('numa_topology'))

AttributeError: 'NoneType' object has no attribute 'get'
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 145, in wait
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     x.wait()
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 497, in run_service
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     service.start()
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 183, in start
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1291, in pre_start_hook
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6240, in update_available_resource
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     rt.update_available_resource(context)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 402, in update_available_resource
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     self._update_available_resource(context, resources)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 436, in _update_available_resource
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     'numa_topology'])
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 161, in wrapper
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     args, kwargs)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 335, in object_class_action
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     objver=objver, args=args, kwargs=kwargs)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     retry=self.retry)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     timeout=timeout, retry=retry)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     retry=retry)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     raise result
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup AttributeError: 'NoneType' object has no attribute 'get'
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 422, in _object_dispatch
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     return getattr(target, method)(*args, **kwargs)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in wrapper
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     result = fn(cls, context, *args, **kwargs)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1152, in get_by_host_and_node
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     expected_attrs)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1068, in _make_instance_list
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     expected_attrs=expected_attrs)
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 501, in _from_db_object
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     db_inst.get('extra').get('numa_topology'))
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup AttributeError: 'NoneType' object has no attribute 'get'
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 

Both the compute node and the controller node are running the latest
kilo ubuntu packages under Ubuntu 14.04.

root@compute2:~# dpkg -l nova-compute
ii  nova-compute                                         1:2015.1.0-0ubuntu1~cloud0      all                             OpenStack Compute - compute node base
root@compute2:~# 

root@controller1:~# dpkg -l nova-conductor
ii  nova-conductor                                       1:2015.1.0-0ubuntu1~cloud0      all                             OpenStack Compute - conductor service
root@controller1:~#

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1460053

Title:
  nova-compute fails with "AttributeError: 'NoneType' object has no
  attribute 'get'" after kilo upgrade

Status in OpenStack Compute (Nova):
  New

Bug description:
  On compute node 'compute2', nova-compute fails to start with the
  following exception:

  2015-05-29 14:12:42.545 16355 ERROR nova.openstack.common.threadgroup [req-a1d0fd3b-e3ff-48af-a568-4198ca22e3bc - - - - -] 'NoneType' object has no attribute 'get'
  Traceback (most recent call last):

    File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 422, in _object_dispatch
      return getattr(target, method)(*args, **kwargs)

    File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in wrapper
      result = fn(cls, context, *args, **kwargs)

    File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1152, in get_by_host_and_node
      expected_attrs)

    File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1068, in _make_instance_list
      expected_attrs=expected_attrs)

    File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 501, in _from_db_object
      db_inst.get('extra').get('numa_topology'))

  AttributeError: 'NoneType' object has no attribute 'get'
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 145, in wait
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     x.wait()
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 47, in wait
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 497, in run_service
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     service.start()
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 183, in start
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1291, in pre_start_hook
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6240, in update_available_resource
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     rt.update_available_resource(context)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 402, in update_available_resource
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     self._update_available_resource(context, resources)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 436, in _update_available_resource
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     'numa_topology'])
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 161, in wrapper
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     args, kwargs)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 335, in object_class_action
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     objver=objver, args=args, kwargs=kwargs)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     retry=self.retry)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     timeout=timeout, retry=retry)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     retry=retry)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     raise result
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup AttributeError: 'NoneType' object has no attribute 'get'
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 422, in _object_dispatch
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     return getattr(target, method)(*args, **kwargs)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in wrapper
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     result = fn(cls, context, *args, **kwargs)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1152, in get_by_host_and_node
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     expected_attrs)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1068, in _make_instance_list
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     expected_attrs=expected_attrs)
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 501, in _from_db_object
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup     db_inst.get('extra').get('numa_topology'))
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup AttributeError: 'NoneType' object has no attribute 'get'
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 
  2015-05-29 14:12:42.545 16355 TRACE nova.openstack.common.threadgroup 

  Both the compute node and the controller node are running the latest
  kilo ubuntu packages under Ubuntu 14.04.

  root@compute2:~# dpkg -l nova-compute
  ii  nova-compute                                         1:2015.1.0-0ubuntu1~cloud0      all                             OpenStack Compute - compute node base
  root@compute2:~# 

  root@controller1:~# dpkg -l nova-conductor
  ii  nova-conductor                                       1:2015.1.0-0ubuntu1~cloud0      all                             OpenStack Compute - conductor service
  root@controller1:~#

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1460053/+subscriptions
Follow ups

[Bug 1460053] [NEW] nova-compute fails with "AttributeError: 'NoneType' object has no attribute 'get'" after kilo upgrade
From: Francois Deppierraz, 2015-05-29
References

[Bug 1460053] [NEW] nova-compute fails with "AttributeError: 'NoneType' object has no attribute 'get'" after kilo upgrade
From: Francois Deppierraz, 2015-05-29