yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1446082] [NEW] Instance without "extra" data crashes nova-compute

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Christoph Dwertmann <1446082@xxxxxxxxxxxxxxxxxx>
Date: Mon, 20 Apr 2015 07:42:06 -0000
Reply-to: Bug 1446082 <1446082@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I'm upgrading from Icehouse to Kilo. I have a single instance that was
created in Icehouse. After the upgrade, nova-compute crashes because
it's looking for instance "extra" data that is not there.

To fix this, we need to check if there is any "extra" data for the
instance before trying to read properties such as "numa_topology".

# dpkg -l | grep nova
ii  nova-common                         1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - common files
ii  nova-compute                        1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - compute node base
ii  nova-compute-kvm                    1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - compute node libvirt support
ii  python-nova                         1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute Python libraries
ii  python-novaclient                   1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

nova-compute.log:
2015-04-20 17:35:09.214 15508 DEBUG oslo_concurrency.lockutils [req-43d3110a-cac7-425e-842c-f725bda91c10 - - - - -] Lock "compute_resources" acquired by "_update_available_resource" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:444
2015-04-20 17:35:09.299 15508 DEBUG oslo_concurrency.lockutils [req-43d3110a-cac7-425e-842c-f725bda91c10 - - - - -] Lock "compute_resources" released by "_update_available_resource" :: held 0.085s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
    timer()
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
    cb(*args, **kw)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 497, in run_service
    service.start()
  File "/usr/lib/python2.7/dist-packages/nova/service.py", line 183, in start
    self.manager.pre_start_hook()
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1287, in pre_start_hook
    self.update_available_resource(nova.context.get_admin_context())
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6236, in update_available_resource
    rt.update_available_resource(context)
  File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 402, in update_available_resource
    self._update_available_resource(context, resources)
  File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 436, in _update_available_resource
    'numa_topology'])
  File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in wrapper
    result = fn(cls, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1152, in get_by_host_and_node
    expected_attrs)
  File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1068, in _make_instance_list
    expected_attrs=expected_attrs)
  File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 501, in _from_db_object
    db_inst.get('extra').get('numa_topology'))
AttributeError: 'NoneType' object has no attribute 'get'
2015-04-20 17:35:09.301 15508 ERROR nova.openstack.common.threadgroup [req-12483464-12a6-4b74-a671-bc6bb943b265 - - - - -] 'NoneType' object has no attribute 'get'
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 145, in wait
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     x.wait()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 497, in run_service
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     service.start()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 183, in start
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1287, in pre_start_hook
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6236, in update_available_resource
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     rt.update_available_resource(context)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 402, in update_available_resource
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     self._update_available_resource(context, resources)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 436, in _update_available_resource
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     'numa_topology'])
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in wrapper
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     result = fn(cls, context, *args, **kwargs)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1152, in get_by_host_and_node
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     expected_attrs)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1068, in _make_instance_list
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     expected_attrs=expected_attrs)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 501, in _from_db_object
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     db_inst.get('extra').get('numa_topology'))
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup AttributeError: 'NoneType' object has no attribute 'get'
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup

** Affects: nova
     Importance: Undecided
     Assignee: Christoph Dwertmann (cdwertmann)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1446082

Title:
  Instance without "extra" data crashes nova-compute

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  I'm upgrading from Icehouse to Kilo. I have a single instance that was
  created in Icehouse. After the upgrade, nova-compute crashes because
  it's looking for instance "extra" data that is not there.

  To fix this, we need to check if there is any "extra" data for the
  instance before trying to read properties such as "numa_topology".

  # dpkg -l | grep nova
  ii  nova-common                         1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - common files
  ii  nova-compute                        1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                    1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - compute node libvirt support
  ii  python-nova                         1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute Python libraries
  ii  python-novaclient                   1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

  nova-compute.log:
  2015-04-20 17:35:09.214 15508 DEBUG oslo_concurrency.lockutils [req-43d3110a-cac7-425e-842c-f725bda91c10 - - - - -] Lock "compute_resources" acquired by "_update_available_resource" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:444
  2015-04-20 17:35:09.299 15508 DEBUG oslo_concurrency.lockutils [req-43d3110a-cac7-425e-842c-f725bda91c10 - - - - -] Lock "compute_resources" released by "_update_available_resource" :: held 0.085s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
      timer()
    File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
      cb(*args, **kw)
    File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
      result = function(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 497, in run_service
      service.start()
    File "/usr/lib/python2.7/dist-packages/nova/service.py", line 183, in start
      self.manager.pre_start_hook()
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1287, in pre_start_hook
      self.update_available_resource(nova.context.get_admin_context())
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6236, in update_available_resource
      rt.update_available_resource(context)
    File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 402, in update_available_resource
      self._update_available_resource(context, resources)
    File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
      return f(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 436, in _update_available_resource
      'numa_topology'])
    File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in wrapper
      result = fn(cls, context, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1152, in get_by_host_and_node
      expected_attrs)
    File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1068, in _make_instance_list
      expected_attrs=expected_attrs)
    File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 501, in _from_db_object
      db_inst.get('extra').get('numa_topology'))
  AttributeError: 'NoneType' object has no attribute 'get'
  2015-04-20 17:35:09.301 15508 ERROR nova.openstack.common.threadgroup [req-12483464-12a6-4b74-a671-bc6bb943b265 - - - - -] 'NoneType' object has no attribute 'get'
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 145, in wait
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     x.wait()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 47, in wait
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 497, in run_service
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     service.start()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 183, in start
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1287, in pre_start_hook
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6236, in update_available_resource
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     rt.update_available_resource(context)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 402, in update_available_resource
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     self._update_available_resource(context, resources)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 436, in _update_available_resource
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     'numa_topology'])
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in wrapper
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     result = fn(cls, context, *args, **kwargs)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1152, in get_by_host_and_node
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     expected_attrs)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1068, in _make_instance_list
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     expected_attrs=expected_attrs)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 501, in _from_db_object
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     db_inst.get('extra').get('numa_topology'))
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup AttributeError: 'NoneType' object has no attribute 'get'
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1446082/+subscriptions
Follow ups

[Bug 1446082] Re: Instance without "extra" data crashes nova-compute
From: Alan Pevec, 2015-07-29
[Bug 1446082] Re: Instance without "extra" data crashes nova-compute
From: Thierry Carrez, 2015-06-24
[Bug 1446082] Re: Instance without "extra" data crashes nova-compute
From: Alan Pevec, 2015-05-15
[Bug 1446082] [NEW] Instance without "extra" data crashes nova-compute
From: Christoph Dwertmann, 2015-04-20
References

[Bug 1446082] [NEW] Instance without "extra" data crashes nova-compute
From: Christoph Dwertmann, 2015-04-20