yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1446082] Re: Instance without "extra" data crashes nova-compute

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alan Pevec <1446082@xxxxxxxxxxxxxxxxxx>
Date: Fri, 15 May 2015 14:13:00 -0000
Reply-to: Bug 1446082 <1446082@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/kilo
   Importance: Undecided
       Status: New

** Changed in: nova/kilo
       Status: New => In Progress

** Changed in: nova/kilo
   Importance: Undecided => High

** Changed in: nova/kilo
     Assignee: (unassigned) => Dan Smith (danms)

** Tags removed: kilo-backport-potential

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1446082

Title:
  Instance without "extra" data crashes nova-compute

Status in OpenStack Compute (Nova):
  Fix Committed
Status in OpenStack Compute (nova) kilo series:
  In Progress

Bug description:
  I'm upgrading from Icehouse to Kilo. I have a single instance that was
  created in Icehouse. After the upgrade, nova-compute crashes because
  it's looking for instance "extra" data that is not there.

  To fix this, we need to check if there is any "extra" data for the
  instance before trying to read properties such as "numa_topology".

  # dpkg -l | grep nova
  ii  nova-common                         1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - common files
  ii  nova-compute                        1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                    1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - compute node libvirt support
  ii  python-nova                         1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute Python libraries
  ii  python-novaclient                   1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

  nova-compute.log:
  2015-04-20 17:35:09.214 15508 DEBUG oslo_concurrency.lockutils [req-43d3110a-cac7-425e-842c-f725bda91c10 - - - - -] Lock "compute_resources" acquired by "_update_available_resource" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:444
  2015-04-20 17:35:09.299 15508 DEBUG oslo_concurrency.lockutils [req-43d3110a-cac7-425e-842c-f725bda91c10 - - - - -] Lock "compute_resources" released by "_update_available_resource" :: held 0.085s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
      timer()
    File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
      cb(*args, **kw)
    File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
      result = function(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 497, in run_service
      service.start()
    File "/usr/lib/python2.7/dist-packages/nova/service.py", line 183, in start
      self.manager.pre_start_hook()
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1287, in pre_start_hook
      self.update_available_resource(nova.context.get_admin_context())
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6236, in update_available_resource
      rt.update_available_resource(context)
    File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 402, in update_available_resource
      self._update_available_resource(context, resources)
    File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
      return f(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 436, in _update_available_resource
      'numa_topology'])
    File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in wrapper
      result = fn(cls, context, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1152, in get_by_host_and_node
      expected_attrs)
    File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1068, in _make_instance_list
      expected_attrs=expected_attrs)
    File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 501, in _from_db_object
      db_inst.get('extra').get('numa_topology'))
  AttributeError: 'NoneType' object has no attribute 'get'
  2015-04-20 17:35:09.301 15508 ERROR nova.openstack.common.threadgroup [req-12483464-12a6-4b74-a671-bc6bb943b265 - - - - -] 'NoneType' object has no attribute 'get'
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 145, in wait
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     x.wait()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 47, in wait
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 497, in run_service
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     service.start()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 183, in start
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1287, in pre_start_hook
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6236, in update_available_resource
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     rt.update_available_resource(context)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 402, in update_available_resource
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     self._update_available_resource(context, resources)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 436, in _update_available_resource
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     'numa_topology'])
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in wrapper
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     result = fn(cls, context, *args, **kwargs)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1152, in get_by_host_and_node
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     expected_attrs)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1068, in _make_instance_list
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     expected_attrs=expected_attrs)
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 501, in _from_db_object
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup     db_inst.get('extra').get('numa_topology'))
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup AttributeError: 'NoneType' object has no attribute 'get'
  2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1446082/+subscriptions
References

[Bug 1446082] [NEW] Instance without "extra" data crashes nova-compute
From: Christoph Dwertmann, 2015-04-20