yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1129441] Re: nova-compute no longer work after reboot with instances running

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Thu, 21 Feb 2013 08:57:18 -0000
Reply-to: Bug 1129441 <1129441@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => grizzly-3

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1129441

Title:
  nova-compute no longer work after reboot with instances running

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  How to reproduce:

  install from the master branch. Make sure
  resume_guests_state_on_host_boot=True

  Start an instance. Once the instance is in running, reboot the compute
  node.

  After reboot, the nova-compute daemon can no longer start. Here is the
  traceback:

  2013-02-18 20:13:10.703 INFO nova.compute.manager [req-2399728a-770e-469a-98d2-018caa9cc482 None None] [instance:
   3422520b-0c31-4f99-8cb0-8c6416a5f15a] Rebooting instance after nova-compute restart.
  2013-02-18 20:13:10.703 DEBUG nova.openstack.common.rpc.amqp [req-2399728a-770e-469a-98d2-018caa9cc482 None None]
   Making synchronous call on conductor ... multicall /usr/local/lib/python2.7/dist-packages/nova/openstack/common/
  rpc/amqp.py:362
  2013-02-18 20:13:10.703 DEBUG nova.openstack.common.rpc.amqp [req-2399728a-770e-469a-98d2-018caa9cc482 None None]
   MSG_ID is 387dff66e09f4c489a53afbafb9d8906 multicall /usr/local/lib/python2.7/dist-packages/nova/openstack/commo
  n/rpc/amqp.py:365
  2013-02-18 20:13:10.779 1291 CRITICAL nova [-] string indices must be integers
  Traceback (most recent call last):

    File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
      rval = self.proxy.dispatch(ctxt, version, method, **args)

    File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
      return getattr(proxyobj, method)(ctxt, **kwargs)

    File "/usr/local/lib/python2.7/dist-packages/nova/conductor/manager.py", line 235, in block_device_mapping_get_
  all_by_instance
      context, instance['uuid'])

  TypeError: string indices must be integers
  2013-02-18 20:13:10.779 1291 TRACE nova Traceback (most recent call last):
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/bin/nova-compute", line 84, in <module>
  2013-02-18 20:13:10.779 1291 TRACE nova     service.wait()
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/service.py", line 209
  , in wait
  2013-02-18 20:13:10.779 1291 TRACE nova     super(ServiceLauncher, self).wait()
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/service.py", line 179
  , in wait
  2013-02-18 20:13:10.779 1291 TRACE nova     service.wait()
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py",
  line 168, in wait
  2013-02-18 20:13:10.779 1291 TRACE nova     return self._exit_event.wait()
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 1
  16, in wait
  2013-02-18 20:13:10.779 1291 TRACE nova     return hubs.get_hub().switch()
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", lin
  e 187, in switch
  2013-02-18 20:13:10.779 1291 TRACE nova     return self.greenlet.switch()
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py",
  line 194, in main
  2013-02-18 20:13:10.779 1291 TRACE nova     result = function(*args, **kwargs)
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/service.py", line 147
  , in run_server
  2013-02-18 20:13:10.779 1291 TRACE nova     server.start()
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/service.py", line 429
  , in start
  2013-02-18 20:13:10.779 1291 TRACE nova     self.manager.init_host()
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py",
  line 516, in init_host
  2013-02-18 20:13:10.779 1291 TRACE nova     self._init_instance(context, instance)
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py",
  line 475, in _init_instance
  2013-02-18 20:13:10.779 1291 TRACE nova     context, instance['uuid'])
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py",
  line 1039, in _get_instance_volume_block_device_info
  2013-02-18 20:13:10.779 1291 TRACE nova     bdms = self._get_instance_volume_bdms(context, instance)
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py",
  line 1023, in _get_instance_volume_bdms
  2013-02-18 20:13:10.779 1291 TRACE nova     context, instance))
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/conductor/api.py", li
  ne 535, in block_device_mapping_get_all_by_instance
  2013-02-18 20:13:10.779 1291 TRACE nova     context, instance)
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/conductor/rpcapi.py",
   line 233, in block_device_mapping_get_all_by_instance
  2013-02-18 20:13:10.779 1291 TRACE nova     return self.call(context, msg, version='1.13')
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
  proxy.py", line 80, in call
  2013-02-18 20:13:10.779 1291 TRACE nova     return rpc.call(context, self._get_topic(topic), msg, timeout)
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
  __init__.py", line 139, in call
  2013-02-18 20:13:10.779 1291 TRACE nova     return _get_impl().call(CONF, context, topic, msg, timeout)
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
  impl_kombu.py", line 772, in call
  2013-02-18 20:13:10.779 1291 TRACE nova     rpc_amqp.get_connection_pool(conf, Connection))
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
  amqp.py", line 379, in call
  2013-02-18 20:13:10.779 1291 TRACE nova     rv = list(rv)
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
  amqp.py", line 347, in __iter__
  2013-02-18 20:13:10.779 1291 TRACE nova     raise result
  2013-02-18 20:13:10.779 1291 TRACE nova TypeError: string indices must be integers
  2013-02-18 20:13:10.779 1291 TRACE nova Traceback (most recent call last):
  2013-02-18 20:13:10.779 1291 TRACE nova
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
  amqp.py", line 276, in _process_data
  2013-02-18 20:13:10.779 1291 TRACE nova     rval = self.proxy.dispatch(ctxt, version, method, **args)
  2013-02-18 20:13:10.779 1291 TRACE nova
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
  dispatcher.py", line 133, in dispatch
  2013-02-18 20:13:10.779 1291 TRACE nova     return getattr(proxyobj, method)(ctxt, **kwargs)
  2013-02-18 20:13:10.779 1291 TRACE nova
  2013-02-18 20:13:10.779 1291 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/nova/conductor/manager.py"
  , line 235, in block_device_mapping_get_all_by_instance
  2013-02-18 20:13:10.779 1291 TRACE nova     context, instance['uuid'])
  2013-02-18 20:13:10.779 1291 TRACE nova
  2013-02-18 20:13:10.779 1291 TRACE nova TypeError: string indices must be integers
  2013-02-18 20:13:10.779 1291 TRACE nova

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1129441/+subscriptions