yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1158503] Re: rpc timeout error when running the nova

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jason <1158503@xxxxxxxxxxxxxxxxxx>
Date: Wed, 08 May 2013 22:21:03 -0000
Reply-to: Bug 1158503 <1158503@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Hi Thierry, thank you very much!

It did not happen again once I checked out the latest nova source code,
I believe some dependency problems caused this before.

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1158503

Title:
  rpc timeout error when running the nova

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  running nova by nova-all command in virtual environment, got the
  errors:

  2013-03-21 13:10:22.330 15888 ERROR nova.openstack.common.rpc.common [-] Timed out waiting for RPC response: timed out
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common   File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 563, in ensure
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common     return method(*args, **kwargs)
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common   File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 643, in _consume
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common     return self.connection.drain_events(timeout=timeout)
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common   File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 110, in drain_events
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common     return self.transport.drain_events(self.connection, **kwargs)
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common   File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200, in drain_events
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common     return connection.drain_events(**kwargs)
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common   File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50, in drain_events
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common     return self.wait_multi(self.channels.values(), timeout=timeout)
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common   File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56, in wait_multi
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common     chanmap.keys(), allowed_methods, timeout=timeout)
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common   File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104, in _wait_multiple
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common     channel, method_sig, args, content = self.read_timeout(timeout)
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common   File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86, in read_timeout
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common     return self.method_reader.read_method()
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common   File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 215, in read_method
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common     raise m
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common timeout: timed out
  2013-03-21 13:10:22.330 15888 TRACE nova.openstack.common.rpc.common
  2013-03-21 13:10:22.342 15888 ERROR nova.service [-] Unhandled exception
  2013-03-21 13:10:22.342 15888 TRACE nova.service Traceback (most recent call last):
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/service.py", line 307, in _start_child
  2013-03-21 13:10:22.342 15888 TRACE nova.service     self._child_process(wrap.server)
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/service.py", line 284, in _child_process
  2013-03-21 13:10:22.342 15888 TRACE nova.service     launcher.run_server(server)
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/service.py", line 147, in run_server
  2013-03-21 13:10:22.342 15888 TRACE nova.service     server.start()
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/service.py", line 429, in start
  2013-03-21 13:10:22.342 15888 TRACE nova.service     self.manager.init_host()
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/compute/manager.py", line 561, in init_host
  2013-03-21 13:10:22.342 15888 TRACE nova.service     self.host)
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/conductor/api.py", line 396, in instance_get_all_by_host
  2013-03-21 13:10:22.342 15888 TRACE nova.service     return self.conductor_rpcapi.instance_get_all_by_host(context, host)
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 309, in instance_get_all_by_host
  2013-03-21 13:10:22.342 15888 TRACE nova.service     return self.call(context, msg, version='1.32')
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/openstack/common/rpc/proxy.py", line 80, in call
  2013-03-21 13:10:22.342 15888 TRACE nova.service     return rpc.call(context, self._get_topic(topic), msg, timeout)
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/openstack/common/rpc/__init__.py", line 140, in call
  2013-03-21 13:10:22.342 15888 TRACE nova.service     return _get_impl().call(CONF, context, topic, msg, timeout)
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 798, in call
  2013-03-21 13:10:22.342 15888 TRACE nova.service     rpc_amqp.get_connection_pool(conf, Connection))
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 569, in call
  2013-03-21 13:10:22.342 15888 TRACE nova.service     rv = list(rv)
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 512, in __iter__
  2013-03-21 13:10:22.342 15888 TRACE nova.service     self.done()
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-03-21 13:10:22.342 15888 TRACE nova.service     self.gen.next()
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 509, in __iter__
  2013-03-21 13:10:22.342 15888 TRACE nova.service     self._iterator.next()
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 648, in iterconsume
  2013-03-21 13:10:22.342 15888 TRACE nova.service     yield self.ensure(_error_callback, _consume)
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 566, in ensure
  2013-03-21 13:10:22.342 15888 TRACE nova.service     error_callback(e)
  2013-03-21 13:10:22.342 15888 TRACE nova.service   File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 629, in _error_callback
  2013-03-21 13:10:22.342 15888 TRACE nova.service     raise rpc_common.Timeout()
  2013-03-21 13:10:22.342 15888 TRACE nova.service Timeout: Timeout while waiting on RPC response.
  2013-03-21 13:10:22.342 15888 TRACE nova.service
  2013-03-21 13:10:22.368 15715 INFO nova.service [-] Child 15888 exited with status 2
  2013-03-21 13:10:22.380 15715 INFO nova.service [-] Started child 15911

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1158503/+subscriptions