yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1154232] Re: Nova Compute generates stack traces on startup trying to talk to conductor

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Russell Bryant <1154232@xxxxxxxxxxxxxxxxxx>
Date: Tue, 12 Mar 2013 18:28:12 -0000
Reply-to: Bug 1154232 <1154232@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Released => In Progress

** Changed in: nova
     Assignee: (unassigned) => Russell Bryant (russellb)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1154232

Title:
  Nova Compute generates stack traces on startup trying to talk to
  conductor

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  Every single startup of nova-compute in any tempest run starts with
  the following stack trace in the logs:

  2013-03-08 17:01:57.139 ERROR nova.openstack.common.rpc.common [req-76037107-4c47-47f7-b9cd-7b3e9e7e1650 None None] Timed out waiting for RPC response: timed out
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common   File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 563, in ensure
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common     return method(*args, **kwargs)
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common   File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 643, in _consume
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common     return self.connection.drain_events(timeout=timeout)
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common     return self.transport.drain_events(self.connection, **kwargs)
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common     return connection.drain_events(**kwargs)
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common     return self.wait_multi(self.channels.values(), timeout=timeout)
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common     chanmap.keys(), allowed_methods, timeout=timeout)
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common     channel, method_sig, args, content = read_timeout(timeout)
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common     return self.method_reader.read_method()
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common     raise m
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common timeout: timed out
  2013-03-08 17:01:57.139 11470 TRACE nova.openstack.common.rpc.common 
  2013-03-08 17:01:57.148 WARNING nova.conductor.api [req-76037107-4c47-47f7-b9cd-7b3e9e7e1650 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor?
  2013-03-08 17:01:57.148 DEBUG nova.openstack.common.rpc.amqp [req-76037107-4c47-47f7-b9cd-7b3e9e7e1650 None None] Making synchronous call on conductor ... multicall /opt/stack/new/nova/nova/openstack/common/rpc/amqp.py:541
  2013-03-08 17:01:57.149 DEBUG nova.openstack.common.rpc.amqp [req-76037107-4c47-47f7-b9cd-7b3e9e7e1650 None None] MSG_ID is 334393ad8374485ab0350037a2f09ecb multicall /opt/stack/new/nova/nova/openstack/common/rpc/amqp.py:544

  
  If this is normal behavior we should suppress this stack trace, because it's a big ugly red herring in the code.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1154232/+subscriptions