yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1177184] Re: nova crashes on start if 503 AMQPConnectionException

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Russell Bryant <1177184@xxxxxxxxxxxxxxxxxx>
Date: Wed, 08 May 2013 15:19:23 -0000
Reply-to: Bug 1177184 <1177184@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
I looked and both the qpid and kombu drivers *attempt* to sit in a loop
and attempt to connect until successful.  Something is obviously
breaking in this process, though, and appears to be specific to the
kombu driver.

** Changed in: nova
       Status: Confirmed => Invalid

** Also affects: oslo
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1177184

Title:
  nova crashes on start if 503 AMQPConnectionException

Status in OpenStack Compute (Nova):
  Invalid
Status in Oslo - a Library of Common OpenStack Code:
  New

Bug description:
  2013-05-06 21:08:11.060 TRACE nova Traceback (most recent call last):
  2013-05-06 21:08:11.060 TRACE nova   File "/usr/local/bin/nova-compute", line 9, in <module>
  2013-05-06 21:08:11.060 TRACE nova     load_entry_point('nova==2013.2.a531.g89463d8', 'console_scripts', 'nova-compute')()
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/cmd/compute.py", line 65, in main
  2013-05-06 21:08:11.060 TRACE nova     service.wait()
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/service.py", line 690, in wait
  2013-05-06 21:08:11.060 TRACE nova     _launcher.wait()
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/service.py", line 210, in wait
  2013-05-06 21:08:11.060 TRACE nova     super(ServiceLauncher, self).wait()
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/service.py", line 180, in wait
  2013-05-06 21:08:11.060 TRACE nova     service.wait()
  2013-05-06 21:08:11.060 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
  2013-05-06 21:08:11.060 TRACE nova     return self._exit_event.wait()
  2013-05-06 21:08:11.060 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
  2013-05-06 21:08:11.060 TRACE nova     return hubs.get_hub().switch()
  2013-05-06 21:08:11.060 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
  2013-05-06 21:08:11.060 TRACE nova     return self.greenlet.switch()
  2013-05-06 21:08:11.060 TRACE nova   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
  2013-05-06 21:08:11.060 TRACE nova     result = function(*args, **kwargs)
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/service.py", line 148, in run_server
  2013-05-06 21:08:11.060 TRACE nova     server.start()
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/service.py", line 430, in start
  2013-05-06 21:08:11.060 TRACE nova     self.manager.init_host()
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/compute/manager.py", line 620, in init_host
  2013-05-06 21:08:11.060 TRACE nova     self.host)
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/conductor/api.py", line 406, in instance_get_all_by_host
  2013-05-06 21:08:11.060 TRACE nova     context, host, columns_to_join=columns_to_join)
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 310, in instance_get_all_by_host
  2013-05-06 21:08:11.060 TRACE nova     return self.call(context, msg, version='1.47')
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 86, in call
  2013-05-06 21:08:11.060 TRACE nova     return rpc.call(context, real_topic, msg, timeout)
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 140, in call
  2013-05-06 21:08:11.060 TRACE nova     return _get_impl().call(CONF, context, topic, msg, timeout)
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 798, in call
  2013-05-06 21:08:11.060 TRACE nova     rpc_amqp.get_connection_pool(conf, Connection))
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 615, in call
  2013-05-06 21:08:11.060 TRACE nova     rv = list(rv)
  2013-05-06 21:08:11.060 TRACE nova   File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 564, in __iter__
  2013-05-06 21:08:11.060 TRACE nova     raise result
  2013-05-06 21:08:11.060 TRACE nova RemoteError: Remote error: AMQPConnectionException (503, u"COMMAND_INVALID - second 'channel.open' seen", (20, 10), 'Channel.open')
  2013-05-06 21:08:11.060 TRACE nova [u'Traceback (most recent call last):\n', u'  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 439, in _process_data\n    ctxt.reply(rval, None, connection_pool=se
  lf.connection_pool)\n', u'  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 272, in reply\n    reply, failure, ending, log_failure)\n', u'  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py",
   line 250, in msg_reply\n    conn.direct_send(msg_id, rpc_common.serialize_msg(msg))\n', u'  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 152, in __exit__\n    self._done()\n', u'  File "/opt/st
  ack/nova/nova/openstack/common/rpc/amqp.py", line 141, in _done\n    self.connection.reset()\n', u'  File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 596, in reset\n    self.channel = self.con
  nection.channel()\n', u'  File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 99, in channel\n    return self.transport.create_channel(self.connection)\n', u'  File "/usr/local/lib/python2.7/d
  ist-packages/kombu/transport/pyamqplib.py", line 197, in create_channel\n    return connection.channel()\n', u'  File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 130, in channel\n
     return Channel(self, channel_id)\n', u'  File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 82, in __init__\n    self._x_open()\n', u'  File "/usr/local/lib/python2.7/dist-packag
  es/amqplib/client_0_8/channel.py", line 471, in _x_open\n    (20, 11),    # Channel.open_ok\n', u'  File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 89, in wait\n    self
  .channel_id, allowed_methods)\n', u'  File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 218, in _wait_method\n    self.wait()\n', u'  File "/usr/local/lib/python2.7/dist-package
  s/amqplib/client_0_8/abstract_channel.py", line 105, in wait\n    return amqp_method(self, args)\n', u'  File "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 367, in _close\n    ra
  ise AMQPConnectionException(reply_code, reply_text, (class_id, method_id))\n', u'AMQPConnectionException: (503, u"COMMAND_INVALID - second \'channel.open\' seen", (20, 10), \'Channel.open\')\n'].

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1177184/+subscriptions