yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1586979] [NEW] AMQP 2.0 prevents services from starting

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jay Jahns <jjahns@xxxxxxxxxx>
Date: Mon, 30 May 2016 09:42:57 -0000
Reply-to: Bug 1586979 <1586979@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

We are running nova from the stable/mitaka branch (not using packages).

Our build process has flagged this.  Previously, when we've built, we've
used the requirements.txt file to pull in dependencies, and one of the
dependencies pulls in AMQP.

Initial investigation leads us to believe that oslo.messaging pulls in
AMQP as a dependency.  It looks to be pulling in the latest version
(2.0).  Our previous successful builds show AMQP of 1.4.9, which is
fully functional.

2.0 completely breaks, for example, the nova-compute service with the
following trace:

2016-05-30 01:04:58.192 11335 CRITICAL nova [req-992ce9e3-fab8-47f9-8879-36b42594fae8 - - - - -] AttributeError: 'Connection' object has no attribute '_frame_writer'
2016-05-30 01:04:58.192 11335 ERROR nova Traceback (most recent call last):
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/bin/nova-compute", line 10, in <module>
2016-05-30 01:04:58.192 11335 ERROR nova     sys.exit(main())
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/cmd/compute.py", line 73, in main
2016-05-30 01:04:58.192 11335 ERROR nova     db_allowed=CONF.conductor.use_local)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", line 218, in create
2016-05-30 01:04:58.192 11335 ERROR nova     db_allowed=db_allowed)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", line 101, in init
2016-05-30 01:04:58.192 11335 ERROR nova     self.conductor_api.wait_until_ready(context.get_admin_context())
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/conductor/api.py", line 157, in wait_until_ready
2016-05-30 01:04:58.192 11335 ERROR nova     timeout=timeout)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/baserpc.py", line 58, in ping
2016-05-30 01:04:58.192 11335 ERROR nova     return cctxt.call(context, 'ping', arg=arg_p)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 157, in call
2016-05-30 01:04:58.192 11335 ERROR nova     retry=self.retry)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 91, in _send
2016-05-30 01:04:58.192 11335 ERROR nova     timeout=timeout, retry=retry)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 466, in send
2016-05-30 01:04:58.192 11335 ERROR nova     retry=retry)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 410, in _send
2016-05-30 01:04:58.192 11335 ERROR nova     msg.update({'_reply_q': self._get_reply_q()})
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 382, in _get_reply_q
2016-05-30 01:04:58.192 11335 ERROR nova     conn = self._get_connection(rpc_common.PURPOSE_LISTEN)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 373, in _get_connection
2016-05-30 01:04:58.192 11335 ERROR nova     purpose=purpose)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py", line 396, in init
2016-05-30 01:04:58.192 11335 ERROR nova     self.connection = connection_pool.create(purpose)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py", line 110, in create
2016-05-30 01:04:58.192 11335 ERROR nova     return self.connection_cls(self.conf, self.url, purpose)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 556, in init
2016-05-30 01:04:58.192 11335 ERROR nova     self.ensure_connection()
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 654, in ensure_connection
2016-05-30 01:04:58.192 11335 ERROR nova     self.ensure(method=lambda: self.connection.connection)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 759, in ensure
2016-05-30 01:04:58.192 11335 ERROR nova     ret, channel = autoretry_method()
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
2016-05-30 01:04:58.192 11335 ERROR nova     return fun(*args, **kwargs)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/connection.py", line 507, in call
2016-05-30 01:04:58.192 11335 ERROR nova     self.revive(create_channel())
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/connection.py", line 242, in channel
2016-05-30 01:04:58.192 11335 ERROR nova     chan = self.transport.create_channel(self.connection)
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 92, in create_channel
2016-05-30 01:04:58.192 11335 ERROR nova     return connection.channel()
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/amqp/connection.py", line 380, in channel
2016-05-30 01:04:58.192 11335 ERROR nova     channel.open()
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/amqp/channel.py", line 455, in open
2016-05-30 01:04:58.192 11335 ERROR nova     spec.Channel.Open, 's', ('',), wait=spec.Channel.OpenOk,
2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 62, in send_method
2016-05-30 01:04:58.192 11335 ERROR nova     conn._frame_writer.send((1, self.channel_id, sig, args, content))
2016-05-30 01:04:58.192 11335 ERROR nova AttributeError: 'Connection' object has no attribute '_frame_writer'
2016-05-30 01:04:58.192 11335 ERROR nova


Investigation also indicates this impacts other services.  I've isolated builds into virtualenvs to prevent external environment from being a concern.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1586979

Title:
  AMQP 2.0 prevents services from starting

Status in OpenStack Compute (nova):
  New

Bug description:
  We are running nova from the stable/mitaka branch (not using
  packages).

  Our build process has flagged this.  Previously, when we've built,
  we've used the requirements.txt file to pull in dependencies, and one
  of the dependencies pulls in AMQP.

  Initial investigation leads us to believe that oslo.messaging pulls in
  AMQP as a dependency.  It looks to be pulling in the latest version
  (2.0).  Our previous successful builds show AMQP of 1.4.9, which is
  fully functional.

  2.0 completely breaks, for example, the nova-compute service with the
  following trace:

  2016-05-30 01:04:58.192 11335 CRITICAL nova [req-992ce9e3-fab8-47f9-8879-36b42594fae8 - - - - -] AttributeError: 'Connection' object has no attribute '_frame_writer'
  2016-05-30 01:04:58.192 11335 ERROR nova Traceback (most recent call last):
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/bin/nova-compute", line 10, in <module>
  2016-05-30 01:04:58.192 11335 ERROR nova     sys.exit(main())
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/cmd/compute.py", line 73, in main
  2016-05-30 01:04:58.192 11335 ERROR nova     db_allowed=CONF.conductor.use_local)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", line 218, in create
  2016-05-30 01:04:58.192 11335 ERROR nova     db_allowed=db_allowed)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py", line 101, in init
  2016-05-30 01:04:58.192 11335 ERROR nova     self.conductor_api.wait_until_ready(context.get_admin_context())
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/conductor/api.py", line 157, in wait_until_ready
  2016-05-30 01:04:58.192 11335 ERROR nova     timeout=timeout)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/baserpc.py", line 58, in ping
  2016-05-30 01:04:58.192 11335 ERROR nova     return cctxt.call(context, 'ping', arg=arg_p)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 157, in call
  2016-05-30 01:04:58.192 11335 ERROR nova     retry=self.retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 91, in _send
  2016-05-30 01:04:58.192 11335 ERROR nova     timeout=timeout, retry=retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 466, in send
  2016-05-30 01:04:58.192 11335 ERROR nova     retry=retry)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 410, in _send
  2016-05-30 01:04:58.192 11335 ERROR nova     msg.update({'_reply_q': self._get_reply_q()})
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 382, in _get_reply_q
  2016-05-30 01:04:58.192 11335 ERROR nova     conn = self._get_connection(rpc_common.PURPOSE_LISTEN)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 373, in _get_connection
  2016-05-30 01:04:58.192 11335 ERROR nova     purpose=purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py", line 396, in init
  2016-05-30 01:04:58.192 11335 ERROR nova     self.connection = connection_pool.create(purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py", line 110, in create
  2016-05-30 01:04:58.192 11335 ERROR nova     return self.connection_cls(self.conf, self.url, purpose)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 556, in init
  2016-05-30 01:04:58.192 11335 ERROR nova     self.ensure_connection()
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 654, in ensure_connection
  2016-05-30 01:04:58.192 11335 ERROR nova     self.ensure(method=lambda: self.connection.connection)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 759, in ensure
  2016-05-30 01:04:58.192 11335 ERROR nova     ret, channel = autoretry_method()
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
  2016-05-30 01:04:58.192 11335 ERROR nova     return fun(*args, **kwargs)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/connection.py", line 507, in call
  2016-05-30 01:04:58.192 11335 ERROR nova     self.revive(create_channel())
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/connection.py", line 242, in channel
  2016-05-30 01:04:58.192 11335 ERROR nova     chan = self.transport.create_channel(self.connection)
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 92, in create_channel
  2016-05-30 01:04:58.192 11335 ERROR nova     return connection.channel()
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/amqp/connection.py", line 380, in channel
  2016-05-30 01:04:58.192 11335 ERROR nova     channel.open()
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/amqp/channel.py", line 455, in open
  2016-05-30 01:04:58.192 11335 ERROR nova     spec.Channel.Open, 's', ('',), wait=spec.Channel.OpenOk,
  2016-05-30 01:04:58.192 11335 ERROR nova   File "/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 62, in send_method
  2016-05-30 01:04:58.192 11335 ERROR nova     conn._frame_writer.send((1, self.channel_id, sig, args, content))
  2016-05-30 01:04:58.192 11335 ERROR nova AttributeError: 'Connection' object has no attribute '_frame_writer'
  2016-05-30 01:04:58.192 11335 ERROR nova

  
  Investigation also indicates this impacts other services.  I've isolated builds into virtualenvs to prevent external environment from being a concern.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1586979/+subscriptions
Follow ups

[Bug 1586979] Re: AMQP 2.0 prevents services from starting
From: Jay Jahns, 2016-05-31
[Bug 1586979] Re: AMQP 2.0 prevents services from starting
From: Fernando Diaz, 2016-05-31
[Bug 1586979] Re: AMQP 2.0 prevents services from starting
From: Jay Jahns, 2016-05-31
[Bug 1586979] Re: AMQP 2.0 prevents services from starting
From: Devananda van der Veen, 2016-05-31
[Bug 1586979] Re: AMQP 2.0 prevents services from starting
From: Jay Jahns, 2016-05-31