yahoo-eng-team team mailing list archive

[Kevin Benton <1496774@xxxxxxxxxxxxxxxxxx>]

It looks like a bug in oslo.messaging. The self.channels var is set to
None somehow.

** Also affects: oslo.messaging
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1496774

Title:
  metadata-agent fills up log-files with " TypeError: 'NoneType' object
  has no attribute '__getitem__'"

Status in neutron:
  New
Status in oslo.messaging:
  New

Bug description:
  Sience Monday, the log file of metadata-agent are getting full with
  the following error / trace:

  2015-09-17 11:06:00.187 8277 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute '__getitem__'
  2015-09-17 11:06:00.187 8276 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute '__getitem__'
  2015-09-17 11:06:00.186 8267 ERROR oslo_messaging._drivers.amqpdriver [-] Failed to process incoming message, retrying...
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver Traceback (most recent call last):
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 197, in poll
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver     self.conn.consume(limit=1)
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1172, in consume
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver     six.next(it)
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1083, in iterconsume
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver     error_callback=_error_callback)
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 870, in ensure
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver     ret, channel = autoretry_method()
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 453, in _ensured
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver     return fun(*args, **kwargs)
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 520, in __call__^C
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver     self.revive(create_channel())
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 251, in channel
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver     chan = self.transport.create_channel(self.connection)^C
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in create_channel
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver     return connection.channel()
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 279, in channel
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver     return self.channels[channel_id]
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver TypeError: 'NoneType' object has no attribute '__getitem__'
  2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver

  There was no change in the configuration before, i'm not exactly sure
  why it does that, the configuration looks fine and all good.

  I have also a 100% load on the server.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1496774/+subscriptions