← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #24551

[Bug 1393391] [NEW] neutron-openvswitch-agent stuck on no queue 'q-agent-notifier-port-update_fanout..

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Under some conditions -TBD-, (port being created, and port being deleted?), neutron-openvswitch-agent can get stuck 
trying to subscribe a fanout queue, 

It stops responding to any other messages, so it stops effectively
working at all.


2014-11-17 06:15:55.842 3042 ERROR root [-] Unexpected exception occurred 61 time(s)... retrying.
2014-11-17 06:15:55.842 3042 TRACE root Traceback (most recent call last):
2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 92, in inner_func
2014-11-17 06:15:55.842 3042 TRACE root     return infunc(*args, **kwargs)
2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 746, in _consumer_thread
2014-11-17 06:15:55.842 3042 TRACE root     self.consume()
2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 737, in consume
2014-11-17 06:15:55.842 3042 TRACE root     six.next(it)
2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 664, in iterconsume
2014-11-17 06:15:55.842 3042 TRACE root     yield self.ensure(_error_callback, _consume)
2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 579, in ensure
2014-11-17 06:15:55.842 3042 TRACE root     return method(*args, **kwargs)
2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 657, in _consume
2014-11-17 06:15:55.842 3042 TRACE root     queues_tail.consume(nowait=False)
2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 190, in consume
2014-11-17 06:15:55.842 3042 TRACE root     self.queue.consume(*args, callback=_callback, **options)
2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 598, in consume
2014-11-17 06:15:55.842 3042 TRACE root     nowait=nowait)
2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1769, in basic_consume
2014-11-17 06:15:55.842 3042 TRACE root     (60, 21),  # Channel.basic_consume_ok
2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 71, in wait
2014-11-17 06:15:55.842 3042 TRACE root     return self.dispatch_method(method_sig, args, content)
2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 88, in dispatch_method
2014-11-17 06:15:55.842 3042 TRACE root     return amqp_method(self, args)
2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 224, in _close
2014-11-17 06:15:55.842 3042 TRACE root     raise ChannelError(reply_code, reply_text, (class_id, method_id))
2014-11-17 06:15:55.842 3042 TRACE root ChannelError: 404: (NOT_FOUND - no queue 'q-agent-notifier-port-update_fanout_c5de1f41f554475492301e03557cdd66' in vhost '/', (60, 20), None)
2014-11-17 06:15:55.842 3042 TRACE root
s /var/log/neutron/os /var/log/neutron/openvswitch-agent.log
[root@mrg-03 ~]# tail /var/log/neutron/openvswitch-agent.log &2014-11-17 06:16:56.085 3042 ERROR root [-] Unexpected exception occurred 60 time(s)... retrying.
2014-11-17 06:16:56.085 3042 TRACE root Traceback (most recent call last):
2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 92, in inner_func
2014-11-17 06:16:56.085 3042 TRACE root     return infunc(*args, **kwargs)
2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 746, in _consumer_thread
2014-11-17 06:16:56.085 3042 TRACE root     self.consume()
2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 737, in consume
2014-11-17 06:16:56.085 3042 TRACE root     six.next(it)
2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 664, in iterconsume
2014-11-17 06:16:56.085 3042 TRACE root     yield self.ensure(_error_callback, _consume)
2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 579, in ensure
2014-11-17 06:16:56.085 3042 TRACE root     return method(*args, **kwargs)
2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 657, in _consume
2014-11-17 06:16:56.085 3042 TRACE root     queues_tail.consume(nowait=False)
2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 190, in consume
2014-11-17 06:16:56.085 3042 TRACE root     self.queue.consume(*args, callback=_callback, **options)
2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 598, in consume
2014-11-17 06:16:56.085 3042 TRACE root     nowait=nowait)
2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1769, in basic_consume
2014-11-17 06:16:56.085 3042 TRACE root     (60, 21),  # Channel.basic_consume_ok
2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 71, in wait
2014-11-17 06:16:56.085 3042 TRACE root     return self.dispatch_method(method_sig, args, content)
2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 88, in dispatch_method
2014-11-17 06:16:56.085 3042 TRACE root     return amqp_method(self, args)
2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 224, in _close
2014-11-17 06:16:56.085 3042 TRACE root     raise ChannelError(reply_code, reply_text, (class_id, method_id))
2014-11-17 06:16:56.085 3042 TRACE root ChannelError: 404: (NOT_FOUND - no queue 'q-agent-notifier-port-update_fanout_c5de1f41f554475492301e03557cdd66' in vhost '/', (60, 20), None)
2014-11-17 06:16:56.085 3042 TRACE root

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1393391

Title:
  neutron-openvswitch-agent stuck on no queue 'q-agent-notifier-port-
  update_fanout..

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  Under some conditions -TBD-, (port being created, and port being deleted?), neutron-openvswitch-agent can get stuck 
  trying to subscribe a fanout queue, 

  It stops responding to any other messages, so it stops effectively
  working at all.

  
  2014-11-17 06:15:55.842 3042 ERROR root [-] Unexpected exception occurred 61 time(s)... retrying.
  2014-11-17 06:15:55.842 3042 TRACE root Traceback (most recent call last):
  2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 92, in inner_func
  2014-11-17 06:15:55.842 3042 TRACE root     return infunc(*args, **kwargs)
  2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 746, in _consumer_thread
  2014-11-17 06:15:55.842 3042 TRACE root     self.consume()
  2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 737, in consume
  2014-11-17 06:15:55.842 3042 TRACE root     six.next(it)
  2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 664, in iterconsume
  2014-11-17 06:15:55.842 3042 TRACE root     yield self.ensure(_error_callback, _consume)
  2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 579, in ensure
  2014-11-17 06:15:55.842 3042 TRACE root     return method(*args, **kwargs)
  2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 657, in _consume
  2014-11-17 06:15:55.842 3042 TRACE root     queues_tail.consume(nowait=False)
  2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 190, in consume
  2014-11-17 06:15:55.842 3042 TRACE root     self.queue.consume(*args, callback=_callback, **options)
  2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 598, in consume
  2014-11-17 06:15:55.842 3042 TRACE root     nowait=nowait)
  2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1769, in basic_consume
  2014-11-17 06:15:55.842 3042 TRACE root     (60, 21),  # Channel.basic_consume_ok
  2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 71, in wait
  2014-11-17 06:15:55.842 3042 TRACE root     return self.dispatch_method(method_sig, args, content)
  2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 88, in dispatch_method
  2014-11-17 06:15:55.842 3042 TRACE root     return amqp_method(self, args)
  2014-11-17 06:15:55.842 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 224, in _close
  2014-11-17 06:15:55.842 3042 TRACE root     raise ChannelError(reply_code, reply_text, (class_id, method_id))
  2014-11-17 06:15:55.842 3042 TRACE root ChannelError: 404: (NOT_FOUND - no queue 'q-agent-notifier-port-update_fanout_c5de1f41f554475492301e03557cdd66' in vhost '/', (60, 20), None)
  2014-11-17 06:15:55.842 3042 TRACE root
  s /var/log/neutron/os /var/log/neutron/openvswitch-agent.log
  [root@mrg-03 ~]# tail /var/log/neutron/openvswitch-agent.log &2014-11-17 06:16:56.085 3042 ERROR root [-] Unexpected exception occurred 60 time(s)... retrying.
  2014-11-17 06:16:56.085 3042 TRACE root Traceback (most recent call last):
  2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 92, in inner_func
  2014-11-17 06:16:56.085 3042 TRACE root     return infunc(*args, **kwargs)
  2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 746, in _consumer_thread
  2014-11-17 06:16:56.085 3042 TRACE root     self.consume()
  2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 737, in consume
  2014-11-17 06:16:56.085 3042 TRACE root     six.next(it)
  2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 664, in iterconsume
  2014-11-17 06:16:56.085 3042 TRACE root     yield self.ensure(_error_callback, _consume)
  2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 579, in ensure
  2014-11-17 06:16:56.085 3042 TRACE root     return method(*args, **kwargs)
  2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 657, in _consume
  2014-11-17 06:16:56.085 3042 TRACE root     queues_tail.consume(nowait=False)
  2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 190, in consume
  2014-11-17 06:16:56.085 3042 TRACE root     self.queue.consume(*args, callback=_callback, **options)
  2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 598, in consume
  2014-11-17 06:16:56.085 3042 TRACE root     nowait=nowait)
  2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1769, in basic_consume
  2014-11-17 06:16:56.085 3042 TRACE root     (60, 21),  # Channel.basic_consume_ok
  2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 71, in wait
  2014-11-17 06:16:56.085 3042 TRACE root     return self.dispatch_method(method_sig, args, content)
  2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 88, in dispatch_method
  2014-11-17 06:16:56.085 3042 TRACE root     return amqp_method(self, args)
  2014-11-17 06:16:56.085 3042 TRACE root   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 224, in _close
  2014-11-17 06:16:56.085 3042 TRACE root     raise ChannelError(reply_code, reply_text, (class_id, method_id))
  2014-11-17 06:16:56.085 3042 TRACE root ChannelError: 404: (NOT_FOUND - no queue 'q-agent-notifier-port-update_fanout_c5de1f41f554475492301e03557cdd66' in vhost '/', (60, 20), None)
  2014-11-17 06:16:56.085 3042 TRACE root

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1393391/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next