yahoo-eng-team team mailing list archive

[Tiantian Gao <gtt116@xxxxxxxxx>]

Public bug reported:

If exceptions raise during neutron-server initiation, will result in
lots of fanout queues leak in RabbitMQ. Since these fanout queues will
continue to receive message, the size of  it will grow infinitely until
RabbitMQ refuse to accept new message.

Reproduction
==============

In file neutron/plugins/openvswitch/agent/ovs_neutron_agent.py :

class OVSNeutronAgent(.....):
    def __init__(self, integ_br, tun_br, local_ip......):
        ....
        ....
        self.setup_rpc()
        ....
        raise Exception("boo")   ## <<< insert a fake exception here
        ......
        self.connection.consume_in_thread()

Start neutron-openvswitch-agent server, it will quit because the
exception.

check out the RabbitMQ queues:

$ sudo rabbitmqctl list_queues|grep fanout

q-agent-notifier-l2population-update_fanout_db311643213548ff95d2c044418c6d90    0
q-agent-notifier-network-delete_fanout_37e3d19330404548870d60128196e73b 0
q-agent-notifier-port-update_fanout_a8eb4b571182445181097a9bd02a4fc0    0
q-agent-notifier-security_group-update_fanout_b376f030b88844089b57e87849113399  0
q-agent-notifier-tunnel-update_fanout_6c46338f0d1a4d08bfe777427c2c5c08  0

Although we are using fake exception in this case, in real world, there
are many cases that will raise exception between self.setup_rpc() and
consume_in_thread(). Even neutron-openvswitch-agent recuperated, these
fanout queue will continue receive messages that from neutron-server,
result in certain infinitely growing queue there.

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

  If exceptions raise during neutron-server initiation, will result in
  lots of fanout queues leak in RabbitMQ. Since these fanout queues will
  continue to receive message, the size of  it will grow infinitely until
  RabbitMQ refuse to accept new message.
  
  Reproduction
  ==============
  
  In file neutron/plugins/openvswitch/agent/ovs_neutron_agent.py :
  
- class OVSNeutronAgent(.....):  
-     def __init__(self, integ_br, tun_br, local_ip......):
-         ....
-         ....
-         self.setup_rpc()
-         ....
-         raise Exception("boo")   ## <<< insert a fake exception here
-         ......
-         self.connection.consume_in_thread()
+ class OVSNeutronAgent(.....):
+     def __init__(self, integ_br, tun_br, local_ip......):
+         ....
+         ....
+         self.setup_rpc()
+         ....
+         raise Exception("boo")   ## <<< insert a fake exception here
+         ......
+         self.connection.consume_in_thread()
  
- 
- Start neutron-openvswitch-agent server, it will quite because the exception.
+ Start neutron-openvswitch-agent server, it will quit because the
+ exception.
  
  check out the RabbitMQ queues:
  
  $ sudo rabbitmqctl list_queues|grep fanout
  
  q-agent-notifier-l2population-update_fanout_db311643213548ff95d2c044418c6d90    0
  q-agent-notifier-network-delete_fanout_37e3d19330404548870d60128196e73b 0
  q-agent-notifier-port-update_fanout_a8eb4b571182445181097a9bd02a4fc0    0
  q-agent-notifier-security_group-update_fanout_b376f030b88844089b57e87849113399  0
  q-agent-notifier-tunnel-update_fanout_6c46338f0d1a4d08bfe777427c2c5c08  0
  
- 
- Although we are using fake exception in this case, in real world, there are many cases that will raise exception between self.setup_rpc() and consume_in_thread(). Even neutron-openvswitch-agent recuperated, these fanout queue will continue receive messages that from neutron-server, result in certain infinitely growing queue there.
+ Although we are using fake exception in this case, in real world, there
+ are many cases that will raise exception between self.setup_rpc() and
+ consume_in_thread(). Even neutron-openvswitch-agent recuperated, these
+ fanout queue will continue receive messages that from neutron-server,
+ result in certain infinitely growing queue there.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1434378

Title:
  openvswitch-agent leak fanout queues

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  If exceptions raise during neutron-server initiation, will result in
  lots of fanout queues leak in RabbitMQ. Since these fanout queues will
  continue to receive message, the size of  it will grow infinitely
  until RabbitMQ refuse to accept new message.

  Reproduction
  ==============

  In file neutron/plugins/openvswitch/agent/ovs_neutron_agent.py :

  class OVSNeutronAgent(.....):
      def __init__(self, integ_br, tun_br, local_ip......):
          ....
          ....
          self.setup_rpc()
          ....
          raise Exception("boo")   ## <<< insert a fake exception here
          ......
          self.connection.consume_in_thread()

  Start neutron-openvswitch-agent server, it will quit because the
  exception.

  check out the RabbitMQ queues:

  $ sudo rabbitmqctl list_queues|grep fanout

  q-agent-notifier-l2population-update_fanout_db311643213548ff95d2c044418c6d90    0
  q-agent-notifier-network-delete_fanout_37e3d19330404548870d60128196e73b 0
  q-agent-notifier-port-update_fanout_a8eb4b571182445181097a9bd02a4fc0    0
  q-agent-notifier-security_group-update_fanout_b376f030b88844089b57e87849113399  0
  q-agent-notifier-tunnel-update_fanout_6c46338f0d1a4d08bfe777427c2c5c08  0

  Although we are using fake exception in this case, in real world,
  there are many cases that will raise exception between
  self.setup_rpc() and consume_in_thread(). Even neutron-openvswitch-
  agent recuperated, these fanout queue will continue receive messages
  that from neutron-server, result in certain infinitely growing queue
  there.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1434378/+subscriptions