yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1498472] Re: lbaas:after 871 namespace was created for 1v1 mapping, the new LB pool with pending_create status

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: spark <spark.liu@xxxxxxxxx>
Date: Mon, 19 Oct 2015 07:50:00 -0000
Reply-to: Bug 1498472 <1498472@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
don;t see this issue on kilo setup , so close it

** Changed in: neutron
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1498472

Title:
  lbaas:after 871  namespace  was created for  1v1 mapping, the new LB
  pool with pending_create status

Status in neutron:
  Invalid

Bug description:
  after 817 tenants was create for namespace 1v1 maping, the new LB pool
  with pending_creat status

  test step
  1 create  a tenant
  2 create a pool network with subnet 10.1.1.0
    create a vip network with subnet 20.1.1.0
  3 create a router and add 2 interface on it for subnet 10 and subnet20
  4 create a LB pool with subnet 10
  5 add a VIP with vip address 20.1.1.100 on this pool with protocol HTTP and port 80
  6(option ) add one or two member
  7 repeat above step to continuous add 1v1 map( 1 tenant map 1 lb namespce) for scale test

  add to following tenant
  80 tenants  with 2 backend and 1 client (LB monitor enable )with simple traffic
  100 tenants  with 1 backend and 1 client (LB monitor enable) with simple traffic
  1067 tenants without member and monitor
  total of 1247  pools , total of 817 pools with active , total of 376 pools with error or pending-create status ) log in attachment
  the issue :after add 817 namespace , created new lb pool with pending-status or error status

  after check log,  AMQP can't connect to server ,
  and then reset rabbitmq service on controller node " service rabbitmq-server restart"
  the setup can't recover , the new lb pool still with pending-create status

  for recover setup, start to delete tenants and lb-pool and vip ( 1v1 maping)
  after delete tenant to 328(remain), the setup recover , new lb pool can be create with active status , so it was a scale issue , and file this bug for trace this issue

  log

  [root@nsj17 ~]# tail -n 200 /var/log/neutron/lbaas-agent.log
  2015-08-11 20:35:51.939 2581 TRACE oslo.messaging._drivers.impl_rabbit
  2015-08-11 20:35:51.944 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-08-11 20:35:51.970 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit     return method()
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit     return self.connection.drain_events(timeout=timeout)
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit     return self.transport.drain_events(self.connection, **kwargs)
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit     return connection.drain_events(**kwargs)
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(channel, args)
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit     (class_id, method_id), ConnectionError)
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
  2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit
  2015-08-11 20:35:51.972 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-08-11 20:35:52.946 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
  2015-08-11 20:35:52.971 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
  2015-08-11 20:35:52.973 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
  2015-08-11 20:35:52.993 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
  2015-08-11 20:35:53.990 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-08-11 20:35:53.999 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-08-11 20:35:55.000 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
  2015-08-11 20:35:55.023 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
  2015-08-11 20:35:55.060 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672
  2015-08-11 20:35:55.083 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672
  2015-08-11 20:35:57.681 2581 WARNING oslo.messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : ed3ca0b705914e4b9e0542fd1cdc3eda, message : {u'_unique_id': u'4b941437471a4de99f0871e34216a70a', u'failure': None, u'result': None}
  2015-08-11 20:35:57.683 2581 WARNING oslo.messaging._drivers.amqpdriver [-] _queues: {'060c2e4d42f04e31be500647207d3a47': <Queue at 0x3a6e290 maxsize=None getters[1]>, '78db36a255ae4b20b090554216923a20': <Queue at 0x3c2e5d0 maxsize=None>}
  2015-08-11 20:35:57.737 2581 WARNING oslo.messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : ed3ca0b705914e4b9e0542fd1cdc3eda, message : {u'_unique_id': u'e9bfc396a78c49a2993f6db149711c17', u'failure': None, u'result': None, u'ending': True}
  2015-08-11 20:35:57.738 2581 WARNING oslo.messaging._drivers.amqpdriver [-] _queues: {'060c2e4d42f04e31be500647207d3a47': <Queue at 0x3a6e290 maxsize=None getters[1] tasks=1 _cond=<Event at 0x3b09050 result=NOT_USED _exc=None _waiters[0]>>, '78db36a255ae4b20b090554216923a20': <Queue at 0x3c2e5d0 maxsize=None>}
  2015-08-11 20:36:14.044 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'n-lbaas-plugin': [Errno 32] Broken pipe
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     return method()
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 752, in _publish
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic=topic, **kwargs)
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 378, in __init__
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     **options)
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 214, in revive
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 100, in declare
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 163, in declare
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.channel_id, allowed_methods)
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 270, in _wait_method
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.wait()
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 523, in _close
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     self._x_close_ok()
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     self._send_method((10, 51))
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.channel_id, method_sig, args, content,
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     write_frame(1, channel, payload)
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 177, in write_frame
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     frame_type, channel, size, payload, 0xce,
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     tail = self.send(data, flags)
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit     total_sent += fd.send(data[total_sent:], flags)
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe
  2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
  2015-08-11 20:36:14.050 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-08-11 20:36:15.052 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
  2015-08-11 20:36:15.080 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672
  2015-08-11 20:36:57.747 2581 ERROR neutron.services.loadbalancer.agent.agent_manager [-] Unable to deploy instance for pool: 92fb078d-f203-4972-a555-e2ba033d50d8
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager Traceback (most recent call last):
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_manager.py", line 169, in _reload_pool
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager     logical_config = self.plugin_rpc.get_logical_device(pool_id)
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_api.py", line 57, in get_logical_device
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager     host=self.host
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron/common/log.py", line 34, in wrapper
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager     return method(*args, **kwargs)
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 161, in call
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager     context, msg, rpc_method='call', **kwargs)
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 187, in __call_rpc_method
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager     return func(context, msg['method'], **msg['args'])
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 389, in call
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager     return self.prepare().call(ctxt, method, **kwargs)
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager     retry=self.retry)
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager     timeout=timeout, retry=retry)
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager     retry=retry)
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager     result = self._waiter.wait(msg_id, timeout)
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager     reply, ending = self._poll_connection(msg_id, timeout)
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager     % msg_id)
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID 78db36a255ae4b20b090554216923a20
  2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager
  2015-08-11 20:37:15.791 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit     return method()
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit     return self.connection.drain_events(timeout=timeout)
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit     return self.transport.drain_events(self.connection, **kwargs)
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit     return connection.drain_events(**kwargs)
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(channel, args)
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit     (class_id, method_id), ConnectionError)
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
  2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit
  2015-08-11 20:37:15.795 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-08-11 20:37:16.796 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
  2015-08-11 20:37:16.815 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
  2015-08-11 20:37:17.832 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
  2015-08-11 20:37:18.855 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
  2015-08-11 20:37:18.903 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672
  2015-08-11 20:37:23.258 2581 ERROR oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Failed to publish message to topic 'n-lbaas-plugin': [Errno 32] Broken pipe
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     return method()
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 752, in _publish
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic=topic, **kwargs)
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 378, in __init__
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     **options)
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 214, in revive
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 100, in declare
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 163, in declare
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     (40, 11),  # Channel.exchange_declare_ok
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.channel_id, allowed_methods)
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 270, in _wait_method
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.wait()
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     return self.dispatch_method(method_sig, args, content)
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(self, args)
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 523, in _close
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     self._x_close_ok()
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     self._send_method((10, 51))
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     self.channel_id, method_sig, args, content,
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     write_frame(1, channel, payload)
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 177, in write_frame
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     frame_type, channel, size, payload, 0xce,
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     tail = self.send(data, flags)
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit     total_sent += fd.send(data[total_sent:], flags)
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe
  2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
  2015-08-11 20:37:23.266 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Delaying reconnect for 1.0 seconds...
  2015-08-11 20:37:24.277 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connecting to AMQP server on 10.53.87.192:5672
  2015-08-11 20:37:24.337 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connected to AMQP server on 10.53.87.192:5672
  2015-08-11 20:37:24.344 2581 ERROR oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit     return method()
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit     return self.connection.drain_events(timeout=timeout)
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit     return self.transport.drain_events(self.connection, **kwargs)
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit     return connection.drain_events(**kwargs)
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit     return amqp_method(channel, args)
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit     (class_id, method_id), ConnectionError)
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
  2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit
  2015-08-11 20:37:24.346 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Delaying reconnect for 1.0 seconds...
  2015-08-11 20:37:25.347 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connecting to AMQP server on 10.53.87.192:5672
  2015-08-11 20:37:25.387 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connected to AMQP server on 10.53.87.192:5672
  [root@nsj17 ~]#

  Notes Log:
  IBM     Licensed Material - Property of IBM ? Cop

  correct the number
  180 with traffic
  690 without traffic
  Total Active LB Namespace = 870
  Total LB Namespace = 1247
  Inactive = 377

  [root@nsj17 ~]# sar -P ALL 60 1
  Linux 3.10.0-229.7.2.el7.x86_64 (nsj17)         08/12/2015      _x86_64_        (80 CPU)

  07:28:50 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
  07:29:50 PM     all      2.45      0.00     72.87      0.00      0.00     24.68
  07:29:50 PM       0      3.09      0.00     86.10      0.00      0.00     10.81
  07:29:50 PM       1      3.24      0.00     83.92      0.00      0.00     12.84
  07:29:50 PM       2      2.88      0.00     77.57      0.00      0.00     19.54
  07:29:50 PM       3      2.16      0.00     75.61      0.00      0.00     22.23
  07:29:50 PM       4      2.37      0.00     76.48      0.00      0.00     21.15
  07:29:50 PM       5      2.44      0.00     75.88      0.00      0.00     21.69
  07:29:50 PM       6      2.23      0.00     78.28      0.00      0.00     19.48

  
  Average:        CPU     %user     %nice   %system   %iowait    %steal     %idle
  Average:        all      2.45      0.00     72.87      0.00      0.00     24.68
  Average:          0      3.09      0.00     86.10      0.00      0.00     10.81
  Average:          1      3.24      0.00     83.92      0.00      0.00     12.84
  Average:          2      2.88      0.00     77.57      0.00      0.00     19.54
  Average:          3      2.16      0.00     75.61      0.00      0.00     22.23
  Average:          4      2.37      0.00     76.48      0.00      0.00     21.15
  Average:          5      2.44      0.00     75.88      0.00      0.00     21.69
  Average:          6      2.23      0.00     78.28      0.00      0.00     19.48
  Average:          7      2.49      0.00     72.97      0.00      0.00     24.54
  Average:          8      2.63      0.00     75.52      0.00      0.00     21.84
  Average:          9      2.95      0.00     71.80      0.00      0.00     25.25
  Average:         10      2.23      0.00     76.04      0.00      0.00     21.73

  [root@nsj17 ~]#  free
                total        used        free      shared  buff/cache   available
  Mem:      263961496    19596608   220503484     4236472    23861404   231497040
  Swap:       1048572           0     1048572
  [root@nsj17 ~]#

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1498472/+subscriptions
References

[Bug 1498472] [NEW] after 871 namespace was created for 1v1 maping, the new LB pool with pending_creat status
From: spark, 2015-09-22