yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #38775
[Bug 1498472] [NEW] after 871 namespace was created for 1v1 maping, the new LB pool with pending_creat status
Public bug reported:
[Alchemy][LB_namespaceV1]after 817 tenants was create for namespace 1v1 maping, the new LB pool with pending_creat status
test step
1 create a tenant
2 create a pool network with subnet 10.1.1.0
create a vip network with subnet 20.1.1.0
3 create a router and add 2 interface on it for subnet 10 and subnet20
4 create a LB pool with subnet 10
5 add a VIP with vip address 20.1.1.100 on this pool with protocol HTTP and port 80
6(option ) add one or two member
7 repeat above step to continuous add 1v1 map( 1 tenant map 1 lb namespce) for scale test
add to following tenant
80 tenants with 2 backend and 1 client (LB monitor enable )with simple traffic
100 tenants with 1 backend and 1 client (LB monitor enable) with simple traffic
1067 tenants without member and monitor
total of 1247 pools , total of 817 pools with active , total of 376 pools with error or pending-create status ) log in attachment
the issue :after add 817 namespace , created new lb pool with pending-status or error status
after check log, AMQP can't connect to server ,
and then reset rabbitmq service on controller node " service rabbitmq-server restart"
the setup can't recover , the new lb pool still with pending-create status
for recover setup, start to delete tenants and lb-pool and vip ( 1v1 maping)
after delete tenant to 328(remain), the setup recover , new lb pool can be create with active status , so it was a scale issue , and file this bug for trace this issue
log
[root@nsj17 ~]# tail -n 200 /var/log/neutron/lbaas-agent.log
2015-08-11 20:35:51.939 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:35:51.944 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:35:51.970 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return method()
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:35:51.972 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:35:52.946 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:52.971 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-08-11 20:35:52.973 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:52.993 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-08-11 20:35:53.990 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:35:53.999 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:35:55.000 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:55.023 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:55.060 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:55.083 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:57.681 2581 WARNING oslo.messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : ed3ca0b705914e4b9e0542fd1cdc3eda, message : {u'_unique_id': u'4b941437471a4de99f0871e34216a70a', u'failure': None, u'result': None}
2015-08-11 20:35:57.683 2581 WARNING oslo.messaging._drivers.amqpdriver [-] _queues: {'060c2e4d42f04e31be500647207d3a47': <Queue at 0x3a6e290 maxsize=None getters[1]>, '78db36a255ae4b20b090554216923a20': <Queue at 0x3c2e5d0 maxsize=None>}
2015-08-11 20:35:57.737 2581 WARNING oslo.messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : ed3ca0b705914e4b9e0542fd1cdc3eda, message : {u'_unique_id': u'e9bfc396a78c49a2993f6db149711c17', u'failure': None, u'result': None, u'ending': True}
2015-08-11 20:35:57.738 2581 WARNING oslo.messaging._drivers.amqpdriver [-] _queues: {'060c2e4d42f04e31be500647207d3a47': <Queue at 0x3a6e290 maxsize=None getters[1] tasks=1 _cond=<Event at 0x3b09050 result=NOT_USED _exc=None _waiters[0]>>, '78db36a255ae4b20b090554216923a20': <Queue at 0x3c2e5d0 maxsize=None>}
2015-08-11 20:36:14.044 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'n-lbaas-plugin': [Errno 32] Broken pipe
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit return method()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 752, in _publish
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic=topic, **kwargs)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 378, in __init__
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit **options)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 214, in revive
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.declare()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 100, in declare
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 163, in declare
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive,
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, allowed_methods)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 270, in _wait_method
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.wait()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 523, in _close
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self._x_close_ok()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((10, 51))
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content,
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 177, in write_frame
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce,
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:36:14.050 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:36:15.052 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:36:15.080 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:36:57.747 2581 ERROR neutron.services.loadbalancer.agent.agent_manager [-] Unable to deploy instance for pool: 92fb078d-f203-4972-a555-e2ba033d50d8
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager Traceback (most recent call last):
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_manager.py", line 169, in _reload_pool
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager logical_config = self.plugin_rpc.get_logical_device(pool_id)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_api.py", line 57, in get_logical_device
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager host=self.host
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/log.py", line 34, in wrapper
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager return method(*args, **kwargs)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 161, in call
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager context, msg, rpc_method='call', **kwargs)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 187, in __call_rpc_method
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager return func(context, msg['method'], **msg['args'])
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 389, in call
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager return self.prepare().call(ctxt, method, **kwargs)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager retry=self.retry)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager timeout=timeout, retry=retry)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager retry=retry)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager result = self._waiter.wait(msg_id, timeout)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager reply, ending = self._poll_connection(msg_id, timeout)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager % msg_id)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID 78db36a255ae4b20b090554216923a20
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager
2015-08-11 20:37:15.791 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return method()
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:37:15.795 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:37:16.796 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:16.815 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-08-11 20:37:17.832 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:37:18.855 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:18.903 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:23.258 2581 ERROR oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Failed to publish message to topic 'n-lbaas-plugin': [Errno 32] Broken pipe
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit return method()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 752, in _publish
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic=topic, **kwargs)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 378, in __init__
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit **options)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 214, in revive
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.declare()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 100, in declare
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 163, in declare
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive,
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, allowed_methods)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 270, in _wait_method
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.wait()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 523, in _close
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self._x_close_ok()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((10, 51))
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content,
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 177, in write_frame
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce,
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:37:23.266 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Delaying reconnect for 1.0 seconds...
2015-08-11 20:37:24.277 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:24.337 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:24.344 2581 ERROR oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return method()
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:37:24.346 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Delaying reconnect for 1.0 seconds...
2015-08-11 20:37:25.347 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:25.387 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connected to AMQP server on 10.53.87.192:5672
[root@nsj17 ~]#
Notes Log:
IBM Licensed Material - Property of IBM ? Cop
==== State: Open by: liujgwx on 18 August 2015 11:25:02 ====
correct the number
180 with traffic
690 without traffic
Total Active LB Namespace = 870
Total LB Namespace = 1247
Inactive = 377
==== State: Open by: liujgwx on 13 August 2015 01:02:45 ====
[root@nsj17 ~]# sar -P ALL 60 1
Linux 3.10.0-229.7.2.el7.x86_64 (nsj17) 08/12/2015 _x86_64_ (80 CPU)
07:28:50 PM CPU %user %nice %system %iowait %steal %idle
07:29:50 PM all 2.45 0.00 72.87 0.00 0.00 24.68
07:29:50 PM 0 3.09 0.00 86.10 0.00 0.00 10.81
07:29:50 PM 1 3.24 0.00 83.92 0.00 0.00 12.84
07:29:50 PM 2 2.88 0.00 77.57 0.00 0.00 19.54
07:29:50 PM 3 2.16 0.00 75.61 0.00 0.00 22.23
07:29:50 PM 4 2.37 0.00 76.48 0.00 0.00 21.15
07:29:50 PM 5 2.44 0.00 75.88 0.00 0.00 21.69
07:29:50 PM 6 2.23 0.00 78.28 0.00 0.00 19.48
07:29:50 PM 7 2.49 0.00 72.97 0.00 0.00 24.54
07:29:50 PM 8 2.63 0.00 75.52 0.00 0.00 21.84
07:29:50 PM 9 2.95 0.00 71.80 0.00 0.00 25.25
07:29:50 PM 10 2.23 0.00 76.04 0.00 0.00 21.73
07:29:50 PM 11 2.26 0.00 75.26 0.00 0.00 22.48
07:29:50 PM 12 2.43 0.00 74.74 0.00 0.00 22.83
07:29:50 PM 13 1.95 0.00 73.81 0.00 0.00 24.23
07:29:50 PM 14 2.33 0.00 72.21 0.00 0.00 25.46
07:29:50 PM 15 2.22 0.00 73.05 0.00 0.00 24.74
07:29:50 PM 16 2.08 0.00 72.20 0.00 0.00 25.72
07:29:50 PM 17 2.13 0.00 71.66 0.00 0.00 26.21
07:29:50 PM 18 2.11 0.00 71.55 0.00 0.00 26.33
07:29:50 PM 19 2.18 0.00 72.35 0.00 0.00 25.47
07:29:50 PM 20 1.81 0.00 73.13 0.00 0.00 25.06
07:29:50 PM 21 1.57 0.00 73.98 0.00 0.00 24.45
07:29:50 PM 22 1.94 0.00 73.67 0.00 0.00 24.39
07:29:50 PM 23 1.79 0.00 75.57 0.00 0.00 22.64
07:29:50 PM 24 2.03 0.00 75.06 0.00 0.00 22.91
07:29:50 PM 25 1.85 0.00 73.46 0.00 0.00 24.69
07:29:50 PM 26 1.62 0.00 70.14 0.00 0.00 28.24
07:29:50 PM 27 1.85 0.00 73.08 0.00 0.00 25.07
07:29:50 PM 28 1.77 0.00 73.98 0.00 0.00 24.25
07:29:50 PM 29 1.58 0.00 72.81 0.00 0.00 25.60
07:29:50 PM 30 2.82 0.00 70.37 0.00 0.00 26.81
07:29:50 PM 31 2.74 0.00 73.64 0.00 0.00 23.62
07:29:50 PM 32 2.77 0.00 70.33 0.00 0.00 26.91
07:29:50 PM 33 2.57 0.00 72.26 0.00 0.00 25.17
07:29:50 PM 34 3.16 0.00 71.70 0.00 0.00 25.14
07:29:50 PM 35 2.61 0.00 70.21 0.00 0.00 27.18
07:29:50 PM 36 2.42 0.00 72.79 0.00 0.00 24.79
07:29:50 PM 37 2.31 0.00 74.05 0.00 0.00 23.64
07:29:50 PM 38 3.01 0.00 77.82 0.00 0.00 19.17
07:29:50 PM 39 2.60 0.00 74.25 0.00 0.00 23.15
07:29:50 PM 40 3.72 0.00 70.53 0.00 0.00 25.74
07:29:50 PM 41 2.77 0.00 68.57 0.00 0.00 28.66
07:29:50 PM 42 2.79 0.00 71.16 0.00 0.00 26.06
07:29:50 PM 43 2.94 0.00 71.23 0.00 0.00 25.83
07:29:50 PM 44 2.64 0.00 68.09 0.00 0.00 29.27
07:29:50 PM 45 2.63 0.00 68.67 0.00 0.00 28.70
07:29:50 PM 46 2.16 0.00 63.88 0.00 0.00 33.96
07:29:50 PM 47 3.06 0.00 66.90 0.00 0.00 30.04
07:29:50 PM 48 2.64 0.00 66.79 0.00 0.00 30.57
07:29:50 PM 49 2.98 0.00 65.90 0.00 0.00 31.12
07:29:50 PM 50 2.56 0.00 73.78 0.00 0.00 23.65
07:29:50 PM 51 2.16 0.00 76.16 0.00 0.00 21.67
07:29:50 PM 52 2.60 0.00 74.06 0.00 0.00 23.33
07:29:50 PM 53 2.46 0.00 73.68 0.00 0.00 23.87
07:29:50 PM 54 2.52 0.00 72.06 0.00 0.00 25.42
07:29:50 PM 55 2.24 0.00 73.17 0.00 0.00 24.60
07:29:50 PM 56 2.42 0.00 72.65 0.00 0.00 24.93
07:29:50 PM 57 2.49 0.00 72.79 0.00 0.00 24.72
07:29:50 PM 58 2.32 0.00 70.32 0.00 0.00 27.35
07:29:50 PM 59 1.96 0.00 68.68 0.00 0.00 29.37
07:29:50 PM 60 1.91 0.00 72.05 0.00 0.00 26.04
07:29:50 PM 61 2.35 0.00 74.71 0.00 0.00 22.93
07:29:50 PM 62 1.85 0.00 74.49 0.00 0.00 23.66
07:29:50 PM 63 2.49 0.00 73.61 0.00 0.00 23.90
07:29:50 PM 64 2.40 0.00 76.26 0.00 0.00 21.34
07:29:50 PM 65 1.97 0.00 73.83 0.00 0.00 24.20
07:29:50 PM 66 1.93 0.00 74.30 0.00 0.00 23.77
07:29:50 PM 67 2.24 0.00 70.79 0.00 0.00 26.98
07:29:50 PM 68 2.01 0.00 67.80 0.00 0.00 30.19
07:29:50 PM 69 1.85 0.00 71.32 0.00 0.00 26.83
07:29:50 PM 70 3.24 0.00 75.25 0.00 0.00 21.51
07:29:50 PM 71 3.20 0.00 77.97 0.00 0.00 18.83
07:29:50 PM 72 3.20 0.00 71.82 0.00 0.00 24.98
07:29:50 PM 73 2.74 0.00 73.68 0.00 0.00 23.59
07:29:50 PM 74 2.89 0.00 73.68 0.00 0.00 23.42
07:29:50 PM 75 3.24 0.00 74.40 0.00 0.00 22.35
07:29:50 PM 76 2.81 0.00 70.84 0.00 0.00 26.35
07:29:50 PM 77 2.83 0.00 73.13 0.00 0.00 24.04
07:29:50 PM 78 2.64 0.00 69.46 0.00 0.00 27.90
07:29:50 PM 79 2.88 0.00 64.28 0.00 0.00 32.84
Average: CPU %user %nice %system %iowait %steal %idle
Average: all 2.45 0.00 72.87 0.00 0.00 24.68
Average: 0 3.09 0.00 86.10 0.00 0.00 10.81
Average: 1 3.24 0.00 83.92 0.00 0.00 12.84
Average: 2 2.88 0.00 77.57 0.00 0.00 19.54
Average: 3 2.16 0.00 75.61 0.00 0.00 22.23
Average: 4 2.37 0.00 76.48 0.00 0.00 21.15
Average: 5 2.44 0.00 75.88 0.00 0.00 21.69
Average: 6 2.23 0.00 78.28 0.00 0.00 19.48
Average: 7 2.49 0.00 72.97 0.00 0.00 24.54
Average: 8 2.63 0.00 75.52 0.00 0.00 21.84
Average: 9 2.95 0.00 71.80 0.00 0.00 25.25
Average: 10 2.23 0.00 76.04 0.00 0.00 21.73
Average: 11 2.26 0.00 75.26 0.00 0.00 22.48
Average: 12 2.43 0.00 74.74 0.00 0.00 22.83
Average: 13 1.95 0.00 73.81 0.00 0.00 24.23
Average: 14 2.33 0.00 72.21 0.00 0.00 25.46
Average: 15 2.22 0.00 73.05 0.00 0.00 24.74
Average: 16 2.08 0.00 72.20 0.00 0.00 25.72
Average: 17 2.13 0.00 71.66 0.00 0.00 26.21
Average: 18 2.11 0.00 71.55 0.00 0.00 26.33
Average: 19 2.18 0.00 72.35 0.00 0.00 25.47
Average: 20 1.81 0.00 73.13 0.00 0.00 25.06
Average: 21 1.57 0.00 73.98 0.00 0.00 24.45
Average: 22 1.94 0.00 73.67 0.00 0.00 24.39
Average: 23 1.79 0.00 75.57 0.00 0.00 22.64
Average: 24 2.03 0.00 75.06 0.00 0.00 22.91
Average: 25 1.85 0.00 73.46 0.00 0.00 24.69
Average: 26 1.62 0.00 70.14 0.00 0.00 28.24
Average: 27 1.85 0.00 73.08 0.00 0.00 25.07
Average: 28 1.77 0.00 73.98 0.00 0.00 24.25
Average: 29 1.58 0.00 72.81 0.00 0.00 25.60
Average: 30 2.82 0.00 70.37 0.00 0.00 26.81
Average: 31 2.74 0.00 73.64 0.00 0.00 23.62
Average: 32 2.77 0.00 70.33 0.00 0.00 26.91
Average: 33 2.57 0.00 72.26 0.00 0.00 25.17
Average: 34 3.16 0.00 71.70 0.00 0.00 25.14
Average: 35 2.61 0.00 70.21 0.00 0.00 27.18
Average: 36 2.42 0.00 72.79 0.00 0.00 24.79
Average: 37 2.31 0.00 74.05 0.00 0.00 23.64
Average: 38 3.01 0.00 77.82 0.00 0.00 19.17
Average: 39 2.60 0.00 74.25 0.00 0.00 23.15
Average: 40 3.72 0.00 70.53 0.00 0.00 25.74
Average: 41 2.77 0.00 68.57 0.00 0.00 28.66
Average: 42 2.79 0.00 71.16 0.00 0.00 26.06
Average: 43 2.94 0.00 71.23 0.00 0.00 25.83
Average: 44 2.64 0.00 68.09 0.00 0.00 29.27
Average: 45 2.63 0.00 68.67 0.00 0.00 28.70
Average: 46 2.16 0.00 63.88 0.00 0.00 33.96
Average: 47 3.06 0.00 66.90 0.00 0.00 30.04
Average: 48 2.64 0.00 66.79 0.00 0.00 30.57
Average: 49 2.98 0.00 65.90 0.00 0.00 31.12
Average: 50 2.56 0.00 73.78 0.00 0.00 23.65
Average: 51 2.16 0.00 76.16 0.00 0.00 21.67
Average: 52 2.60 0.00 74.06 0.00 0.00 23.33
Average: 53 2.46 0.00 73.68 0.00 0.00 23.87
Average: 54 2.52 0.00 72.06 0.00 0.00 25.42
Average: 55 2.24 0.00 73.17 0.00 0.00 24.60
Average: 56 2.42 0.00 72.65 0.00 0.00 24.93
Average: 57 2.49 0.00 72.79 0.00 0.00 24.72
Average: 58 2.32 0.00 70.32 0.00 0.00 27.35
Average: 59 1.96 0.00 68.68 0.00 0.00 29.37
Average: 60 1.91 0.00 72.05 0.00 0.00 26.04
Average: 61 2.35 0.00 74.71 0.00 0.00 22.93
Average: 62 1.85 0.00 74.49 0.00 0.00 23.66
Average: 63 2.49 0.00 73.61 0.00 0.00 23.90
Average: 64 2.40 0.00 76.26 0.00 0.00 21.34
Average: 65 1.97 0.00 73.83 0.00 0.00 24.20
Average: 66 1.93 0.00 74.30 0.00 0.00 23.77
Average: 67 2.24 0.00 70.79 0.00 0.00 26.98
Average: 68 2.01 0.00 67.80 0.00 0.00 30.19
Average: 69 1.85 0.00 71.32 0.00 0.00 26.83
Average: 70 3.24 0.00 75.25 0.00 0.00 21.51
Average: 71 3.20 0.00 77.97 0.00 0.00 18.83
Average: 72 3.20 0.00 71.82 0.00 0.00 24.98
Average: 73 2.74 0.00 73.68 0.00 0.00 23.59
Average: 74 2.89 0.00 73.68 0.00 0.00 23.42
Average: 75 3.24 0.00 74.40 0.00 0.00 22.35
Average: 76 2.81 0.00 70.84 0.00 0.00 26.35
Average: 77 2.83 0.00 73.13 0.00 0.00 24.04
Average: 78 2.64 0.00 69.46 0.00 0.00 27.90
Average: 79 2.88 0.00 64.28 0.00 0.00 32.84
[root@nsj17 ~]# free
total used free shared buff/cache available
Mem: 263961496 19596608 220503484 4236472 23861404 231497040
Swap: 1048572 0 1048572
[root@nsj17 ~]#
==== State: Open by: liujgwx on 12 August 2015 02:52:28 ====
liujgwx (liujgwx@xxxxxxxxxx) added native attachment /opt/IBM/WebSphere/AppServer/profiles/cqweb/temp/ralcqweb4Node01/server1/TeamEAR/cqweb.war/console output .txt on 2015-08-12 02:52:28
liujgwx (liujgwx@xxxxxxxxxx) added native attachment /opt/IBM/WebSphere/AppServer/profiles/cqweb/temp/ralcqweb4Node01/server1/TeamEAR/cqweb.war/total pools .txt on 2015-08-12 02:52:28
==== State: Open by: liujgwx on 12 August 2015 02:45:28 ====
Notice: This Note entry was modified. 1 non-ascii character(s) were
replaced with question marks.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1498472
Title:
after 871 namespace was created for 1v1 maping, the new LB pool
with pending_creat status
Status in neutron:
New
Bug description:
[Alchemy][LB_namespaceV1]after 817 tenants was create for namespace 1v1 maping, the new LB pool with pending_creat status
test step
1 create a tenant
2 create a pool network with subnet 10.1.1.0
create a vip network with subnet 20.1.1.0
3 create a router and add 2 interface on it for subnet 10 and subnet20
4 create a LB pool with subnet 10
5 add a VIP with vip address 20.1.1.100 on this pool with protocol HTTP and port 80
6(option ) add one or two member
7 repeat above step to continuous add 1v1 map( 1 tenant map 1 lb namespce) for scale test
add to following tenant
80 tenants with 2 backend and 1 client (LB monitor enable )with simple traffic
100 tenants with 1 backend and 1 client (LB monitor enable) with simple traffic
1067 tenants without member and monitor
total of 1247 pools , total of 817 pools with active , total of 376 pools with error or pending-create status ) log in attachment
the issue :after add 817 namespace , created new lb pool with pending-status or error status
after check log, AMQP can't connect to server ,
and then reset rabbitmq service on controller node " service rabbitmq-server restart"
the setup can't recover , the new lb pool still with pending-create status
for recover setup, start to delete tenants and lb-pool and vip ( 1v1 maping)
after delete tenant to 328(remain), the setup recover , new lb pool can be create with active status , so it was a scale issue , and file this bug for trace this issue
log
[root@nsj17 ~]# tail -n 200 /var/log/neutron/lbaas-agent.log
2015-08-11 20:35:51.939 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:35:51.944 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:35:51.970 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return method()
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:35:51.972 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:35:52.946 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:52.971 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-08-11 20:35:52.973 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:52.993 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-08-11 20:35:53.990 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:35:53.999 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:35:55.000 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:55.023 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:55.060 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:55.083 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:57.681 2581 WARNING oslo.messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : ed3ca0b705914e4b9e0542fd1cdc3eda, message : {u'_unique_id': u'4b941437471a4de99f0871e34216a70a', u'failure': None, u'result': None}
2015-08-11 20:35:57.683 2581 WARNING oslo.messaging._drivers.amqpdriver [-] _queues: {'060c2e4d42f04e31be500647207d3a47': <Queue at 0x3a6e290 maxsize=None getters[1]>, '78db36a255ae4b20b090554216923a20': <Queue at 0x3c2e5d0 maxsize=None>}
2015-08-11 20:35:57.737 2581 WARNING oslo.messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : ed3ca0b705914e4b9e0542fd1cdc3eda, message : {u'_unique_id': u'e9bfc396a78c49a2993f6db149711c17', u'failure': None, u'result': None, u'ending': True}
2015-08-11 20:35:57.738 2581 WARNING oslo.messaging._drivers.amqpdriver [-] _queues: {'060c2e4d42f04e31be500647207d3a47': <Queue at 0x3a6e290 maxsize=None getters[1] tasks=1 _cond=<Event at 0x3b09050 result=NOT_USED _exc=None _waiters[0]>>, '78db36a255ae4b20b090554216923a20': <Queue at 0x3c2e5d0 maxsize=None>}
2015-08-11 20:36:14.044 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'n-lbaas-plugin': [Errno 32] Broken pipe
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit return method()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 752, in _publish
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic=topic, **kwargs)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 378, in __init__
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit **options)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 214, in revive
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.declare()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 100, in declare
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 163, in declare
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive,
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, allowed_methods)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 270, in _wait_method
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.wait()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 523, in _close
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self._x_close_ok()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((10, 51))
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content,
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 177, in write_frame
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce,
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:36:14.050 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:36:15.052 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:36:15.080 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:36:57.747 2581 ERROR neutron.services.loadbalancer.agent.agent_manager [-] Unable to deploy instance for pool: 92fb078d-f203-4972-a555-e2ba033d50d8
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager Traceback (most recent call last):
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_manager.py", line 169, in _reload_pool
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager logical_config = self.plugin_rpc.get_logical_device(pool_id)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_api.py", line 57, in get_logical_device
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager host=self.host
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/log.py", line 34, in wrapper
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager return method(*args, **kwargs)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 161, in call
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager context, msg, rpc_method='call', **kwargs)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 187, in __call_rpc_method
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager return func(context, msg['method'], **msg['args'])
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 389, in call
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager return self.prepare().call(ctxt, method, **kwargs)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager retry=self.retry)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager timeout=timeout, retry=retry)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager retry=retry)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager result = self._waiter.wait(msg_id, timeout)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager reply, ending = self._poll_connection(msg_id, timeout)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager % msg_id)
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID 78db36a255ae4b20b090554216923a20
2015-08-11 20:36:57.747 2581 TRACE neutron.services.loadbalancer.agent.agent_manager
2015-08-11 20:37:15.791 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return method()
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:37:15.795 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:37:16.796 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:16.815 2581 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-08-11 20:37:17.832 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2015-08-11 20:37:18.855 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:18.903 2581 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:23.258 2581 ERROR oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Failed to publish message to topic 'n-lbaas-plugin': [Errno 32] Broken pipe
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit return method()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 752, in _publish
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic=topic, **kwargs)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 378, in __init__
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit **options)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 326, in __init__
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 334, in reconnect
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 214, in revive
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.declare()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 100, in declare
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 163, in declare
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive,
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit (40, 11), # Channel.exchange_declare_ok
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, allowed_methods)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 270, in _wait_method
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.wait()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.dispatch_method(method_sig, args, content)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(self, args)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 523, in _close
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self._x_close_ok()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((10, 51))
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content,
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 177, in write_frame
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce,
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:37:23.266 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Delaying reconnect for 1.0 seconds...
2015-08-11 20:37:24.277 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:24.337 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:24.344 2581 ERROR oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 655, in ensure
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return method()
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 735, in _consume
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in drain_events
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit (class_id, method_id), ConnectionError)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:37:24.346 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Delaying reconnect for 1.0 seconds...
2015-08-11 20:37:25.347 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:25.387 2581 INFO oslo.messaging._drivers.impl_rabbit [req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connected to AMQP server on 10.53.87.192:5672
[root@nsj17 ~]#
Notes Log:
IBM Licensed Material - Property of IBM ? Cop
==== State: Open by: liujgwx on 18 August 2015 11:25:02 ====
correct the number
180 with traffic
690 without traffic
Total Active LB Namespace = 870
Total LB Namespace = 1247
Inactive = 377
==== State: Open by: liujgwx on 13 August 2015 01:02:45 ====
[root@nsj17 ~]# sar -P ALL 60 1
Linux 3.10.0-229.7.2.el7.x86_64 (nsj17) 08/12/2015 _x86_64_ (80 CPU)
07:28:50 PM CPU %user %nice %system %iowait %steal %idle
07:29:50 PM all 2.45 0.00 72.87 0.00 0.00 24.68
07:29:50 PM 0 3.09 0.00 86.10 0.00 0.00 10.81
07:29:50 PM 1 3.24 0.00 83.92 0.00 0.00 12.84
07:29:50 PM 2 2.88 0.00 77.57 0.00 0.00 19.54
07:29:50 PM 3 2.16 0.00 75.61 0.00 0.00 22.23
07:29:50 PM 4 2.37 0.00 76.48 0.00 0.00 21.15
07:29:50 PM 5 2.44 0.00 75.88 0.00 0.00 21.69
07:29:50 PM 6 2.23 0.00 78.28 0.00 0.00 19.48
07:29:50 PM 7 2.49 0.00 72.97 0.00 0.00 24.54
07:29:50 PM 8 2.63 0.00 75.52 0.00 0.00 21.84
07:29:50 PM 9 2.95 0.00 71.80 0.00 0.00 25.25
07:29:50 PM 10 2.23 0.00 76.04 0.00 0.00 21.73
07:29:50 PM 11 2.26 0.00 75.26 0.00 0.00 22.48
07:29:50 PM 12 2.43 0.00 74.74 0.00 0.00 22.83
07:29:50 PM 13 1.95 0.00 73.81 0.00 0.00 24.23
07:29:50 PM 14 2.33 0.00 72.21 0.00 0.00 25.46
07:29:50 PM 15 2.22 0.00 73.05 0.00 0.00 24.74
07:29:50 PM 16 2.08 0.00 72.20 0.00 0.00 25.72
07:29:50 PM 17 2.13 0.00 71.66 0.00 0.00 26.21
07:29:50 PM 18 2.11 0.00 71.55 0.00 0.00 26.33
07:29:50 PM 19 2.18 0.00 72.35 0.00 0.00 25.47
07:29:50 PM 20 1.81 0.00 73.13 0.00 0.00 25.06
07:29:50 PM 21 1.57 0.00 73.98 0.00 0.00 24.45
07:29:50 PM 22 1.94 0.00 73.67 0.00 0.00 24.39
07:29:50 PM 23 1.79 0.00 75.57 0.00 0.00 22.64
07:29:50 PM 24 2.03 0.00 75.06 0.00 0.00 22.91
07:29:50 PM 25 1.85 0.00 73.46 0.00 0.00 24.69
07:29:50 PM 26 1.62 0.00 70.14 0.00 0.00 28.24
07:29:50 PM 27 1.85 0.00 73.08 0.00 0.00 25.07
07:29:50 PM 28 1.77 0.00 73.98 0.00 0.00 24.25
07:29:50 PM 29 1.58 0.00 72.81 0.00 0.00 25.60
07:29:50 PM 30 2.82 0.00 70.37 0.00 0.00 26.81
07:29:50 PM 31 2.74 0.00 73.64 0.00 0.00 23.62
07:29:50 PM 32 2.77 0.00 70.33 0.00 0.00 26.91
07:29:50 PM 33 2.57 0.00 72.26 0.00 0.00 25.17
07:29:50 PM 34 3.16 0.00 71.70 0.00 0.00 25.14
07:29:50 PM 35 2.61 0.00 70.21 0.00 0.00 27.18
07:29:50 PM 36 2.42 0.00 72.79 0.00 0.00 24.79
07:29:50 PM 37 2.31 0.00 74.05 0.00 0.00 23.64
07:29:50 PM 38 3.01 0.00 77.82 0.00 0.00 19.17
07:29:50 PM 39 2.60 0.00 74.25 0.00 0.00 23.15
07:29:50 PM 40 3.72 0.00 70.53 0.00 0.00 25.74
07:29:50 PM 41 2.77 0.00 68.57 0.00 0.00 28.66
07:29:50 PM 42 2.79 0.00 71.16 0.00 0.00 26.06
07:29:50 PM 43 2.94 0.00 71.23 0.00 0.00 25.83
07:29:50 PM 44 2.64 0.00 68.09 0.00 0.00 29.27
07:29:50 PM 45 2.63 0.00 68.67 0.00 0.00 28.70
07:29:50 PM 46 2.16 0.00 63.88 0.00 0.00 33.96
07:29:50 PM 47 3.06 0.00 66.90 0.00 0.00 30.04
07:29:50 PM 48 2.64 0.00 66.79 0.00 0.00 30.57
07:29:50 PM 49 2.98 0.00 65.90 0.00 0.00 31.12
07:29:50 PM 50 2.56 0.00 73.78 0.00 0.00 23.65
07:29:50 PM 51 2.16 0.00 76.16 0.00 0.00 21.67
07:29:50 PM 52 2.60 0.00 74.06 0.00 0.00 23.33
07:29:50 PM 53 2.46 0.00 73.68 0.00 0.00 23.87
07:29:50 PM 54 2.52 0.00 72.06 0.00 0.00 25.42
07:29:50 PM 55 2.24 0.00 73.17 0.00 0.00 24.60
07:29:50 PM 56 2.42 0.00 72.65 0.00 0.00 24.93
07:29:50 PM 57 2.49 0.00 72.79 0.00 0.00 24.72
07:29:50 PM 58 2.32 0.00 70.32 0.00 0.00 27.35
07:29:50 PM 59 1.96 0.00 68.68 0.00 0.00 29.37
07:29:50 PM 60 1.91 0.00 72.05 0.00 0.00 26.04
07:29:50 PM 61 2.35 0.00 74.71 0.00 0.00 22.93
07:29:50 PM 62 1.85 0.00 74.49 0.00 0.00 23.66
07:29:50 PM 63 2.49 0.00 73.61 0.00 0.00 23.90
07:29:50 PM 64 2.40 0.00 76.26 0.00 0.00 21.34
07:29:50 PM 65 1.97 0.00 73.83 0.00 0.00 24.20
07:29:50 PM 66 1.93 0.00 74.30 0.00 0.00 23.77
07:29:50 PM 67 2.24 0.00 70.79 0.00 0.00 26.98
07:29:50 PM 68 2.01 0.00 67.80 0.00 0.00 30.19
07:29:50 PM 69 1.85 0.00 71.32 0.00 0.00 26.83
07:29:50 PM 70 3.24 0.00 75.25 0.00 0.00 21.51
07:29:50 PM 71 3.20 0.00 77.97 0.00 0.00 18.83
07:29:50 PM 72 3.20 0.00 71.82 0.00 0.00 24.98
07:29:50 PM 73 2.74 0.00 73.68 0.00 0.00 23.59
07:29:50 PM 74 2.89 0.00 73.68 0.00 0.00 23.42
07:29:50 PM 75 3.24 0.00 74.40 0.00 0.00 22.35
07:29:50 PM 76 2.81 0.00 70.84 0.00 0.00 26.35
07:29:50 PM 77 2.83 0.00 73.13 0.00 0.00 24.04
07:29:50 PM 78 2.64 0.00 69.46 0.00 0.00 27.90
07:29:50 PM 79 2.88 0.00 64.28 0.00 0.00 32.84
Average: CPU %user %nice %system %iowait %steal %idle
Average: all 2.45 0.00 72.87 0.00 0.00 24.68
Average: 0 3.09 0.00 86.10 0.00 0.00 10.81
Average: 1 3.24 0.00 83.92 0.00 0.00 12.84
Average: 2 2.88 0.00 77.57 0.00 0.00 19.54
Average: 3 2.16 0.00 75.61 0.00 0.00 22.23
Average: 4 2.37 0.00 76.48 0.00 0.00 21.15
Average: 5 2.44 0.00 75.88 0.00 0.00 21.69
Average: 6 2.23 0.00 78.28 0.00 0.00 19.48
Average: 7 2.49 0.00 72.97 0.00 0.00 24.54
Average: 8 2.63 0.00 75.52 0.00 0.00 21.84
Average: 9 2.95 0.00 71.80 0.00 0.00 25.25
Average: 10 2.23 0.00 76.04 0.00 0.00 21.73
Average: 11 2.26 0.00 75.26 0.00 0.00 22.48
Average: 12 2.43 0.00 74.74 0.00 0.00 22.83
Average: 13 1.95 0.00 73.81 0.00 0.00 24.23
Average: 14 2.33 0.00 72.21 0.00 0.00 25.46
Average: 15 2.22 0.00 73.05 0.00 0.00 24.74
Average: 16 2.08 0.00 72.20 0.00 0.00 25.72
Average: 17 2.13 0.00 71.66 0.00 0.00 26.21
Average: 18 2.11 0.00 71.55 0.00 0.00 26.33
Average: 19 2.18 0.00 72.35 0.00 0.00 25.47
Average: 20 1.81 0.00 73.13 0.00 0.00 25.06
Average: 21 1.57 0.00 73.98 0.00 0.00 24.45
Average: 22 1.94 0.00 73.67 0.00 0.00 24.39
Average: 23 1.79 0.00 75.57 0.00 0.00 22.64
Average: 24 2.03 0.00 75.06 0.00 0.00 22.91
Average: 25 1.85 0.00 73.46 0.00 0.00 24.69
Average: 26 1.62 0.00 70.14 0.00 0.00 28.24
Average: 27 1.85 0.00 73.08 0.00 0.00 25.07
Average: 28 1.77 0.00 73.98 0.00 0.00 24.25
Average: 29 1.58 0.00 72.81 0.00 0.00 25.60
Average: 30 2.82 0.00 70.37 0.00 0.00 26.81
Average: 31 2.74 0.00 73.64 0.00 0.00 23.62
Average: 32 2.77 0.00 70.33 0.00 0.00 26.91
Average: 33 2.57 0.00 72.26 0.00 0.00 25.17
Average: 34 3.16 0.00 71.70 0.00 0.00 25.14
Average: 35 2.61 0.00 70.21 0.00 0.00 27.18
Average: 36 2.42 0.00 72.79 0.00 0.00 24.79
Average: 37 2.31 0.00 74.05 0.00 0.00 23.64
Average: 38 3.01 0.00 77.82 0.00 0.00 19.17
Average: 39 2.60 0.00 74.25 0.00 0.00 23.15
Average: 40 3.72 0.00 70.53 0.00 0.00 25.74
Average: 41 2.77 0.00 68.57 0.00 0.00 28.66
Average: 42 2.79 0.00 71.16 0.00 0.00 26.06
Average: 43 2.94 0.00 71.23 0.00 0.00 25.83
Average: 44 2.64 0.00 68.09 0.00 0.00 29.27
Average: 45 2.63 0.00 68.67 0.00 0.00 28.70
Average: 46 2.16 0.00 63.88 0.00 0.00 33.96
Average: 47 3.06 0.00 66.90 0.00 0.00 30.04
Average: 48 2.64 0.00 66.79 0.00 0.00 30.57
Average: 49 2.98 0.00 65.90 0.00 0.00 31.12
Average: 50 2.56 0.00 73.78 0.00 0.00 23.65
Average: 51 2.16 0.00 76.16 0.00 0.00 21.67
Average: 52 2.60 0.00 74.06 0.00 0.00 23.33
Average: 53 2.46 0.00 73.68 0.00 0.00 23.87
Average: 54 2.52 0.00 72.06 0.00 0.00 25.42
Average: 55 2.24 0.00 73.17 0.00 0.00 24.60
Average: 56 2.42 0.00 72.65 0.00 0.00 24.93
Average: 57 2.49 0.00 72.79 0.00 0.00 24.72
Average: 58 2.32 0.00 70.32 0.00 0.00 27.35
Average: 59 1.96 0.00 68.68 0.00 0.00 29.37
Average: 60 1.91 0.00 72.05 0.00 0.00 26.04
Average: 61 2.35 0.00 74.71 0.00 0.00 22.93
Average: 62 1.85 0.00 74.49 0.00 0.00 23.66
Average: 63 2.49 0.00 73.61 0.00 0.00 23.90
Average: 64 2.40 0.00 76.26 0.00 0.00 21.34
Average: 65 1.97 0.00 73.83 0.00 0.00 24.20
Average: 66 1.93 0.00 74.30 0.00 0.00 23.77
Average: 67 2.24 0.00 70.79 0.00 0.00 26.98
Average: 68 2.01 0.00 67.80 0.00 0.00 30.19
Average: 69 1.85 0.00 71.32 0.00 0.00 26.83
Average: 70 3.24 0.00 75.25 0.00 0.00 21.51
Average: 71 3.20 0.00 77.97 0.00 0.00 18.83
Average: 72 3.20 0.00 71.82 0.00 0.00 24.98
Average: 73 2.74 0.00 73.68 0.00 0.00 23.59
Average: 74 2.89 0.00 73.68 0.00 0.00 23.42
Average: 75 3.24 0.00 74.40 0.00 0.00 22.35
Average: 76 2.81 0.00 70.84 0.00 0.00 26.35
Average: 77 2.83 0.00 73.13 0.00 0.00 24.04
Average: 78 2.64 0.00 69.46 0.00 0.00 27.90
Average: 79 2.88 0.00 64.28 0.00 0.00 32.84
[root@nsj17 ~]# free
total used free shared buff/cache available
Mem: 263961496 19596608 220503484 4236472 23861404 231497040
Swap: 1048572 0 1048572
[root@nsj17 ~]#
==== State: Open by: liujgwx on 12 August 2015 02:52:28 ====
liujgwx (liujgwx@xxxxxxxxxx) added native attachment /opt/IBM/WebSphere/AppServer/profiles/cqweb/temp/ralcqweb4Node01/server1/TeamEAR/cqweb.war/console output .txt on 2015-08-12 02:52:28
liujgwx (liujgwx@xxxxxxxxxx) added native attachment /opt/IBM/WebSphere/AppServer/profiles/cqweb/temp/ralcqweb4Node01/server1/TeamEAR/cqweb.war/total pools .txt on 2015-08-12 02:52:28
==== State: Open by: liujgwx on 12 August 2015 02:45:28 ====
Notice: This Note entry was modified. 1 non-ascii character(s) were
replaced with question marks.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1498472/+subscriptions
Follow ups
