← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #08178

[Bug 1268661] [NEW] OVS agent gets timeout on fetching security groups from neutron server

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:


The following traceback is observed in ovs-agent (+ovs plugin) in a deployment with 100+ compute nodes.

14:58:39.761 21728 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     **args)
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 95, in security_groups_member_updated
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     self.sg_agent.security_groups_member_updated(security_groups)
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 137, in security_groups_member_updated
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     'security_group_source_groups')
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 147, in _security_group_updated
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     self.refresh_firewall(devices)
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 175, in refresh_firewall
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     self.context, device_ids)
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 58, in security_group_rules_for_devices
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     topic=self.topic)
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/proxy.py", line 126, in call
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     result = rpc.call(context, real_topic, msg, timeout)
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/__init__.py", line 140, in call
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     return _get_impl().call(CONF, context, topic, msg, timeout)
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 796, in call
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     rpc_amqp.get_connection_pool(conf, Connection))
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 551, in call
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     rv = list(rv)
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 516, in __iter__
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     raise result
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp RemoteError: Remote error: TimeoutError QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data\n    **args)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/common/rpc.py", line 44, in dispatch\n    neutron_ctxt, version, method, namespace, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n    result = getattr(proxyobj, method)(ctxt, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/db/securitygroups_rpc_base.py", line 142, in security_group_rules_for_devices\n    port = self.get_port_from_device(device)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/ovs_neutron_plugin.py", line 95, in get_port_from_device\n    port = ovs_db_v2.get_port_from_device(device)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/ovs_db_v2.py", line 331, in get_port_from_device\n    port_and_sgs = query.all()\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2115, in all\n    return list(self)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2227, in __iter__\n    return self._execute_and_instances(context)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2240, in _execute_and_instances\n    close_with_result=True)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2231, in _connection_from_session\n    **kw)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 777, in connection\n    close_with_result=close_with_result)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 783, in _connection_for_bind\n    return engine.contextual_connect(**kwargs)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 2489, in contextual_connect\n    self.pool.connect(),\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 236, in connect\n    return _ConnectionFairy(self).checkout()\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 401, in __init__\n    rec = self._connection_record = pool._do_get()\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 738, in _do_get\n    (self.size(), self.overflow(), self._timeout))\n', u'TimeoutError: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10\n'].


Neutron is Havana 2013.2

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: db ovs

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1268661

Title:
  OVS agent gets timeout on fetching security groups from neutron server

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  
  The following traceback is observed in ovs-agent (+ovs plugin) in a deployment with 100+ compute nodes.

  14:58:39.761 21728 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     **args)
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 95, in security_groups_member_updated
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     self.sg_agent.security_groups_member_updated(security_groups)
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 137, in security_groups_member_updated
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     'security_group_source_groups')
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 147, in _security_group_updated
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     self.refresh_firewall(devices)
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 175, in refresh_firewall
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     self.context, device_ids)
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 58, in security_group_rules_for_devices
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     topic=self.topic)
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/proxy.py", line 126, in call
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     result = rpc.call(context, real_topic, msg, timeout)
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/__init__.py", line 140, in call
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     return _get_impl().call(CONF, context, topic, msg, timeout)
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 796, in call
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     rpc_amqp.get_connection_pool(conf, Connection))
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 551, in call
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     rv = list(rv)
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 516, in __iter__
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp     raise result
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp RemoteError: Remote error: TimeoutError QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
  14:58:39.761 21728 TRACE neutron.openstack.common.rpc.amqp [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data\n    **args)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/common/rpc.py", line 44, in dispatch\n    neutron_ctxt, version, method, namespace, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n    result = getattr(proxyobj, method)(ctxt, **kwargs)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/db/securitygroups_rpc_base.py", line 142, in security_group_rules_for_devices\n    port = self.get_port_from_device(device)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/ovs_neutron_plugin.py", line 95, in get_port_from_device\n    port = ovs_db_v2.get_port_from_device(device)\n', u'  File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/ovs_db_v2.py", line 331, in get_port_from_device\n    port_and_sgs = query.all()\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2115, in all\n    return list(self)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2227, in __iter__\n    return self._execute_and_instances(context)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2240, in _execute_and_instances\n    close_with_result=True)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2231, in _connection_from_session\n    **kw)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 777, in connection\n    close_with_result=close_with_result)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 783, in _connection_for_bind\n    return engine.contextual_connect(**kwargs)\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 2489, in contextual_connect\n    self.pool.connect(),\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 236, in connect\n    return _ConnectionFairy(self).checkout()\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 401, in __init__\n    rec = self._connection_record = pool._do_get()\n', u'  File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 738, in _do_get\n    (self.size(), self.overflow(), self._timeout))\n', u'TimeoutError: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10\n'].

  
  Neutron is Havana 2013.2

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1268661/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next