yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1640029] Re: [stable/newton] Deleting heat stack failed due to error "QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andreas Scheuring <andreas.scheuring@xxxxxxxxxx>
Date: Wed, 09 Nov 2016 10:52:26 -0000
Reply-to: Bug 1640029 <1640029@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Hi Sujai, thanks for reporting this. The issue is not yet clear to me. 
Can you please provide: 
- max_pool_size, retry_interval, max_overflow settings of your first run
- the nova error message of the first and the second run
- the neutron error message of the first and second run
At the moment you provided a neutron message for the first run and a nova message for the second. So I can't see what changed after updating the settings. In general it looks like that for deleting 500 instances in parallel a lot of sql queries are required. It seems like for such a use case you need to tune your parameters..

But let's compare your error message of the first and second run first.
If they are equal, I tend to set this to invalid

** Also affects: nova
   Importance: Undecided
       Status: New

** Changed in: nova
       Status: New => Incomplete

** Changed in: neutron
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1640029

Title:
  [stable/newton] Deleting heat stack failed due to error "QueuePool
  limit of size 50 overflow 50 reached, connection timed out, timeout
  30"

Status in neutron:
  Incomplete
Status in OpenStack Compute (nova):
  Incomplete

Bug description:
  In my CH + stable/newton setup, I brought up 5 heat stacks each having 100 nova instances in the same /16 network.
  Deleting those heat stacks failed due to the below error.

  "
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30
  "

  Because of this error, out of 500 instances, deletion of about 67 instances got failed.
  With default parameters in neutron.conf, I'm getting the below neutron error.

  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource [req-a0022887-cc01-4f2e-980d-490136524363 admin -] delete failed: No details.
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/resource.py", line 79, in resource
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 555, in delete
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     return self._delete(request, id, **kwargs)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/api.py", line 88, in wrapped
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/api.py", line 84, in wrapped
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/api.py", line 124, in wrapped
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     traceback.format_exc())
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     self.force_reraise()
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/db/api.py", line 119, in wrapped
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/opt/stack/neutron/neutron/api/v2/base.py", line 577, in _delete
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 1814, in delete_port
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     self.disassociate_floatingips(context, port_id)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/opt/stack/vmware-nsx/vmware_nsx/plugins/nsx_v3/plugin.py", line 2804, in disassociate_floatingips
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     for fip_db in fip_dbs:
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in __iter__
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     return self._execute_and_instances(context)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     close_with_result=True)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     **kw)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     execution_options=execution_options)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 900, in _connection_for_bind
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     conn = engine.contextual_connect(**kw)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     self._wrap_pool_connect(self.pool.connect, None),
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     return fn()
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     return _ConnectionFairy._checkout(self)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     fairy = _ConnectionRecord.checkout(pool)
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     rec = pool._do_get()
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1053, in _do_get
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource     (self.size(), self.overflow(), self._timeout))
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource TimeoutError: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
  2016-11-02 20:42:06.557 18058 ERROR neutron.api.v2.resource


  After changing the below parameters in /etc/neutron/neutron.conf and 
   
  max_pool_size = 50
  retry_interval = 10
  max_overflow = 50
  pool_max_size = 50
  pool_max_overflow = 50
  pool_timeout = 30

  below parameters in nova.conf and restarted the services and re-
  executed the testcase.Still deleting heat stack is failing with the
  below error

  max_pool_size = 50
  max_overflow = 50

  n-api.log:

  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] Unexpected exception in API method
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 914, in delete
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     self._delete(req.environ['nova.context'], req, id)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 755, in _delete
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     instance = self._get_server(context, req, instance_uuid)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 382, in _get_server
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     expected_attrs=expected_attrs)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/common.py", line 496, in get_instance
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     expected_attrs=expected_attrs)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 2215, in get
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     expected_attrs)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 2198, in _get_instance
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     context, instance_uuid, expected_attrs=expected_attrs)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 444, in get_by_uuid
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     use_slave=use_slave)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 225, in wrapper
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 436, in _db_instance_get_by_uuid
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/api.py", line 725, in instance_get_by_uuid
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 169, in wrapper
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 270, in wrapped
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     return f(context, *args, **kwargs)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1867, in instance_get_by_uuid
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     columns_to_join=columns_to_join)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1872, in _instance_get_by_uuid
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     filter_by(uuid=uuid).\
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2659, in first
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     ret = list(self[0:1])
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     return list(res)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in __iter__
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     return self._execute_and_instances(context)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     close_with_result=True)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     **kw)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     execution_options=execution_options)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     engine, execution_options)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     conn = bind.contextual_connect()
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     self._wrap_pool_connect(self.pool.connect, None),
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     return fn()
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     return _ConnectionFairy._checkout(self)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     fairy = _ConnectionRecord.checkout(pool)
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     rec = pool._do_get()
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1053, in _do_get
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions     (self.size(), self.overflow(), self._timeout))
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30
  2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions
  2016-11-03 17:27:34.148 2399 INFO nova.api.openstack.wsgi [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.TimeoutError'>
  2016-11-03 17:27:34.148 2399 DEBUG nova.api.openstack.wsgi [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.TimeoutError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1044

  Please look into this.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1640029/+subscriptions
References

[Bug 1640029] [NEW] [stable/newton] Deleting heat stack failed due to error "QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30"
From: Sujai, 2016-11-08