← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #35083

[Bug 1472347] [NEW] With multiple Neutron api/rpc workers enabled, intermittent failure deleting dhcp_port

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Neutron multiple workers are enabled as follows in neutron.conf:
   - api_workers=3
   - rpc_workers=3

The following were configured:
   - 20 tenants
   - Each tenant had 5 tenant networks
   - For each network, one VM at each Compute nodes (2) for a total of 10 VMs
   - Total 100 VLANs/200 VMs
 
A script which did the following at tenant-1:
   - Delete all 10 VMs
   - For each network, delete its router interface
   - Delete the subnet
   - Delete the network
   - Re-create the network, subnet and router interface
   - For each network, launch 2 VMs (one at each Compute node)
   - Repeat steps 1 – 6

Intermittently the following delete port error is encountered:

2015-07-06 16:17:51.903 43190 DEBUG neutron.plugins.ml2.plugin [req-f18af2a1-0047-4301-9fa1-01632fa5b2b8 None] Calling delete_port for fcf17b5d-235c-466b-b54b-ce80acca7359 owned by network:dhcp delete_p
ort /usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py:1076
2015-07-06 16:17:51.904 43216 ERROR oslo.db.sqlalchemy.exc_filters [req-cbb23fa8-5043-405c-a569-fcfdc912555a ] DB exception wrapped.
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters Traceback (most recent call last):
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 781, in fetchall
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters     l = self.process_rows(self._fetchall_impl())
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 750, in _fetchall_impl
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters     self._non_result()
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 755, in _non_result
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters     "This result object does not return rows. "
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters ResourceClosedError: This result object does not return rows. It has been closed automatically.
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters 
2015-07-06 16:17:51.906 43216 DEBUG neutron.openstack.common.lockutils [req-cbb23fa8-5043-405c-a569-fcfdc912555a ] Releasing semaphore "db-access" lock /usr/lib/python2.7/site-packages/neutron/openstack
/common/lockutils.py:238
2015-07-06 16:17:51.906 43216 ERROR neutron.api.v2.resource [req-cbb23fa8-5043-405c-a569-fcfdc912555a None] delete failed
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource Traceback (most recent call last):
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 81, in resource
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     result = method(request=request, **args)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 476, in delete
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 680, in delete_network
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     continue
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 82, in __exit__
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 640, in delete_network
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     with_lockmode('update').all())
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2300, in all
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     return list(self)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 66, in instances
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     fetch = cursor.fetchall()
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 787, in fetchall
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     self.cursor, self.context)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     util.raise_from_cause(newraise, exc_info)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     reraise(type(exception), exception, tb=exc_tb)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 781, in fetchall
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     l = self.process_rows(self._fetchall_impl())
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 750, in _fetchall_impl
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     self._non_result()
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 755, in _non_result
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     "This result object does not return rows. "
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource DBError: This result object does not return rows. It has been closed automatically.


As a result, the network is not deleted, and hence duplicate network when the network is created again with the same name.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1472347

Title:
  With multiple Neutron api/rpc workers enabled, intermittent failure
  deleting dhcp_port

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  Neutron multiple workers are enabled as follows in neutron.conf:
     - api_workers=3
     - rpc_workers=3

  The following were configured:
     - 20 tenants
     - Each tenant had 5 tenant networks
     - For each network, one VM at each Compute nodes (2) for a total of 10 VMs
     - Total 100 VLANs/200 VMs
   
  A script which did the following at tenant-1:
     - Delete all 10 VMs
     - For each network, delete its router interface
     - Delete the subnet
     - Delete the network
     - Re-create the network, subnet and router interface
     - For each network, launch 2 VMs (one at each Compute node)
     - Repeat steps 1 – 6

  Intermittently the following delete port error is encountered:

  2015-07-06 16:17:51.903 43190 DEBUG neutron.plugins.ml2.plugin [req-f18af2a1-0047-4301-9fa1-01632fa5b2b8 None] Calling delete_port for fcf17b5d-235c-466b-b54b-ce80acca7359 owned by network:dhcp delete_p
  ort /usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py:1076
  2015-07-06 16:17:51.904 43216 ERROR oslo.db.sqlalchemy.exc_filters [req-cbb23fa8-5043-405c-a569-fcfdc912555a ] DB exception wrapped.
  2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters Traceback (most recent call last):
  2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 781, in fetchall
  2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters     l = self.process_rows(self._fetchall_impl())
  2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 750, in _fetchall_impl
  2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters     self._non_result()
  2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 755, in _non_result
  2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters     "This result object does not return rows. "
  2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters ResourceClosedError: This result object does not return rows. It has been closed automatically.
  2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters 
  2015-07-06 16:17:51.906 43216 DEBUG neutron.openstack.common.lockutils [req-cbb23fa8-5043-405c-a569-fcfdc912555a ] Releasing semaphore "db-access" lock /usr/lib/python2.7/site-packages/neutron/openstack
  /common/lockutils.py:238
  2015-07-06 16:17:51.906 43216 ERROR neutron.api.v2.resource [req-cbb23fa8-5043-405c-a569-fcfdc912555a None] delete failed
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource Traceback (most recent call last):
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 81, in resource
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     result = method(request=request, **args)
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 476, in delete
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     obj_deleter(request.context, id, **kwargs)
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 680, in delete_network
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     continue
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 82, in __exit__
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 640, in delete_network
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     with_lockmode('update').all())
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2300, in all
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     return list(self)
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 66, in instances
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     fetch = cursor.fetchall()
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 787, in fetchall
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     self.cursor, self.context)
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     util.raise_from_cause(newraise, exc_info)
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     reraise(type(exception), exception, tb=exc_tb)
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 781, in fetchall
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     l = self.process_rows(self._fetchall_impl())
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 750, in _fetchall_impl
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     self._non_result()
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 755, in _non_result
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource     "This result object does not return rows. "
  2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource DBError: This result object does not return rows. It has been closed automatically.

  
  As a result, the network is not deleted, and hence duplicate network when the network is created again with the same name.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1472347/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next