yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1475218] [NEW] oslo db retry decorator doesn't apply to add_router_interface

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kevin Benton <1475218@xxxxxxxxxxxxxxxxxx>
Date: Thu, 16 Jul 2015 09:10:44 -0000
Reply-to: Bug 1475218 <1475218@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

http://logs.openstack.org/64/197564/17/check/gate-tempest-dsvm-neutron-
full/1493b21/logs/screen-q-svc.txt.gz#_2015-07-16_07_38_42_299

2015-07-16 07:38:42.299 ERROR neutron.api.v2.resource [req-38378c88-9223-434e-92b5-2124b1a6442a tempest-TestNetworkAdvancedServerOps-2072830200 tempest-TestNetworkAdvancedServerOps-1241923202] add_router_interface failed
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource Traceback (most recent call last):
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 210, in _handle_action
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     return getattr(self._plugin, name)(*arg_list, **kwargs)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 291, in add_router_interface
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     context, router, interface_info['subnet_id'], device_owner)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 603, in _add_interface_by_subnet
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     'name': ''}}), [subnet], True
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1015, in create_port
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     result, mech_context = self._create_port_db(context, port)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 990, in _create_port_db
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     result = super(Ml2Plugin, self).create_port(context, port)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 837, in create_port
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     self.ipam.allocate_ips_for_port_and_store(context, port, port_id)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_non_pluggable_backend.py", line 205, in allocate_ips_for_port_and_store
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     ips = self._allocate_ips_for_port(context, port)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_non_pluggable_backend.py", line 396, in _allocate_ips_for_port
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     p['mac_address'])
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_non_pluggable_backend.py", line 329, in _allocate_fixed_ips
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     context, fixed['subnet_id'], fixed['ip_address'])
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_non_pluggable_backend.py", line 147, in _allocate_specific_ip
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     for ip_range in results:
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     return self._execute_and_instances(context)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     result = conn.execute(querycontext.statement, self._params)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     return meth(self, multiparams, params)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     return connection._execute_clauseelement(self, multiparams, params)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     compiled_sql, distilled_params
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     context)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     util.raise_from_cause(newraise, exc_info)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     reraise(type(exception), exception, tb=exc_tb)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     context)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     cursor.execute(statement, parameters)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 205, in execute
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     self.errorhandler(self, exc, value)
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     raise errorclass, errorvalue
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource DBDeadlock: (_mysql_exceptions.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') [SQL: u'SELECT ipavailabilityranges.allocation_pool_id AS ipavailabilityranges_allocation_pool_id, ipavailabilityranges.first_ip AS ipavailabilityranges_first_ip, ipavailabilityranges.last_ip AS ipavailabilityranges_last_ip \nFROM ipavailabilityranges INNER JOIN ipallocationpools ON ipallocationpools.id = ipavailabilityranges.allocation_pool_id \nWHERE ipallocationpools.subnet_id = %s FOR UPDATE'] [parameters: ('131feb81-5f02-4167-ac32-340b22809e56',)]
2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource

** Affects: neutron
     Importance: Undecided
     Assignee: Kevin Benton (kevinbenton)
         Status: In Progress

** Changed in: neutron
     Assignee: (unassigned) => Kevin Benton (kevinbenton)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1475218

Title:
  oslo db retry decorator doesn't apply to add_router_interface

Status in neutron:
  In Progress

Bug description:
  http://logs.openstack.org/64/197564/17/check/gate-tempest-dsvm-
  neutron-full/1493b21/logs/screen-q-svc.txt.gz#_2015-07-16_07_38_42_299

  2015-07-16 07:38:42.299 ERROR neutron.api.v2.resource [req-38378c88-9223-434e-92b5-2124b1a6442a tempest-TestNetworkAdvancedServerOps-2072830200 tempest-TestNetworkAdvancedServerOps-1241923202] add_router_interface failed
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource Traceback (most recent call last):
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     result = method(request=request, **args)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 210, in _handle_action
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     return getattr(self._plugin, name)(*arg_list, **kwargs)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 291, in add_router_interface
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     context, router, interface_info['subnet_id'], device_owner)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 603, in _add_interface_by_subnet
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     'name': ''}}), [subnet], True
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1015, in create_port
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     result, mech_context = self._create_port_db(context, port)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 990, in _create_port_db
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     result = super(Ml2Plugin, self).create_port(context, port)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 837, in create_port
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     self.ipam.allocate_ips_for_port_and_store(context, port, port_id)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_non_pluggable_backend.py", line 205, in allocate_ips_for_port_and_store
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     ips = self._allocate_ips_for_port(context, port)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_non_pluggable_backend.py", line 396, in _allocate_ips_for_port
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     p['mac_address'])
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_non_pluggable_backend.py", line 329, in _allocate_fixed_ips
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     context, fixed['subnet_id'], fixed['ip_address'])
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/opt/stack/new/neutron/neutron/db/ipam_non_pluggable_backend.py", line 147, in _allocate_specific_ip
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     for ip_range in results:
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     return self._execute_and_instances(context)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2531, in _execute_and_instances
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     result = conn.execute(querycontext.statement, self._params)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     return meth(self, multiparams, params)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     return connection._execute_clauseelement(self, multiparams, params)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     compiled_sql, distilled_params
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     context)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     util.raise_from_cause(newraise, exc_info)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     reraise(type(exception), exception, tb=exc_tb)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     context)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     cursor.execute(statement, parameters)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 205, in execute
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     self.errorhandler(self, exc, value)
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource     raise errorclass, errorvalue
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource DBDeadlock: (_mysql_exceptions.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') [SQL: u'SELECT ipavailabilityranges.allocation_pool_id AS ipavailabilityranges_allocation_pool_id, ipavailabilityranges.first_ip AS ipavailabilityranges_first_ip, ipavailabilityranges.last_ip AS ipavailabilityranges_last_ip \nFROM ipavailabilityranges INNER JOIN ipallocationpools ON ipallocationpools.id = ipavailabilityranges.allocation_pool_id \nWHERE ipallocationpools.subnet_id = %s FOR UPDATE'] [parameters: ('131feb81-5f02-4167-ac32-340b22809e56',)]
  2015-07-16 07:38:42.299 9946 ERROR neutron.api.v2.resource

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1475218/+subscriptions
Follow ups

[Bug 1475218] Re: oslo db retry decorator doesn't apply to add_router_interface
From: Doug Hellmann, 2015-07-29