yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1217998] Re: CONSTRAINT routerl3agentbindings failure during gate tests

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Thu, 03 Oct 2013 07:55:03 -0000
Reply-to: Bug 1217998 <1217998@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: neutron
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1217998

Title:
  CONSTRAINT routerl3agentbindings failure during gate tests

Status in OpenStack Neutron (virtual network service):
  Fix Released

Bug description:
  http://logs.openstack.org/60/43760/3/check/gate-tempest-devstack-vm-
  neutron/dd1e380/logs/screen-q-svc.txt.gz

  Shows a constraint failure, which is almost always evidence of some
  kind of race/poor error handling:

  2013-08-27 18:45:06.141 29478 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/openstack/common/rpc/amqp.py", line 424, in _process_data
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp     **args)
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/common/rpc.py", line 44, in dispatch
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp     neutron_ctxt, version, method, namespace, **kwargs)
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/db/l3_rpc_base.py", line 47, in sync_routers
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp     plugin.auto_schedule_routers(context, host, router_ids)
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/db/agentschedulers_db.py", line 302, in auto_schedule_routers
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp     self, context, host, router_ids)
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 113, in auto_schedule_routers
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp     context.session.add(binding)
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 456, in __exit__
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp     self.commit()
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 368, in commit
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp     self._prepare_impl()
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 347, in _prepare_impl
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp     self.session.flush()
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp   File "/opt/stack/new/neutron/neutron/openstack/common/db/sqlalchemy/session.py", line 542, in _wrap
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp     raise exception.DBError(e)
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp DBError: (IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ovs_neutron`.`routerl3agentbindings`, CONSTRAINT `routerl3agentbindings_ibfk_2` FOREIGN KEY (`router_id`) REFERENCES `routers` (`id`) ON DELETE CASCADE)') 'INSERT INTO routerl3agentbindings (id, router_id, l3_agent_id) VALUES (%s, %s, %s)' ('c0771426-7e66-430e-beb9-4c3334e43039', '4dae479a-33a1-4d60-9a82-f0ee09cb9491', '039d8fe8-8993-4c9e-89a4-196dccb2878a')
  2013-08-27 18:45:06.141 29478 TRACE neutron.openstack.common.rpc.amqp 
  2013-08-27 18:45:06.142 29478 ERROR neutron.openstack.common.rpc.common [-] Returning exception (IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ovs_neutron`.`routerl3agentbindings`, CONSTRAINT `routerl3agentbindings_ibfk_2` FOREIGN KEY (`router_id`) REFERENCES `routers` (`id`) ON DELETE CASCADE)') 'INSERT INTO routerl3agentbindings (id, router_id, l3_agent_id) VALUES (%s, %s, %s)' ('c0771426-7e66-430e-beb9-4c3334e43039', '4dae479a-33a1-4d60-9a82-f0ee09cb9491', '039d8fe8-8993-4c9e-89a4-196dccb2878a') to caller
  2013-08-27 18:45:06.142 29478 ERROR neutron.openstack.common.rpc.common [-] ['Traceback (most recent call last):\n', '  File "/opt/stack/new/neutron/neutron/openstack/common/rpc/amqp.py", line 424, in _process_data\n    **args)\n', '  File "/opt/stack/new/neutron/neutron/common/rpc.py", line 44, in dispatch\n    neutron_ctxt, version, method, namespace, **kwargs)\n', '  File "/opt/stack/new/neutron/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n    result = getattr(proxyobj, method)(ctxt, **kwargs)\n', '  File "/opt/stack/new/neutron/neutron/db/l3_rpc_base.py", line 47, in sync_routers\n    plugin.auto_schedule_routers(context, host, router_ids)\n', '  File "/opt/stack/new/neutron/neutron/db/agentschedulers_db.py", line 302, in auto_schedule_routers\n    self, context, host, router_ids)\n', '  File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 113, in auto_schedule_routers\n    context.session.add(binding)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 456, in __exit__\n    self.commit()\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 368, in commit\n    self._prepare_impl()\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 347, in _prepare_impl\n    self.session.flush()\n', '  File "/opt/stack/new/neutron/neutron/openstack/common/db/sqlalchemy/session.py", line 542, in _wrap\n    raise exception.DBError(e)\n', "DBError: (IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ovs_neutron`.`routerl3agentbindings`, CONSTRAINT `routerl3agentbindings_ibfk_2` FOREIGN KEY (`router_id`) REFERENCES `routers` (`id`) ON DELETE CASCADE)') 'INSERT INTO routerl3agentbindings (id, router_id, l3_agent_id) VALUES (%s, %s, %s)' ('c0771426-7e66-430e-beb9-4c3334e43039', '4dae479a-33a1-4d60-9a82-f0ee09cb9491', '039d8fe8-8993-4c9e-89a4-196dccb2878a')\n"]

  Full gate logs are here:

  http://logs.openstack.org/60/43760/3/check/gate-tempest-devstack-vm-
  neutron/dd1e380/

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1217998/+subscriptions