yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1892496] [NEW] 500 on SG deletion: Cannot delete or update a parent row

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Michal Dulko <1892496@xxxxxxxxxxxxxxxxxx>
Date: Fri, 21 Aug 2020 12:32:55 -0000
Reply-to: Bug 1892496 <1892496@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

In kuryr-kubernetes gate runing NetworkPolicy tests from K8s itself we
started to see Neutron returns 500 on SG delete requests. This is found
in q-svc logs:

Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource [None req-14b3a3ce-74d3-4b84-a458-57acf8e74c0e service kuryr] delete failed: No details.: oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`neutron`.`securitygroupportbindings`, CONSTRAINT `securitygroupportbindings_ibfk_2` FOREIGN KEY (`security_group_id`) REFERENCES `securitygroups` (`id`))')
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: [SQL: DELETE FROM securitygroups WHERE securitygroups.id = %(id)s]
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: [parameters: {'id': 'f62aa0c2-c513-4794-b48a-003a388d0ba0'}]
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: (Background on this error at: http://sqlalche.me/e/13/gkpj)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource Traceback (most recent call last):
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     cursor, statement, parameters, context
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     cursor.execute(statement, parameters)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 163, in execute
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     result = self._query(query)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 321, in _query
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     conn.query(q)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 505, in query
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 724, in _read_query_result
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     result.read()
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 1069, in read
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     first_packet = self.connection._read_packet()
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 676, in _read_packet
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     packet.raise_for_error()
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/protocol.py", line 223, in raise_for_error
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     err.raise_mysql_exception(self._data)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     raise errorclass(errno, errval)
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource pymysql.err.IntegrityError: (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`neutron`.`securitygroupportbindings`, CONSTRAINT `securitygroupportbindings_ibfk_2` FOREIGN KEY (`security_group_id`) REFERENCES `securitygroups` (`id`))')
Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource

Full log: https://508cf8e3a004102dbdaa-
fe9c830c883bfe814f1bba253125965d.ssl.cf1.rackcdn.com/747233/2/check
/kuryr-kubernetes-e2e-np-containerized-ovn-provider-
ovn/c4fdef2/controller/logs/screen-q-svc.txt

It seems like we do not run any POST request for SG rules after that
DELETE is called, but honestly I wouldn't be too surprised it may happen
as well. Besides that I'm not exactly sure what causes it.

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

  In kuryr-kubernetes gate runing NetworkPolicy tests from K8s itself we
  started to see Neutron returns 500 on SG delete requests. This is found
  in q-svc logs:
  
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource [None req-14b3a3ce-74d3-4b84-a458-57acf8e74c0e service kuryr] delete failed: No details.: oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`neutron`.`securitygroupportbindings`, CONSTRAINT `securitygroupportbindings_ibfk_2` FOREIGN KEY (`security_group_id`) REFERENCES `securitygroups` (`id`))')
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: [SQL: DELETE FROM securitygroups WHERE securitygroups.id = %(id)s]
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: [parameters: {'id': 'f62aa0c2-c513-4794-b48a-003a388d0ba0'}]
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: (Background on this error at: http://sqlalche.me/e/13/gkpj)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource Traceback (most recent call last):
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     cursor, statement, parameters, context
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     cursor.execute(statement, parameters)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 163, in execute
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     result = self._query(query)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 321, in _query
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     conn.query(q)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 505, in query
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 724, in _read_query_result
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     result.read()
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 1069, in read
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     first_packet = self.connection._read_packet()
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 676, in _read_packet
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     packet.raise_for_error()
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/protocol.py", line 223, in raise_for_error
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     err.raise_mysql_exception(self._data)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     raise errorclass(errno, errval)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource pymysql.err.IntegrityError: (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`neutron`.`securitygroupportbindings`, CONSTRAINT `securitygroupportbindings_ibfk_2` FOREIGN KEY (`security_group_id`) REFERENCES `securitygroups` (`id`))')
- Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource 
+ Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource
  
  Full log: https://508cf8e3a004102dbdaa-
  fe9c830c883bfe814f1bba253125965d.ssl.cf1.rackcdn.com/747233/2/check
  /kuryr-kubernetes-e2e-np-containerized-ovn-provider-
- ovn/c4fdef2/controller/logs/kubernetes/pod_logs/kube-system-kuryr-
- controller-5d559448c4-g5xst-controller-prev.txt
+ ovn/c4fdef2/controller/logs/screen-q-svc.txt
  
  It seems like we do not run any POST request for SG rules after that
  DELETE is called, but honestly I wouldn't be too surprised it may happen
  as well. Besides that I'm not exactly sure what causes it.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1892496

Title:
  500 on SG deletion: Cannot delete or update a parent row

Status in neutron:
  New

Bug description:
  In kuryr-kubernetes gate runing NetworkPolicy tests from K8s itself we
  started to see Neutron returns 500 on SG delete requests. This is
  found in q-svc logs:

  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource [None req-14b3a3ce-74d3-4b84-a458-57acf8e74c0e service kuryr] delete failed: No details.: oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`neutron`.`securitygroupportbindings`, CONSTRAINT `securitygroupportbindings_ibfk_2` FOREIGN KEY (`security_group_id`) REFERENCES `securitygroups` (`id`))')
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: [SQL: DELETE FROM securitygroups WHERE securitygroups.id = %(id)s]
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: [parameters: {'id': 'f62aa0c2-c513-4794-b48a-003a388d0ba0'}]
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: (Background on this error at: http://sqlalche.me/e/13/gkpj)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource Traceback (most recent call last):
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     cursor, statement, parameters, context
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     cursor.execute(statement, parameters)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 163, in execute
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     result = self._query(query)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 321, in _query
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     conn.query(q)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 505, in query
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 724, in _read_query_result
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     result.read()
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 1069, in read
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     first_packet = self.connection._read_packet()
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 676, in _read_packet
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     packet.raise_for_error()
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/protocol.py", line 223, in raise_for_error
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     err.raise_mysql_exception(self._data)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource   File "/usr/local/lib/python3.6/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource     raise errorclass(errno, errval)
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource pymysql.err.IntegrityError: (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`neutron`.`securitygroupportbindings`, CONSTRAINT `securitygroupportbindings_ibfk_2` FOREIGN KEY (`security_group_id`) REFERENCES `securitygroups` (`id`))')
  Aug 20 19:43:56.358683 ubuntu-bionic-rax-iad-0019371817 neutron-server[10089]: ERROR neutron.api.v2.resource

  Full log: https://508cf8e3a004102dbdaa-
  fe9c830c883bfe814f1bba253125965d.ssl.cf1.rackcdn.com/747233/2/check
  /kuryr-kubernetes-e2e-np-containerized-ovn-provider-
  ovn/c4fdef2/controller/logs/screen-q-svc.txt

  It seems like we do not run any POST request for SG rules after that
  DELETE is called, but honestly I wouldn't be too surprised it may
  happen as well. Besides that I'm not exactly sure what causes it.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1892496/+subscriptions