yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2084977] [NEW] Deleting a security group fails with "Cannot find Port_Group with name=pg_<id>" when neutron-api is run by httpd + mod_wsgi

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Takashi Kajinami <2084977@xxxxxxxxxxxxxxxxxx>
Date: Sat, 19 Oct 2024 13:43:23 -0000
Reply-to: Bug 2084977 <2084977@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

In Puppet OpenStack project we've been attempting to switch neutron
deployment from standalone eventlet server to api run by httpd +
mod_wsgi and separate worker processes.

However we've seeing the strange failure consistently and I've not yet
able to identify the cause, so am opening this but in case anyone can
help dig into the issue.

The current problem is that deleting a security group fails because
neutron-api can't find the port group in ovn db.

Example:
 build: https://zuul.opendev.org/t/openstack/build/3f8cb2013ad143589979befc43234c9c
 tempest report: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ee4/890924/30/check/puppet-openstack-integration-7-scenario005-tempest-centos-9-stream/ee49397/logs/testr_results.html

The tempest failure we actually see is;

tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'SecurityGroupInUse', 'message': 'Security Group None cannot perform precommit_delete due to Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._delete_security_group_precommit-623828 failed with "Cannot find Port_Group with name=pg_8f86d456_755d_42fd_8951_cd21fea8377e".', 'detail': ''}


However from neutron api log I see the pg was created successfully.
https://97af450d4496db857e42-e03b399dba5d60e417a0a61de1589ca2.ssl.cf5.rackcdn.com/890924/30/check/puppet-openstack-integration-7-scenario003-tempest-centos-9-stream/3f8cb20/logs/neutron/app.txt

```
2024-10-18 13:08:40.508 76845 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-36a5b37b-170f-471a-a214-dcba3dd0de19 - - - - - -] Running txn n=2 command(idx=0): PgAddCommand(_result=0835e7e8-d620-4986-8cbc-fda0b360b97a, name=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1, may_exist=False, columns={'acls': [], 'external_ids': {'neutron:security_group_id': '0d20c3b7-d0cd-4339-988c-ac3b2a49bde1'}}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
2024-10-18 13:08:40.508 76845 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-36a5b37b-170f-471a-a214-dcba3dd0de19 - - - - - -] Running txn n=2 command(idx=1): PgAclAddCommand(_result=4986db11-f42e-4f70-8237-41627c92affc, entity=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1, direction=to-lport, priority=1002, match=outport == @pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1 && ip6 && ip6.src == $pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1_ip6, action=allow-related, log=False, may_exist=True, severity=[], name=[], meter=None, external_ids={'neutron:security_group_rule_id': '87e8cfd3-9ee5-44cd-98b6-36b51077cf0b'}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
2024-10-18 13:08:40.509 76845 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-36a5b37b-170f-471a-a214-dcba3dd0de19 - - - - - -] Running txn n=2 command(idx=2): PgAclAddCommand(_result=c1470e9f-6062-4a75-8d6a-5d34a0dddd5d, entity=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1, direction=to-lport, priority=1002, match=outport == @pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1 && ip4 && ip4.src == $pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1_ip4, action=allow-related, log=False, may_exist=True, severity=[], name=[], meter=None, external_ids={'neutron:security_group_rule_id': 'c08c43af-a7dc-435f-b7d5-d034983f3944'}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
2024-10-18 13:08:40.509 76845 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-36a5b37b-170f-471a-a214-dcba3dd0de19 - - - - - -] Running txn n=2 command(idx=3): PgAclAddCommand(_result=8fe9ed5f-ef77-4f51-8a17-d9a061dafa49, entity=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1, direction=from-lport, priority=1002, match=inport == @pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1 && ip4, action=allow-related, log=False, may_exist=True, severity=[], name=[], meter=None, external_ids={'neutron:security_group_rule_id': 'c4349c63-0dce-4791-998f-290247a9d368'}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
2024-10-18 13:08:40.510 76845 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-36a5b37b-170f-471a-a214-dcba3dd0de19 - - - - - -] Running txn n=2 command(idx=4): PgAclAddCommand(_result=c491b5de-f1bf-4ab9-ad26-cb7f39969300, entity=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1, direction=from-lport, priority=1002, match=inport == @pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1 && ip6, action=allow-related, log=False, may_exist=True, severity=[], name=[], meter=None, external_ids={'neutron:security_group_rule_id': 'eaea975a-9ac8-416b-9375-574a642b5d61'}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
...
2024-10-18 13:08:40.574 76845 DEBUG neutron_lib.db.resource_extend [None req-4089cacb-ccc8-41ab-be87-6dacc03f09c6 - - - - - -] It took 0.44 seconds to run function 'neutron_lib.db.resource_extend.apply_funcs' wrapper /usr/lib/python3.9/site-packages/oslo_utils/timeutils.py:336
2024-10-18 13:08:40.608 76845 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-36a5b37b-170f-471a-a214-dcba3dd0de19 - - - - - -] There are no commands to commit do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78
2024-10-18 13:08:40.610 76845 INFO neutron.db.ovn_revision_numbers_db [None req-98423aa8-628e-4b89-9371-1bb4728052e2 2b9e784b5daf4f9f9db8827ed7d9d640 30b043d8c61c4173bcf52e6b28acbbff - - default default] Successfully bumped revision number for resource 0d20c3b7-d0cd-4339-988c-ac3b2a49bde1 (type: security_groups) to 1
```

However the pg is not found later.

```
2024-10-18 13:08:40.786 76846 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): PgAclDelCommand(_result=None, entity=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 1002), ('match', '=', 'outport == @pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1 && ip6 && ip6.src == $pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1_ip6')]) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
2024-10-18 13:08:40.791 76846 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-290a49d3-5ab3-4954-8b6f-6936bf2cf779 2b9e784b5daf4f9f9db8827ed7d9d640 30b043d8c61c4173bcf52e6b28acbbff - - default default] Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
    command.run_idl(txn)
  File "/usr/lib/python3.9/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 209, in run_idl
    entity = self.api.lookup(self.lookup_table, self.entity)
  File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 184, in lookup
    return self._lookup(table, record)
  File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 235, in _lookup
    row = idlutils.row_by_value(self, rl.table, rl.column, record)
  File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
    raise RowNotFound(table=table, col=column, match=match)
ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1
```

I don't see anything suspicious from access log generated by apache.
```
::1 - - [18/Oct/2024:13:08:38 +0000] "GET /v2.0/security-groups?tenant_id=65999afc449642ea94850cde0d1a5e13&name=default HTTP/1.1" 200 2925 "-" "python-urllib3/1.26.5"
::1 - - [18/Oct/2024:13:08:40 +0000] "DELETE /v2.0/security-groups/0d20c3b7-d0cd-4339-988c-ac3b2a49bde1 HTTP/1.1" 409 349 "-" "python-urllib3/1.26.5"
```

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2084977

Title:
  Deleting a security group fails with "Cannot find Port_Group with
  name=pg_<id>" when neutron-api is run by httpd + mod_wsgi

Status in neutron:
  New

Bug description:
  In Puppet OpenStack project we've been attempting to switch neutron
  deployment from standalone eventlet server to api run by httpd +
  mod_wsgi and separate worker processes.

  However we've seeing the strange failure consistently and I've not yet
  able to identify the cause, so am opening this but in case anyone can
  help dig into the issue.

  The current problem is that deleting a security group fails because
  neutron-api can't find the port group in ovn db.

  Example:
   build: https://zuul.opendev.org/t/openstack/build/3f8cb2013ad143589979befc43234c9c
   tempest report: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ee4/890924/30/check/puppet-openstack-integration-7-scenario005-tempest-centos-9-stream/ee49397/logs/testr_results.html

  The tempest failure we actually see is;

  tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'type': 'SecurityGroupInUse', 'message': 'Security Group None cannot perform precommit_delete due to Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._delete_security_group_precommit-623828 failed with "Cannot find Port_Group with name=pg_8f86d456_755d_42fd_8951_cd21fea8377e".', 'detail': ''}

  
  However from neutron api log I see the pg was created successfully.
  https://97af450d4496db857e42-e03b399dba5d60e417a0a61de1589ca2.ssl.cf5.rackcdn.com/890924/30/check/puppet-openstack-integration-7-scenario003-tempest-centos-9-stream/3f8cb20/logs/neutron/app.txt

  ```
  2024-10-18 13:08:40.508 76845 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-36a5b37b-170f-471a-a214-dcba3dd0de19 - - - - - -] Running txn n=2 command(idx=0): PgAddCommand(_result=0835e7e8-d620-4986-8cbc-fda0b360b97a, name=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1, may_exist=False, columns={'acls': [], 'external_ids': {'neutron:security_group_id': '0d20c3b7-d0cd-4339-988c-ac3b2a49bde1'}}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
  2024-10-18 13:08:40.508 76845 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-36a5b37b-170f-471a-a214-dcba3dd0de19 - - - - - -] Running txn n=2 command(idx=1): PgAclAddCommand(_result=4986db11-f42e-4f70-8237-41627c92affc, entity=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1, direction=to-lport, priority=1002, match=outport == @pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1 && ip6 && ip6.src == $pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1_ip6, action=allow-related, log=False, may_exist=True, severity=[], name=[], meter=None, external_ids={'neutron:security_group_rule_id': '87e8cfd3-9ee5-44cd-98b6-36b51077cf0b'}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
  2024-10-18 13:08:40.509 76845 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-36a5b37b-170f-471a-a214-dcba3dd0de19 - - - - - -] Running txn n=2 command(idx=2): PgAclAddCommand(_result=c1470e9f-6062-4a75-8d6a-5d34a0dddd5d, entity=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1, direction=to-lport, priority=1002, match=outport == @pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1 && ip4 && ip4.src == $pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1_ip4, action=allow-related, log=False, may_exist=True, severity=[], name=[], meter=None, external_ids={'neutron:security_group_rule_id': 'c08c43af-a7dc-435f-b7d5-d034983f3944'}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
  2024-10-18 13:08:40.509 76845 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-36a5b37b-170f-471a-a214-dcba3dd0de19 - - - - - -] Running txn n=2 command(idx=3): PgAclAddCommand(_result=8fe9ed5f-ef77-4f51-8a17-d9a061dafa49, entity=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1, direction=from-lport, priority=1002, match=inport == @pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1 && ip4, action=allow-related, log=False, may_exist=True, severity=[], name=[], meter=None, external_ids={'neutron:security_group_rule_id': 'c4349c63-0dce-4791-998f-290247a9d368'}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
  2024-10-18 13:08:40.510 76845 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-36a5b37b-170f-471a-a214-dcba3dd0de19 - - - - - -] Running txn n=2 command(idx=4): PgAclAddCommand(_result=c491b5de-f1bf-4ab9-ad26-cb7f39969300, entity=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1, direction=from-lport, priority=1002, match=inport == @pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1 && ip6, action=allow-related, log=False, may_exist=True, severity=[], name=[], meter=None, external_ids={'neutron:security_group_rule_id': 'eaea975a-9ac8-416b-9375-574a642b5d61'}) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
  ...
  2024-10-18 13:08:40.574 76845 DEBUG neutron_lib.db.resource_extend [None req-4089cacb-ccc8-41ab-be87-6dacc03f09c6 - - - - - -] It took 0.44 seconds to run function 'neutron_lib.db.resource_extend.apply_funcs' wrapper /usr/lib/python3.9/site-packages/oslo_utils/timeutils.py:336
  2024-10-18 13:08:40.608 76845 DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-36a5b37b-170f-471a-a214-dcba3dd0de19 - - - - - -] There are no commands to commit do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78
  2024-10-18 13:08:40.610 76845 INFO neutron.db.ovn_revision_numbers_db [None req-98423aa8-628e-4b89-9371-1bb4728052e2 2b9e784b5daf4f9f9db8827ed7d9d640 30b043d8c61c4173bcf52e6b28acbbff - - default default] Successfully bumped revision number for resource 0d20c3b7-d0cd-4339-988c-ac3b2a49bde1 (type: security_groups) to 1
  ```

  However the pg is not found later.

  ```
  2024-10-18 13:08:40.786 76846 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): PgAclDelCommand(_result=None, entity=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 1002), ('match', '=', 'outport == @pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1 && ip6 && ip6.src == $pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1_ip6')]) do_commit /usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89
  2024-10-18 13:08:40.791 76846 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-290a49d3-5ab3-4954-8b6f-6936bf2cf779 2b9e784b5daf4f9f9db8827ed7d9d640 30b043d8c61c4173bcf52e6b28acbbff - - default default] Traceback (most recent call last):
    File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
      txn.results.put(txn.do_commit())
    File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit
      command.run_idl(txn)
    File "/usr/lib/python3.9/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 209, in run_idl
      entity = self.api.lookup(self.lookup_table, self.entity)
    File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 184, in lookup
      return self._lookup(table, record)
    File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 235, in _lookup
      row = idlutils.row_by_value(self, rl.table, rl.column, record)
    File "/usr/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
      raise RowNotFound(table=table, col=column, match=match)
  ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_0d20c3b7_d0cd_4339_988c_ac3b2a49bde1
  ```

  I don't see anything suspicious from access log generated by apache.
  ```
  ::1 - - [18/Oct/2024:13:08:38 +0000] "GET /v2.0/security-groups?tenant_id=65999afc449642ea94850cde0d1a5e13&name=default HTTP/1.1" 200 2925 "-" "python-urllib3/1.26.5"
  ::1 - - [18/Oct/2024:13:08:40 +0000] "DELETE /v2.0/security-groups/0d20c3b7-d0cd-4339-988c-ac3b2a49bde1 HTTP/1.1" 409 349 "-" "python-urllib3/1.26.5"
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2084977/+subscriptions