yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1933638] Re: neutron with ovn returns Conflict on security group rules delete

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1933638@xxxxxxxxxxxxxxxxxx>
Date: Wed, 30 Jun 2021 05:03:16 -0000
Reply-to: Bug 1933638 <1933638@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/neutron/+/798718
Committed: https://opendev.org/openstack/neutron/commit/6a74cd76fd006d7c88575fc7ae39a3b499ac7d54
Submitter: "Zuul (22348)"
Branch:    master

commit 6a74cd76fd006d7c88575fc7ae39a3b499ac7d54
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Tue Jun 29 16:48:26 2021 +0000

    [OVN] Do not fail when processing SG rule deletion
    
    When a security group rule deletion command is issued, before executing
    the database operations, a "BEFORE_DELETE" event is raised.
    
    The OVN handler attending to this event should not fail if the security
    group rule does not exist; the database transaction [1]  will in case of
    not finding it, raising the correct exception and HTTP 404 error:
    
      Jun 29 16:58:28 dev20 neutron-server[8820]: INFO neutron.wsgi [None \
        req-1821ec9f-2439-420b-80eb-1138896de865 demo admin] 192.168.10.70 \
        "GET /v2.0/security-group-rules/missing_sg_rule_example HTTP/1.1" \
        status: 404 len: 348 time: 0.0352871
    
    [1]https://github.com/openstack/neutron/blob/6196c0873b4af9abd9055bf92478ff05ae090104/neutron/db/securitygroups_db.py#L858-L868
    
    Change-Id: I58f6e5b309e089f6681d2c4bbff4ff7fda96435f
    Closes-Bug: #1933638


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1933638

Title:
  neutron with ovn returns Conflict on security group rules delete

Status in neutron:
  Fix Released

Bug description:
  This issue was caught in an Octavia CI job
  (https://zuul.opendev.org/t/openstack/build/9cb24aa49cbb47e6abeb580e5d5ec6f0/logs)

  During the deletion of a load balancer, Octavia deletes security group
  rules in neutron. It seems that Octavia is trying to delete many times
  the same security group rule, and it sometimes receives a Conflict
  exception while the exception message explains that the security group
  rule doesn't exist:

  
  Jun 22 12:36:00.226969 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 neutron-server[88368]: INFO neutron.wsgi [None req-e2baf119-3462-4af0-8b08-1e35cf0ba6d2 admin admin] 199.19.213.147,199.19.213.147 "DELETE /v2.0/security-group-rules/ec7d4cb6-a872-4709-854a-efaca7527822 HTTP/1.1" status: 204  len: 173 time: 0.0580175
  Jun 22 12:36:00.228298 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 neutron-server[88367]: INFO neutron.api.v2.resource [None req-42ecd2c1-85d8-4ea6-b9ec-b98af458a8aa admin admin] delete failed (client error): The resource could not be found.
  Jun 22 12:36:00.229361 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 neutron-server[88367]: INFO neutron.wsgi [None req-42ecd2c1-85d8-4ea6-b9ec-b98af458a8aa admin admin] 199.19.213.147,199.19.213.147 "DELETE /v2.0/security-group-rules/ec7d4cb6-a872-4709-854a-efaca7527822 HTTP/1.1" status: 404  len: 361 time: 0.0507255
  Jun 22 12:36:00.230639 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 neutron-server[88368]: DEBUG neutron.api.rpc.handlers.resources_rpc [None req-e2baf119-3462-4af0-8b08-1e35cf0ba6d2 admin admin] Pushing event deleted for resources: {'SecurityGroupRule': ['ID=ec7d4cb6-a872-4709-854a-efaca7527822,revision_number=None']} {{(pid=88368) push /opt/stack/neutron/neutron/api/rpc/handlers/resources_rpc.py:237}}
  Jun 22 12:36:00.230973 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 neutron-server[88367]: DEBUG neutron_lib.callbacks.manager [None req-92d0b84a-6fce-40e8-8c6c-f08c4c362481 admin admin] Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._process_sg_rule_notification-750270 raised Security group rule ec7d4cb6-a872-4709-854a-efaca7527822 does not exist {{(pid=88367) _notify_loop /usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py:209}}
  Jun 22 12:36:00.231248 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 neutron-server[88367]: DEBUG neutron_lib.callbacks.manager [None req-92d0b84a-6fce-40e8-8c6c-f08c4c362481 admin admin] Notify callbacks [] for security_group_rule, abort_delete {{(pid=88367) _notify_loop /usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py:192}}
  Jun 22 12:36:00.231444 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 neutron-server[88368]: DEBUG oslo_concurrency.lockutils [None req-e2baf119-3462-4af0-8b08-1e35cf0ba6d2 admin admin] Lock "event-dispatch" released by "neutron.plugins.ml2.ovo_rpc._ObjectChangeHandler.dispatch_events" :: held 0.008s {{(pid=88368) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:367}}
  Jun 22 12:36:00.231819 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 neutron-server[88367]: INFO neutron.api.v2.resource [None req-92d0b84a-6fce-40e8-8c6c-f08c4c362481 admin admin] delete failed (client error): There was a conflict when trying to complete your request.
  Jun 22 12:36:00.232958 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 neutron-server[88367]: INFO neutron.wsgi [None req-92d0b84a-6fce-40e8-8c6c-f08c4c362481 admin admin] 199.19.213.147,199.19.213.147 "DELETE /v2.0/security-group-rules/ec7d4cb6-a872-4709-854a-efaca7527822 HTTP/1.1" status: 409  len: 588 time: 0.0547035


  In the octavia logs, we received:

  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last):
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker   File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker     result = task.execute(**arguments)
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker   File "/opt/stack/octavia/octavia/controller/worker/v1/tasks/network_tasks.py", line 519, in execute
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker     self.network_driver.update_vip(loadbalancer, for_delete=True)
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker   File "/opt/stack/octavia/octavia/network/drivers/neutron/allowed_address_pairs.py", line 622, in update_vip
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker     self._update_security_group_rules(load_balancer,
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker   File "/opt/stack/octavia/octavia/network/drivers/neutron/allowed_address_pairs.py", line 207, in _update_security_group_rules
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker     self.neutron_client.delete_security_group_rule(rule_id)
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker   File "/usr/local/lib/python3.8/dist-packages/neutronclient/v2_0/client.py", line 1043, in delete_security_group_rule
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker     return self.delete(self.security_group_rule_path %
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker   File "/usr/local/lib/python3.8/dist-packages/neutronclient/v2_0/client.py", line 352, in delete
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker     return self.retry_request("DELETE", action, body=body,
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker   File "/usr/local/lib/python3.8/dist-packages/neutronclient/v2_0/client.py", line 333, in retry_request
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker     return self.do_request(method, action, body=body,
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker   File "/usr/local/lib/python3.8/dist-packages/neutronclient/v2_0/client.py", line 297, in do_request
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker     self._handle_fault_response(status_code, replybody, resp)
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker   File "/usr/local/lib/python3.8/dist-packages/neutronclient/v2_0/client.py", line 272, in _handle_fault_response
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker     exception_handler_v20(status_code, error_body)
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker   File "/usr/local/lib/python3.8/dist-packages/neutronclient/v2_0/client.py", line 90, in exception_handler_v20
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker     raise client_exc(message=error_message,
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker neutronclient.common.exceptions.Conflict: Security Group Rule ec7d4cb6-a872-4709-854a-efaca7527822 cannot perform before_delete due to Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._process_sg_rule_notification-750270 failed with "Security group rule ec7d4cb6-a872-4709-854a-efaca7527822 does not exist".
  Jun 22 12:36:00.263676 nested-virt-ubuntu-focal-vexxhost-ca-ymq-1-0025224760 octavia-worker[127003]: ERROR octavia.controller.worker.v1.controller_worker Neutron server returns request_ids: ['req-92d0b84a-6fce-40e8-8c6c-f08c4c362481']

  
  I think we should have received a NotFound exception (that we catch in Octavia) instead of this Conflict exception.

  
  This issue has appeared after the switch to ML2/OVN.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1933638/+subscriptions
References

[Bug 1933638] [NEW] neutronclient returns Conflict on security group rules delete
From: Gregory Thiemonge, 2021-06-25