yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1878632] Re: Race condition in subnet and segment delete: The segment is still bound with port(s)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1878632@xxxxxxxxxxxxxxxxxx>
Date: Thu, 09 Jul 2020 20:19:34 -0000
Reply-to: Bug 1878632 <1878632@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/729250
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=da45bbbff4e10e3d924ac3d85c9f382623708b11
Submitter: Zuul
Branch:    master

commit da45bbbff4e10e3d924ac3d85c9f382623708b11
Author: Bence Romsics <bence.romsics@xxxxxxxxx>
Date:   Tue May 19 13:21:52 2020 +0200

    Auto-delete dhcp ports on segment delete
    
    Subnet delete triggers dhcp port deletion but asynchronously,
    therefore in the condition described in the bug report we may
    get a conflict when deleting the segment too fast after the subnet.
    
    Here we follow the example of how we auto-delete ports in net delete.
    
    Please also find a fullstack test in Related-Change below.
    
    Change-Id: Iba02f5a2211b18c2deb9097daad6be5e7d21faf8
    Closes-Bug: #1878632
    Related-Change: https://review.opendev.org/728904


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1878632

Title:
  Race condition in subnet and segment delete: The segment is still
  bound with port(s)

Status in neutron:
  Fix Released

Bug description:
  The HOT template below may expose a race condition and by that make
  stack deletion fail. On the neutron API this means that a segment
  delete fails with "The segment is still bound with port(s)". The
  reproduction uses a HOT template but I don't think this problem is
  Heat specific. Rather I think it depends on quick succession of API
  calls, which Heat does rather well.

  Configuration:

  ml2_conf.ini
  [ml2]
  mechanism_drivers = openvswitch,linuxbridge,sriovnicswitch,l2population
  tenant_network_types = vxlan,vlan
  [ml2_type_vlan]
  network_vlan_ranges = physnet0:200:400,physnet1:200:400,...

  sriov_agent.ini
  [sriov_nic]
  physical_device_mappings = physnet1:ens5

  ovs_agent.ini
  [ovs]
  bridge_mappings = physnet0:br-physnet0,...

  dhcp_agent.ini
  [DEFAULT]
  interface_driver = openvswitch

  Reproduction:

  # make this config live
  $ sudo systemctl restart devstack@neutron-*

  # create stack
  $ openstack stack create -t ~/bug-segment-delete-does-not-delete-dhcp-ports.yml s0 --wait -f value -c stack_status
  2020-05-14 14:23:23Z [s0]: CREATE_IN_PROGRESS  Stack CREATE started
  2020-05-14 14:23:24Z [s0.net0]: CREATE_IN_PROGRESS  state changed
  2020-05-14 14:23:24Z [s0.net0]: CREATE_COMPLETE  state changed
  2020-05-14 14:23:24Z [s0.segment0]: CREATE_IN_PROGRESS  state changed
  2020-05-14 14:23:25Z [s0.segment0]: CREATE_COMPLETE  state changed
  2020-05-14 14:23:25Z [s0.subnet0]: CREATE_IN_PROGRESS  state changed
  2020-05-14 14:23:26Z [s0.subnet0]: CREATE_COMPLETE  state changed
  2020-05-14 14:23:26Z [s0]: CREATE_COMPLETE  Stack CREATE completed successfully
  CREATE_COMPLETE

  # wait until the dhcp port is created and it becomes ACTIVE
  $ openstack stack resource show s0 net0 -f value -c physical_resource_id | xargs -r openstack port list --device-owner network:dhcp --network
  +--------------------------------------+------+-------------------+-------------------------------------------------------------------------+--------+
  | ID                                   | Name | MAC Address       | Fixed IP Addresses                                                      | Status |
  +--------------------------------------+------+-------------------+-------------------------------------------------------------------------+--------+
  | 8cf8f188-5ea4-41b0-aa3a-fb8a8802888d |      | fa:16:3e:d1:59:1d | ip_address='10.0.4.2', subnet_id='376d2581-3765-4071-bb8e-41a684e235ff' | ACTIVE |
  +--------------------------------------+------+-------------------+-------------------------------------------------------------------------+--------+

  # the dhcp port is not created by heat of course                                                                                                                                                                     
  $ openstack stack resource list s0
  +---------------+--------------------------------------+----------------------+-----------------+----------------------+
  | resource_name | physical_resource_id                 | resource_type        | resource_status | updated_time         |
  +---------------+--------------------------------------+----------------------+-----------------+----------------------+
  | net0          | 80ad5941-f581-4477-95e8-909c6140063d | OS::Neutron::Net     | CREATE_COMPLETE | 2020-05-14T14:28:03Z |
  | subnet0       | 376d2581-3765-4071-bb8e-41a684e235ff | OS::Neutron::Subnet  | CREATE_COMPLETE | 2020-05-14T14:28:03Z |
  | segment0      | 641c8c60-59c9-4972-bf82-3637f3e0f1cb | OS::Neutron::Segment | CREATE_COMPLETE | 2020-05-14T14:28:03Z |
  +---------------+--------------------------------------+----------------------+-----------------+----------------------+

  # stack delete fails
  $ openstack stack delete s0 --yes --wait                                                                                                                                                                             
  2020-05-14 14:37:10Z [s0]: DELETE_IN_PROGRESS  Stack DELETE started
  2020-05-14 14:37:10Z [s0.subnet0]: DELETE_IN_PROGRESS  state changed
  2020-05-14 14:37:11Z [s0.subnet0]: DELETE_COMPLETE  state changed
  2020-05-14 14:37:11Z [s0.segment0]: DELETE_IN_PROGRESS  state changed
  2020-05-14 14:37:11Z [s0.segment0]: DELETE_FAILED  ConflictException: resources.segment0: ConflictException: 409: Client Error for url: http://192.168.122.246:9696/v2.0/segments/641c8c60-59c9-4972-bf82-3637f3e0f1cb, Segment '641c8c60-59c9-4972-bf82-3637f3e0f1cb' cannot be deleted: The segment is still bou
  2020-05-14 14:37:11Z [s0]: DELETE_FAILED  Resource DELETE failed: ConflictException: resources.segment0: ConflictException: 409: Client Error for url: http://192.168.122.246:9696/v2.0/segments/641c8c60-59c9-4972-bf82-3637f3e0f1cb, Segment '641c8c60-59c9-4972-bf82-3637f3e0f1cb' cannot be deleted:·

   Stack s0 DELETE_FAILED·

  Unable to delete 1 of the 1 stacks.

  # during that heat-engine logged this
  $ sudo journalctl -u devstack@h-eng -f | egrep -w ERROR
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource Traceback (most recent call last):
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/resource.py", line 918, in _action_recorder
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     yield
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/resource.py", line 2051, in delete
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     *action_args)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/scheduler.py", line 326, in wrapper
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     step = next(subtask)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/resource.py", line 972, in action_handler_task
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     handler_data = handler(*args)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/resources/openstack/neutron/segment.py", line 146, in handle_delete
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     self.client('openstack').network.delete_segment(self.resource_id)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/network/v2/_proxy.py", line 3312, in delete_segment
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     self._delete(_segment.Segment, segment, ignore_missing=ignore_missing)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/proxy.py", line 46, in check
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     return method(self, expected, actual, *args, **kwargs)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/network/v2/_proxy.py", line 75, in _delete
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     rv = res.delete(self, if_revision=if_revision)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/resource.py", line 1615, in delete
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     self._translate_response(response, has_body=False, **kwargs)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/resource.py", line 1113, in _translate_response
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     exceptions.raise_from_response(response, error_message=error_message)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/exceptions.py", line 236, in raise_from_response
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     http_status=http_status, request_id=request_id
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: http://192.168.122.246:9696/v2.0/segments/641c8c60-59c9-4972-bf82-3637f3e0f1cb, Segment '641c8c60-59c9-4972-bf82-3637f3e0f1cb' cannot be deleted: The segment is still bound with port(s) 8cf8f188-5ea4-41b0-aa3a-fb8a8802888d.
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource

  # a few seconds later a second delete succeeds
  $ openstack stack delete s0 --yes --wait
  2020-05-14 14:24:26Z [s0]: DELETE_IN_PROGRESS  Stack DELETE started

  I have an idea what the root cause is. I'll describe that in a
  comment.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1878632/+subscriptions
References

[Bug 1878632] [NEW] Race condition in subnet and segment delete: The segment is still bound with port(s)
From: Bence Romsics, 2020-05-14