← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #82638

[Bug 1878632] [NEW] Race condition in subnet and segment delete: The segment is still bound with port(s)

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

The HOT template below may expose a race condition and by that make
stack deletion fail. On the neutron API this means that a segment delete
fails with "The segment is still bound with port(s)". The reproduction
uses a HOT template but I don't think this problem is Heat specific.
Rather I think it depends on quick succession of API calls, which Heat
does rather well.

Configuration:

ml2_conf.ini
[ml2]
mechanism_drivers = openvswitch,linuxbridge,sriovnicswitch,l2population
tenant_network_types = vxlan,vlan
[ml2_type_vlan]
network_vlan_ranges = physnet0:200:400,physnet1:200:400,...

sriov_agent.ini
[sriov_nic]
physical_device_mappings = physnet1:ens5

ovs_agent.ini
[ovs]
bridge_mappings = physnet0:br-physnet0,...

dhcp_agent.ini
[DEFAULT]
interface_driver = openvswitch

Reproduction:

# make this config live
$ sudo systemctl restart devstack@neutron-*

# create stack
$ openstack stack create -t ~/bug-segment-delete-does-not-delete-dhcp-ports.yml s0 --wait -f value -c stack_status
2020-05-14 14:23:23Z [s0]: CREATE_IN_PROGRESS  Stack CREATE started
2020-05-14 14:23:24Z [s0.net0]: CREATE_IN_PROGRESS  state changed
2020-05-14 14:23:24Z [s0.net0]: CREATE_COMPLETE  state changed
2020-05-14 14:23:24Z [s0.segment0]: CREATE_IN_PROGRESS  state changed
2020-05-14 14:23:25Z [s0.segment0]: CREATE_COMPLETE  state changed
2020-05-14 14:23:25Z [s0.subnet0]: CREATE_IN_PROGRESS  state changed
2020-05-14 14:23:26Z [s0.subnet0]: CREATE_COMPLETE  state changed
2020-05-14 14:23:26Z [s0]: CREATE_COMPLETE  Stack CREATE completed successfully
CREATE_COMPLETE

# wait until the dhcp port is created and it becomes ACTIVE
$ openstack stack resource show s0 net0 -f value -c physical_resource_id | xargs -r openstack port list --device-owner network:dhcp --network
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------+--------+
| ID                                   | Name | MAC Address       | Fixed IP Addresses                                                      | Status |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------+--------+
| 8cf8f188-5ea4-41b0-aa3a-fb8a8802888d |      | fa:16:3e:d1:59:1d | ip_address='10.0.4.2', subnet_id='376d2581-3765-4071-bb8e-41a684e235ff' | ACTIVE |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------+--------+

# the dhcp port is not created by heat of course                                                                                                                                                                     
$ openstack stack resource list s0
+---------------+--------------------------------------+----------------------+-----------------+----------------------+
| resource_name | physical_resource_id                 | resource_type        | resource_status | updated_time         |
+---------------+--------------------------------------+----------------------+-----------------+----------------------+
| net0          | 80ad5941-f581-4477-95e8-909c6140063d | OS::Neutron::Net     | CREATE_COMPLETE | 2020-05-14T14:28:03Z |
| subnet0       | 376d2581-3765-4071-bb8e-41a684e235ff | OS::Neutron::Subnet  | CREATE_COMPLETE | 2020-05-14T14:28:03Z |
| segment0      | 641c8c60-59c9-4972-bf82-3637f3e0f1cb | OS::Neutron::Segment | CREATE_COMPLETE | 2020-05-14T14:28:03Z |
+---------------+--------------------------------------+----------------------+-----------------+----------------------+

# stack delete fails
$ openstack stack delete s0 --yes --wait                                                                                                                                                                             
2020-05-14 14:37:10Z [s0]: DELETE_IN_PROGRESS  Stack DELETE started
2020-05-14 14:37:10Z [s0.subnet0]: DELETE_IN_PROGRESS  state changed
2020-05-14 14:37:11Z [s0.subnet0]: DELETE_COMPLETE  state changed
2020-05-14 14:37:11Z [s0.segment0]: DELETE_IN_PROGRESS  state changed
2020-05-14 14:37:11Z [s0.segment0]: DELETE_FAILED  ConflictException: resources.segment0: ConflictException: 409: Client Error for url: http://192.168.122.246:9696/v2.0/segments/641c8c60-59c9-4972-bf82-3637f3e0f1cb, Segment '641c8c60-59c9-4972-bf82-3637f3e0f1cb' cannot be deleted: The segment is still bou
2020-05-14 14:37:11Z [s0]: DELETE_FAILED  Resource DELETE failed: ConflictException: resources.segment0: ConflictException: 409: Client Error for url: http://192.168.122.246:9696/v2.0/segments/641c8c60-59c9-4972-bf82-3637f3e0f1cb, Segment '641c8c60-59c9-4972-bf82-3637f3e0f1cb' cannot be deleted:·

 Stack s0 DELETE_FAILED·

Unable to delete 1 of the 1 stacks.

# during that heat-engine logged this
$ sudo journalctl -u devstack@h-eng -f | egrep -w ERROR
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource Traceback (most recent call last):
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/resource.py", line 918, in _action_recorder
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     yield
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/resource.py", line 2051, in delete
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     *action_args)
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/scheduler.py", line 326, in wrapper
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     step = next(subtask)
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/resource.py", line 972, in action_handler_task
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     handler_data = handler(*args)
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/resources/openstack/neutron/segment.py", line 146, in handle_delete
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     self.client('openstack').network.delete_segment(self.resource_id)
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/network/v2/_proxy.py", line 3312, in delete_segment
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     self._delete(_segment.Segment, segment, ignore_missing=ignore_missing)
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/proxy.py", line 46, in check
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     return method(self, expected, actual, *args, **kwargs)
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/network/v2/_proxy.py", line 75, in _delete
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     rv = res.delete(self, if_revision=if_revision)
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/resource.py", line 1615, in delete
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     self._translate_response(response, has_body=False, **kwargs)
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/resource.py", line 1113, in _translate_response
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     exceptions.raise_from_response(response, error_message=error_message)
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/exceptions.py", line 236, in raise_from_response
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     http_status=http_status, request_id=request_id
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: http://192.168.122.246:9696/v2.0/segments/641c8c60-59c9-4972-bf82-3637f3e0f1cb, Segment '641c8c60-59c9-4972-bf82-3637f3e0f1cb' cannot be deleted: The segment is still bound with port(s) 8cf8f188-5ea4-41b0-aa3a-fb8a8802888d.
máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource

# a few seconds later a second delete succeeds
$ openstack stack delete s0 --yes --wait
2020-05-14 14:24:26Z [s0]: DELETE_IN_PROGRESS  Stack DELETE started

I have an idea what the root cause is. I'll describe that in a comment.

** Affects: neutron
     Importance: Medium
     Assignee: Bence Romsics (bence-romsics)
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1878632

Title:
  Race condition in subnet and segment delete: The segment is still
  bound with port(s)

Status in neutron:
  New

Bug description:
  The HOT template below may expose a race condition and by that make
  stack deletion fail. On the neutron API this means that a segment
  delete fails with "The segment is still bound with port(s)". The
  reproduction uses a HOT template but I don't think this problem is
  Heat specific. Rather I think it depends on quick succession of API
  calls, which Heat does rather well.

  Configuration:

  ml2_conf.ini
  [ml2]
  mechanism_drivers = openvswitch,linuxbridge,sriovnicswitch,l2population
  tenant_network_types = vxlan,vlan
  [ml2_type_vlan]
  network_vlan_ranges = physnet0:200:400,physnet1:200:400,...

  sriov_agent.ini
  [sriov_nic]
  physical_device_mappings = physnet1:ens5

  ovs_agent.ini
  [ovs]
  bridge_mappings = physnet0:br-physnet0,...

  dhcp_agent.ini
  [DEFAULT]
  interface_driver = openvswitch

  Reproduction:

  # make this config live
  $ sudo systemctl restart devstack@neutron-*

  # create stack
  $ openstack stack create -t ~/bug-segment-delete-does-not-delete-dhcp-ports.yml s0 --wait -f value -c stack_status
  2020-05-14 14:23:23Z [s0]: CREATE_IN_PROGRESS  Stack CREATE started
  2020-05-14 14:23:24Z [s0.net0]: CREATE_IN_PROGRESS  state changed
  2020-05-14 14:23:24Z [s0.net0]: CREATE_COMPLETE  state changed
  2020-05-14 14:23:24Z [s0.segment0]: CREATE_IN_PROGRESS  state changed
  2020-05-14 14:23:25Z [s0.segment0]: CREATE_COMPLETE  state changed
  2020-05-14 14:23:25Z [s0.subnet0]: CREATE_IN_PROGRESS  state changed
  2020-05-14 14:23:26Z [s0.subnet0]: CREATE_COMPLETE  state changed
  2020-05-14 14:23:26Z [s0]: CREATE_COMPLETE  Stack CREATE completed successfully
  CREATE_COMPLETE

  # wait until the dhcp port is created and it becomes ACTIVE
  $ openstack stack resource show s0 net0 -f value -c physical_resource_id | xargs -r openstack port list --device-owner network:dhcp --network
  +--------------------------------------+------+-------------------+-------------------------------------------------------------------------+--------+
  | ID                                   | Name | MAC Address       | Fixed IP Addresses                                                      | Status |
  +--------------------------------------+------+-------------------+-------------------------------------------------------------------------+--------+
  | 8cf8f188-5ea4-41b0-aa3a-fb8a8802888d |      | fa:16:3e:d1:59:1d | ip_address='10.0.4.2', subnet_id='376d2581-3765-4071-bb8e-41a684e235ff' | ACTIVE |
  +--------------------------------------+------+-------------------+-------------------------------------------------------------------------+--------+

  # the dhcp port is not created by heat of course                                                                                                                                                                     
  $ openstack stack resource list s0
  +---------------+--------------------------------------+----------------------+-----------------+----------------------+
  | resource_name | physical_resource_id                 | resource_type        | resource_status | updated_time         |
  +---------------+--------------------------------------+----------------------+-----------------+----------------------+
  | net0          | 80ad5941-f581-4477-95e8-909c6140063d | OS::Neutron::Net     | CREATE_COMPLETE | 2020-05-14T14:28:03Z |
  | subnet0       | 376d2581-3765-4071-bb8e-41a684e235ff | OS::Neutron::Subnet  | CREATE_COMPLETE | 2020-05-14T14:28:03Z |
  | segment0      | 641c8c60-59c9-4972-bf82-3637f3e0f1cb | OS::Neutron::Segment | CREATE_COMPLETE | 2020-05-14T14:28:03Z |
  +---------------+--------------------------------------+----------------------+-----------------+----------------------+

  # stack delete fails
  $ openstack stack delete s0 --yes --wait                                                                                                                                                                             
  2020-05-14 14:37:10Z [s0]: DELETE_IN_PROGRESS  Stack DELETE started
  2020-05-14 14:37:10Z [s0.subnet0]: DELETE_IN_PROGRESS  state changed
  2020-05-14 14:37:11Z [s0.subnet0]: DELETE_COMPLETE  state changed
  2020-05-14 14:37:11Z [s0.segment0]: DELETE_IN_PROGRESS  state changed
  2020-05-14 14:37:11Z [s0.segment0]: DELETE_FAILED  ConflictException: resources.segment0: ConflictException: 409: Client Error for url: http://192.168.122.246:9696/v2.0/segments/641c8c60-59c9-4972-bf82-3637f3e0f1cb, Segment '641c8c60-59c9-4972-bf82-3637f3e0f1cb' cannot be deleted: The segment is still bou
  2020-05-14 14:37:11Z [s0]: DELETE_FAILED  Resource DELETE failed: ConflictException: resources.segment0: ConflictException: 409: Client Error for url: http://192.168.122.246:9696/v2.0/segments/641c8c60-59c9-4972-bf82-3637f3e0f1cb, Segment '641c8c60-59c9-4972-bf82-3637f3e0f1cb' cannot be deleted:·

   Stack s0 DELETE_FAILED·

  Unable to delete 1 of the 1 stacks.

  # during that heat-engine logged this
  $ sudo journalctl -u devstack@h-eng -f | egrep -w ERROR
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource Traceback (most recent call last):
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/resource.py", line 918, in _action_recorder
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     yield
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/resource.py", line 2051, in delete
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     *action_args)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/scheduler.py", line 326, in wrapper
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     step = next(subtask)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/resource.py", line 972, in action_handler_task
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     handler_data = handler(*args)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/opt/stack/heat/heat/engine/resources/openstack/neutron/segment.py", line 146, in handle_delete
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     self.client('openstack').network.delete_segment(self.resource_id)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/network/v2/_proxy.py", line 3312, in delete_segment
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     self._delete(_segment.Segment, segment, ignore_missing=ignore_missing)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/proxy.py", line 46, in check
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     return method(self, expected, actual, *args, **kwargs)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/network/v2/_proxy.py", line 75, in _delete
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     rv = res.delete(self, if_revision=if_revision)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/resource.py", line 1615, in delete
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     self._translate_response(response, has_body=False, **kwargs)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/resource.py", line 1113, in _translate_response
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     exceptions.raise_from_response(response, error_message=error_message)
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource   File "/usr/local/lib/python3.6/dist-packages/openstack/exceptions.py", line 236, in raise_from_response
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource     http_status=http_status, request_id=request_id
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: http://192.168.122.246:9696/v2.0/segments/641c8c60-59c9-4972-bf82-3637f3e0f1cb, Segment '641c8c60-59c9-4972-bf82-3637f3e0f1cb' cannot be deleted: The segment is still bound with port(s) 8cf8f188-5ea4-41b0-aa3a-fb8a8802888d.
  máj 14 14:37:11 devstack1 heat-engine[12508]: ERROR heat.engine.resource

  # a few seconds later a second delete succeeds
  $ openstack stack delete s0 --yes --wait
  2020-05-14 14:24:26Z [s0]: DELETE_IN_PROGRESS  Stack DELETE started

  I have an idea what the root cause is. I'll describe that in a
  comment.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1878632/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next