yahoo-eng-team team mailing list archive

[Bence Romsics <bence.romsics@xxxxxxxxx>]

While I agree that it would be way more user friendly to give a
warning/error in the problematic API workflow that would entail some
cross project changes because today:

* nova does not know when an already bound port is added to a trunk
* neutron does not know if nova is supposed to auto-delete a port

That means neither nova nor neutron can detect the error condition in
itself.

Again, I believe changing the workflow to pre-create the parent port for
the server stops the problem described in this bug report completely.

So I'm setting this bug as Invalid. But let me know if you see other
alternatives.

** Changed in: neutron
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1878031

Title:
   Unable to delete an instance | Conflict: Port [port-id] is currently
  a parent port for trunk [trunk-id]

Status in neutron:
  Invalid

Bug description:
  When you create a trunk in Neutron you create a parent port for the
  trunk and attach the trunk to the parent.  Then subports can be
  created on the trunk.  When instances are created on the trunk, first
  a port is created and then an instance is associated with a free port.
  It looks to me that's this is the oversight in the logic.

  From the perspective of the code, the parent port looks like any other
  port attached to the trunk bridge.  It doesn't have an instance
  attached to it so it looks like it's not being used for anything
  (which is technically correct).  So it becomes an eligible port for an
  instance to bind to.  That is all fine and dandy until you go to
  delete the instance and you get the "Port [port-id] is currently a
  parent port for trunk [trunk-id]" exception just as happened here.
  Anecdotally, it's seems rare that an instance will actually bind to
  it, but that is what happened for the user in this case and I have had
  several pings over the past year about people in a similar state.

  I propose that when a port is made parent port for a trunk, that the
  trunk be established as the owner of the port.  That way it will be
  ineligible for instances seeking to bind to the port.

  See also old bug: https://bugs.launchpad.net/neutron/+bug/1700428

  Description of problem:

  Attempting to delete instance failed with error in nova-compute

  ~~~
  2020-03-04 09:52:46.257 1 WARNING nova.network.neutronv2.api [req-0dd45fe4-861c-46d3-a5ec-7db36352da58 02c6d1bc10fe4ffaa289c786cd09b146 695c417810ac460480055b074bc41817 - default default] [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4] Failed to delete port 991e4e50-481a-4ca6-9ea6-69f848c4ca9f for instance.: Conflict: Port 991e4e50-481a-4ca6-9ea6-69f848c4ca9f is currently a parent port for trunk 5800ee0f-b558-46cb-bb0b-92799dbe02cf.
  ~~~

  ~~~
  [stack@migration-host ~]$ openstack network trunk show 5800ee0f-b558-46cb-bb0b-92799dbe02cf
  +-----------------+--------------------------------------+
  | Field           | Value                                |
  +-----------------+--------------------------------------+
  | admin_state_up  | UP                                   |
  | created_at      | 2020-03-04T09:01:23Z                 |
  | description     |                                      |
  | id              | 5800ee0f-b558-46cb-bb0b-92799dbe02cf |
  | name            | WIN-TRUNK                            |
  | port_id         | 991e4e50-481a-4ca6-9ea6-69f848c4ca9f |
  | project_id      | 695c417810ac460480055b074bc41817     |
  | revision_number | 3                                    |
  | status          | ACTIVE                               |
  | sub_ports       |                                      |
  | tags            | []                                   |
  | tenant_id       | 695c417810ac460480055b074bc41817     |
  | updated_at      | 2020-03-04T10:20:46Z                 |
  +-----------------+--------------------------------------+

  
  [stack@migration-host ~]$ nova interface-list 2f9e3740-b425-4f00-a949-e1aacf2239c4
  +------------+--------------------------------------+--------------------------------------+--------------+-------------------+
  | Port State | Port ID                              | Net ID                               | IP addresses | MAC Addr          |
  +------------+--------------------------------------+--------------------------------------+--------------+-------------------+
  | DOWN       | 991e4e50-481a-4ca6-9ea6-69f848c4ca9f | 9be62c82-4274-48b4-bba0-39ccbdd5bb1b | 192.168.0.19 | fa:16:3e:0a:2b:9b |
  +------------+--------------------------------------+--------------------------------------+--------------+-------------------+
  [stack@migration-host ~]$ openstack port show 991e4e50-481a-4ca6-9ea6-69f848c4ca9f
  +-----------------------+---------------------------------------------------------------------------------------------------+
  | Field                 | Value                                                                                             |
  +-----------------------+---------------------------------------------------------------------------------------------------+
  | admin_state_up        | UP                                                                                                |
  | allowed_address_pairs |                                                                                                   |
  | binding_host_id       | cnibydc01cmp1.pl.cni.local                                                                        |
  | binding_profile       |                                                                                                   |
  | binding_vif_details   | port_filter='True'                                                                                |
  | binding_vif_type      | ovs                                                                                               |
  | binding_vnic_type     | normal                                                                                            |
  | created_at            | 2020-03-03T12:21:34Z                                                                              |
  | data_plane_status     | None                                                                                              |
  | description           |                                                                                                   |
  | device_id             | 2f9e3740-b425-4f00-a949-e1aacf2239c4                                                              |
  | device_owner          | compute:nested                                                                                    |
  | dns_assignment        | fqdn='host-192-168-0-19.openstacklocal.', hostname='host-192-168-0-19', ip_address='192.168.0.19' |
  | dns_name              |                                                                                                   |
  | extra_dhcp_opts       |                                                                                                   |
  | fixed_ips             | ip_address='192.168.0.19', subnet_id='efe88dec-368f-4e0e-b94a-5889915704a6'                       |
  | id                    | 991e4e50-481a-4ca6-9ea6-69f848c4ca9f                                                              |
  | ip_address            | None                                                                                              |
  | mac_address           | fa:16:3e:0a:2b:9b                                                                                 |
  | name                  |                                                                                                   |
  | network_id            | 9be62c82-4274-48b4-bba0-39ccbdd5bb1b                                                              |
  | option_name           | None                                                                                              |
  | option_value          | None                                                                                              |
  | port_security_enabled | True                                                                                              |
  | project_id            | 695c417810ac460480055b074bc41817                                                                  |
  | qos_policy_id         | None                                                                                              |
  | revision_number       | 26                                                                                                |
  | security_group_ids    |                                                                                                   |
  | status                | DOWN                                                                                              |
  | subnet_id             | None                                                                                              |
  | tags                  |                                                                                                   |
  | trunk_details         | {u'trunk_id': u'5800ee0f-b558-46cb-bb0b-92799dbe02cf', u'sub_ports': []}                          |
  | updated_at            | 2020-03-05T07:03:12Z                                                                              |
  +-----------------------+-----------------------------------------------------

  
  ~~~

  Detailed breakdown of an instance where this occurred.

  1. I do see 2 interfaces were attached during instance deletion.

      i. First interface which we still see as attached,port-id=991e4e50-481a-4ca6-9ea6-69f848c4ca9f.
         
         ~~~
         2020-03-04 09:52:44.425 1 INFO os_vif [req-0dd45fe4-861c-46d3-a5ec-7db36352da58 02c6d1bc10fe4ffaa289c786cd09b146 
         695c417810ac460480055b074bc41817 - default default] Successfully unplugged vif 
         VIFOpenVSwitch(active=True,address=fa:16:3e:0a:2b:9b,bridge_name='br-int',has_traffic_filtering=True,id=991e4e50-481a-4ca6-9ea6- 
         69f848c4ca9f,network=Network(9be62c82-4274-48b4-bba0- 
         39ccbdd5bb1b),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap991e4e50-48').
   
      ii. 2nd interface which seems deleted with port-id=64f6a222-9e9a-4494-bf3f-bc0ef72b9ec2 & network-id=90246055-5fce-4526-bdb0- 
          5e6466dd7946.
         
          ~~~
          2020-03-04 09:52:44.427 1 INFO os_vif [req-0dd45fe4-861c-46d3-a5ec-7db36352da58 02c6d1bc10fe4ffaa289c786cd09b146 
          695c417810ac460480055b074bc41817 - default default] Successfully unplugged vif 
          VIFOpenVSwitch(active=True,address=fa:16:3e:47:41:c8,bridge_name='br-int',has_traffic_filtering=True,id=64f6a222-9e9a-4494- 
          bf3f-bc0ef72b9ec2,network=Network(90246055-5fce-4526-bdb0-5e6466dd7946),plugin='ovs'
  	,port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap64f6a222-9e')
          ~~~

    2.  Neutron tried to detach & delete interface-64f6a222-9e9a-4494
  -bf3f-bc0ef72b9ec2.

       ~~~
       2020-03-04 09:52:44.580 1 INFO nova.compute.manager [req-0dd45fe4-861c-46d3-a5ec-7db36352da58 02c6d1bc10fe4ffaa289c786cd09b146 695c417810ac460480055b074bc41817 - default default] [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4] Took 1.45 seconds to destroy the instance on the hypervisor.

  2020-03-04 09:52:46.086 1 INFO nova.compute.manager [req-ac6ede7a-5b8f-45af-8239-d3003ad56f50 f378f367ea774fc7bf7e17c160141f85 0a0aed879b1149e18c8e80f5b1379984 - default default] [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4] Neutron deleted interface 64f6a222-9e9a-4494-bf3f-bc0ef72b9ec2; detaching it from the instance and deleting it from the info cache.
      ~~~~

    3. But at the next moment interface detach was failed since the instance was already removed from the hypervisor.
   
      ~~~
      2020-03-04 09:52:46.106 1 WARNING nova.compute.manager [req-ac6ede7a-5b8f-45af-8239-d3003ad56f50 f378f367ea774fc7bf7e17c160141f85 0a0aed879b1149e18c8e80f5b1379984 - default default] [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4] Detach interface failed, port_id=64f6a222-9e9a-4494-bf3f-bc0ef72b9ec2, reason: Instance 2f9e3740-b425-4f00-a949-e1aacf2239c4 could not be found.: InstanceNotFound: Instance 2f9e3740-b425-4f00-a949-e1aacf2239c4 could not be found.
      ~~~

    4. At the next moment, we have below error

     
    ~~~~
    2020-03-04 09:52:46.257 1 WARNING nova.network.neutronv2.api [req-0dd45fe4-861c-46d3-a5ec-7db36352da58 02c6d1bc10fe4ffaa289c786cd09b146 695c417810ac460480055b074bc41817 - default default] [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4] Failed to delete port 991e4e50-481a-4ca6-9ea6-69f848c4ca9f for instance.: Conflict: Port 991e4e50-481a-4ca6-9ea6-69f848c4ca9f is currently a parent port for trunk 5800ee0f-b558-46cb-bb0b-92799dbe02cf.
  ~~~



  [1]
  ~~~

  2020-03-04 09:52:46.257 1 WARNING nova.network.neutronv2.api [req-
  0dd45fe4-861c-46d3-a5ec-7db36352da58 02c6d1bc10fe4ffaa289c786cd09b146
  695c417810ac460480055b074bc41817 - default default] [instance:
  2f9e3740-b425-4f00-a949-e1aacf2239c4] Failed to delete port 991e4e50
  -481a-4ca6-9ea6-69f848c4ca9f for instance.: Conflict: Port 991e4e50
  -481a-4ca6-9ea6-69f848c4ca9f is currently a parent port for trunk
  5800ee0f-b558-46cb-bb0b-92799dbe02cf.

  Neutron server returns request_ids: ['req-dbd7a924-a9d2-4da5-aa41-b930580ad4f2']
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4] Traceback (most recent call last):
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1237, in _delete_ports

  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]     neutron.delete_port(port)
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 115, in wrapper
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]     ret = obj(*args, **kwargs)
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 803, in delete_port
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]     return self.delete(self.port_path % (port))
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 115,in wrapper
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]     ret = obj(*args, **kwargs)
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 350, in delete
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]     headers=headers, params=params)
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 115,in wrapper
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]     ret = obj(*args, **kwargs)
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]     headers=headers, params=params)
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 115,in wrapper

  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]     ret = obj(*args, **kwargs)
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]     self._handle_fault_response(status_code, replybody, resp)
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 115,in wrapper

  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]     ret = obj(*args, **kwargs)
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]     exception_handler_v20(status_code, error_body)
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4]     request_ids=request_ids)

  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4] Conflict: Port 991e4e50-481a-4ca6-9ea6-69f848c4ca9f is currently a parent port for trunk 5800ee0f-b558-46cb-bb0b-92799dbe02cf.
  2020-03-04 09:52:46.257 1 ERROR nova.network.neutronv2.api [instance: 2f9e3740-b425-4f00-a949-e1aacf2239c4] Neutron server returns request_ids: ['req-dbd7a924-a9d2-4da5-aa41-b930580ad4f2']
  ~~~

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1878031/+subscriptions