yahoo-eng-team team mailing list archive

[OpenStack Infra <1992109@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/860649
Committed: https://opendev.org/openstack/neutron/commit/f22aa5dfddae9bd31fbb4fca8aed176e9913d34a
Submitter: "Zuul (22348)"
Branch:    master

commit f22aa5dfddae9bd31fbb4fca8aed176e9913d34a
Author: Arnaud Morin <arnaud.morin@xxxxxxxxxxxx>
Date:   Fri Oct 7 11:05:17 2022 +0200

    Discard port with ofport -1 in _get_ofport_moves
    
    When libvirt (nova) detach a port on OVS bridge, two events are sent:
    * one event with 2 actions "old" and "new": a change on ofport (from a
      regular value to -1)
    * a second event with action "delete"
    
    If, for some reason, the second event is delayed, the rpc_loop iteration
    will consider this port as "updated" instead of "deleted".
    But, because ofport == -1, the port update will be discarded, and
    finally removed from port_info["current"].
    
    As a result, on next iteration, the deletion wont be performed.
    
    Most of the time, we endup with some leftovers (like openflow rules,
    etc.)
    
    The purpose of this patch is very simple, when looping over ports in
    _get_ofport_moves, we will discards the ports that have ofport == -1, so
    the port will not be considered as updated and next iteration will be
    able to delete it correctly.
    
    Closes-Bug: #1992109
    
    Change-Id: Ib4a7183867e1b21810b6915a475a234278bf884c
    Signed-off-by: Arnaud Morin <arnaud.morin@xxxxxxxxxxxx>


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1992109

Title:
  Possible race condition when port unplugged from ovs

Status in neutron:
  Fix Released

Bug description:
  A possible race condition can occur when nova unplug a port on
  integration bridge (when shelving an instance e.g.)

  On such action, openvswitch is sending 2 events to neutron:

  2022-10-07 01:00:31.790 214734 DEBUG neutron.agent.common.async_process [-] Output received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: {"data":[["650d47f8-05c2-41ed-aaa2-701432203f49","old",null,47,null],["","new","tap216e7eb3-bc",-1,["map",[["attached-mac","fa:16:3e:54:bc:55"],["iface-id","216e7eb3-bc4e-44d4-a743-d628e9187924"],["iface-status","active"],["vm-id","e28807f6-826a-49b4-84ee-223be559885e"]]]]],"headings":["row","action","name","ofport","external_ids"]} _read_stdout /opt/openstack/neutron/lib/python3.6/site-packages/neutron/agent/common/async_process.py:262
  2022-10-07 01:00:32.179 214734 DEBUG neutron.agent.common.async_process [-] Output received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: {"data":[["650d47f8-05c2-41ed-aaa2-701432203f49","delete","tap216e7eb3-bc",-1,["map",[["attached-mac","fa:16:3e:54:bc:55"],["iface-id","216e7eb3-bc4e-44d4-a743-d628e9187924"],["iface-status","active"],["vm-id","e28807f6-826a-49b4-84ee-223be559885e"]]]]],"headings":["row","action","name","ofport","external_ids"]} _read_stdout /opt/openstack/neutron/lib/python3.6/site-packages/neutron/agent/common/async_process.py:262

  
  Now imagine that the second event is delayed, so neutron iteration will consider the first event as a port update and will:
  - check the ofport --> -1
  - put the port in skipped_devices
  - put the port DOWN in DB through RPC call
  - remove the port from "current" ports
  - do nothing else, so the port is still configured: openflow rules, etc. stays

  The on next iteration, if the "delete" event is received, the agent will:
  - try to figure out if this port is configured by looking in "current"
  - it's not so it does nothing

  As a result, the port stays configured on the compute. Some openflow
  rules are left over.

  Note that I am running neutron Stein with openvswitch 2.11.4.

  I also check that the 2 events are received on an neutron victoria
  with openvswitch 2.15.0.

  Note also that the race condition is very rare and difficult to
  reproduce, because the port needs to be removed from br-int, but still
  in ovs db with of_port=-1.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1992109/+subscriptions