yahoo-eng-team team mailing list archive

[Artur Korzeniewski <artur.korzeniewski@xxxxxxxxx>]

Public bug reported:

The situation is happening on Liberty-3.

When trying to clear the gateway port and tenant network interface
delete in router, the OVS agent is marking the port as dead instead of
treat them as removed: security group removed and port_unbound

This is causing to left stale OVS flows in br-int, and it may affect the
port_unbound() logic in ovs_neutron_agent.py.

The ovs_neutron_agent is in one iteration of rpc_loop processing the
deleted port via process_deleted_ports() method, marking the qg- port as
dead (ovs flow rule to drop the traffic) and in another iteration, the
ovs_neutron_agent is processing the removed port by
treat_devices_removed() method.

In first iteration, the port deleting is triggered by port_delete() method:
2015-09-04 14:16:20.337 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e43234b1-633b-404d-92d0-0f844dadb586 admin 0f6c0469ea6e4d95a27782c46021243a] port_delete message processed for port 1c749258-74fb-498b-9a08-1fec6725a1cf from (pid=136030) port_delete /opt/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:410

and in second iteration, the device removed is triggered by ovsdb:
2015-09-04 14:16:20.848 DEBUG neutron.agent.linux.ovsdb_monitor [-] Output received from ovsdb monitor: {"data":[["bab86f35-d004-4df6-95c2-0f7432338edb","delete","qg-1c749258-74",49,["map",[["attached-mac","fa:16:3e:99:37:68"],["iface-id","1c749258-74fb-498b-9a08-1fec6725a1cf"],["iface-status","active"]]]]],"headings":["row","action","name","ofport","external_ids"]}
 from (pid=136030) _read_stdout /opt/openstack/neutron/neutron/agent/linux/ovsdb_monitor.py:50

Log from ovs neutron agent:
http://paste.openstack.org/show/445479/

Steps to reproduce:
1. Create router
2. Add tenant network interface to the router
3. Launch a VM
4. Add external network gateway to created router
5. Check the br-int for current port numbers
6. Remove external network gateway
7. Check the br-int for dead port flows (removed port qg-)
8. Remove the network interface from tenant network
9. Check the br-int for dead port flows.

Repeat the steps 4-9 few times to see if dead port flows will appear in
br-int.

This is affecting the legacy, dvr and HA router.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1493414

Title:
  OVS Neutron agent is marking port as dead before they are deleted

Status in neutron:
  New

Bug description:
  The situation is happening on Liberty-3.

  When trying to clear the gateway port and tenant network interface
  delete in router, the OVS agent is marking the port as dead instead of
  treat them as removed: security group removed and port_unbound

  This is causing to left stale OVS flows in br-int, and it may affect
  the port_unbound() logic in ovs_neutron_agent.py.

  The ovs_neutron_agent is in one iteration of rpc_loop processing the
  deleted port via process_deleted_ports() method, marking the qg- port
  as dead (ovs flow rule to drop the traffic) and in another iteration,
  the ovs_neutron_agent is processing the removed port by
  treat_devices_removed() method.

  In first iteration, the port deleting is triggered by port_delete() method:
  2015-09-04 14:16:20.337 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e43234b1-633b-404d-92d0-0f844dadb586 admin 0f6c0469ea6e4d95a27782c46021243a] port_delete message processed for port 1c749258-74fb-498b-9a08-1fec6725a1cf from (pid=136030) port_delete /opt/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:410

  and in second iteration, the device removed is triggered by ovsdb:
  2015-09-04 14:16:20.848 DEBUG neutron.agent.linux.ovsdb_monitor [-] Output received from ovsdb monitor: {"data":[["bab86f35-d004-4df6-95c2-0f7432338edb","delete","qg-1c749258-74",49,["map",[["attached-mac","fa:16:3e:99:37:68"],["iface-id","1c749258-74fb-498b-9a08-1fec6725a1cf"],["iface-status","active"]]]]],"headings":["row","action","name","ofport","external_ids"]}
   from (pid=136030) _read_stdout /opt/openstack/neutron/neutron/agent/linux/ovsdb_monitor.py:50

  Log from ovs neutron agent:
  http://paste.openstack.org/show/445479/

  Steps to reproduce:
  1. Create router
  2. Add tenant network interface to the router
  3. Launch a VM
  4. Add external network gateway to created router
  5. Check the br-int for current port numbers
  6. Remove external network gateway
  7. Check the br-int for dead port flows (removed port qg-)
  8. Remove the network interface from tenant network
  9. Check the br-int for dead port flows.

  Repeat the steps 4-9 few times to see if dead port flows will appear
  in br-int.

  This is affecting the legacy, dvr and HA router.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1493414/+subscriptions