yahoo-eng-team team mailing list archive

[Jaroslav Pulchart <2122495@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

We are observing an issue when restarting the neutron-openvswitch-agent
service on a compute node.

Some active UDP DNS-related flows that were previously permitted by the
OVS firewall (based on Neutron Security Group rules) are being
reclassified with CT_MARK(CT_MARK_INVALID) after the agent restart.

Example output from ovs-appctl dpctl/dump-conntrack shows affected flows
(dport=53):

# ovs-appctl dpctl/dump-conntrack | grep -E 'udp.*dport=53' | grep 10.16.207.60
udp,orig=(src=10.16.207.60,dst=10.16.194.23,sport=39596,dport=53),reply=(src=10.16.194.23,dst=10.16.207.60,sport=53,dport=39596),zone=2,mark=1
udp,orig=(src=10.16.207.60,dst=10.16.194.23,sport=40258,dport=53),reply=(src=10.16.194.23,dst=10.16.207.60,sport=53,dport=40258),zone=2
udp,orig=(src=10.16.207.60,dst=10.16.194.23,sport=57858,dport=53),reply=(src=10.16.194.23,dst=10.16.207.60,sport=53,dport=57858),zone=2
udp,orig=(src=10.16.207.60,dst=10.16.194.23,sport=59072,dport=53),reply=(src=10.16.194.23,dst=10.16.207.60,sport=53,dport=59072),zone=2

Once marked as invalid, their traffic is dropped until we manually flush
the connection tracker entry, for example:

$ ovs-appctl dpctl/flush-conntrack zone=2
"ct_nw_src=10.16.207.60,ct_nw_proto=17,ct_tp_src=39596,ct_tp_dst=53"

Neutron version: Epoxy, 24.2.1
OpenVSwitch: 3.3.4-115 (rdo)

Environment
* Neutron version: Epoxy, 24.2.1
* Open vSwitch: 3.3.4-115 (RDO)

Impact:
* DNS traffic is disrupted after an agent restart.
* Requires manual intervention (flushing conntrack entries) to restore functionality.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2122495

Title:
  [OVS Firewall] UDP DNS flows dropped after neutron-openvswitch-agent
  service restart due to CT_MARK invalidation

Status in neutron:
  New

Bug description:
  We are observing an issue when restarting the neutron-openvswitch-
  agent service on a compute node.

  Some active UDP DNS-related flows that were previously permitted by
  the OVS firewall (based on Neutron Security Group rules) are being
  reclassified with CT_MARK(CT_MARK_INVALID) after the agent restart.

  Example output from ovs-appctl dpctl/dump-conntrack shows affected
  flows (dport=53):

  # ovs-appctl dpctl/dump-conntrack | grep -E 'udp.*dport=53' | grep 10.16.207.60
  udp,orig=(src=10.16.207.60,dst=10.16.194.23,sport=39596,dport=53),reply=(src=10.16.194.23,dst=10.16.207.60,sport=53,dport=39596),zone=2,mark=1
  udp,orig=(src=10.16.207.60,dst=10.16.194.23,sport=40258,dport=53),reply=(src=10.16.194.23,dst=10.16.207.60,sport=53,dport=40258),zone=2
  udp,orig=(src=10.16.207.60,dst=10.16.194.23,sport=57858,dport=53),reply=(src=10.16.194.23,dst=10.16.207.60,sport=53,dport=57858),zone=2
  udp,orig=(src=10.16.207.60,dst=10.16.194.23,sport=59072,dport=53),reply=(src=10.16.194.23,dst=10.16.207.60,sport=53,dport=59072),zone=2

  Once marked as invalid, their traffic is dropped until we manually
  flush the connection tracker entry, for example:

  $ ovs-appctl dpctl/flush-conntrack zone=2
  "ct_nw_src=10.16.207.60,ct_nw_proto=17,ct_tp_src=39596,ct_tp_dst=53"

  Neutron version: Epoxy, 24.2.1
  OpenVSwitch: 3.3.4-115 (rdo)

  Environment
  * Neutron version: Epoxy, 24.2.1
  * Open vSwitch: 3.3.4-115 (RDO)

  Impact:
  * DNS traffic is disrupted after an agent restart.
  * Requires manual intervention (flushing conntrack entries) to restore functionality.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2122495/+subscriptions