yahoo-eng-team team mailing list archive

[Itsuro Oda <oda@xxxxxxxxxxxxx>]

Public bug reported:

When remote group member changed, conntrack entry which should be
deleted is not deleted.

How to reproduce:
* create a VM(VM1) on host-1 (net-a, default security-group) (ex. 10.0.0.3)
* create a VM(VM2) on host-2 (net-a, default security-group) (ex. 10.0.0.4)
* ssh from VM1(10.0.0.3) to VM2(10.0.0.4)
---
host-2:$ sudo conntrack -L |grep 10.0
tcp      6 431986 ESTABLISHED src=10.0.0.3 dst=10.0.0.4 sport=45074 dport=22 src=10.0.0.4 dst=10.0.0.3 sport=22 dport=45074 [ASSURED] mark=0 zone=1 use=1

host-2:$ sudo ipset list
Name: NIPv492469920-ef76-44af-98c7-
Type: hash:net
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 16824
References: 1
Members:
10.0.0.4
10.0.0.3
---

* terminate VM1  (nova delete VM1) 
expected: the conntrack entry showed above is deleted.
actual: not deleted
---
host-2:$ sudo conntrack -L |grep 10.0
tcp      6 431986 ESTABLISHED src=10.0.0.3 dst=10.0.0.4 sport=45074 dport=22 src=10.0.0.4 dst=10.0.0.3 sport=22 dport=45074 [ASSURED] mark=0 zone=1 use=1

host-2:$ sudo ipset list
Name: NIPv492469920-ef76-44af-98c7-
Type: hash:net
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 16824
References: 1
Members:
10.0.0.4
---

Applied:
liberty, mitaka, master

Investigation:
summary - devices_with_updated_sg_members is consumed by remove_devices_filter unintentionally.
* when ovs-agent receives security_groups_member_updated RPC call, 
  sg_ids and affected devices are registered to self.firewall.devices_with_updated_sg_members.
  (original intention is that it is handled when refresh_firewall is called. but...)
* in the main loop of ovs-agent process_deleted_ports is executed before process_network_ports.
  process_deleted_ports calls self.sg_agent.remove_devices_filter.
  if there is any deleted port, 
  remove_devices_filter
  -> defer_apply_off
  -> _remove_conntrack_entries_from_sg_update
  -> _clean_deleted_remote_sg_members_conntrack_entries
  is called. 
  at this time pre_sg_members and sg_members are same since no port info is 
  updated in remove_devices_filter. so no conntrack entry is removed. 
  but nonetheless devices_with_updated_sg_members is cleared !!
* afterwards
  process_network_ports
  -> setup_port_filters -> refresh_firewall -> defer_apply_off
  ...-> _clean_deleted_remote_sg_members_conntrack_entries
  is called.
  at this time pre_sg_members and sg_members are different since port info was updated.
  but no conntrack entry is removed since devices_with_updated_sg_members was cleared.

Note:
deleting conntrack entries was introduced by https://bugs.launchpad.net/neutron/+bug/1335375.
note that conntrack zone was per network at this time. (per port now)
The fix of 1335375 is complicated and I wonder it is incomplete (ex. no care of egress & remote-group rule).
How about simplify to just record affected ports and do "conntrack -D -w <port's zone id>" 
for affected ports ?

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: sg-fw

** Tags added: sg-fw

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1580377

Title:
  conntrack entry is not deleted when security_groups_member_updated

Status in neutron:
  New

Bug description:
  When remote group member changed, conntrack entry which should be
  deleted is not deleted.

  How to reproduce:
  * create a VM(VM1) on host-1 (net-a, default security-group) (ex. 10.0.0.3)
  * create a VM(VM2) on host-2 (net-a, default security-group) (ex. 10.0.0.4)
  * ssh from VM1(10.0.0.3) to VM2(10.0.0.4)
  ---
  host-2:$ sudo conntrack -L |grep 10.0
  tcp      6 431986 ESTABLISHED src=10.0.0.3 dst=10.0.0.4 sport=45074 dport=22 src=10.0.0.4 dst=10.0.0.3 sport=22 dport=45074 [ASSURED] mark=0 zone=1 use=1

  host-2:$ sudo ipset list
  Name: NIPv492469920-ef76-44af-98c7-
  Type: hash:net
  Revision: 4
  Header: family inet hashsize 1024 maxelem 65536
  Size in memory: 16824
  References: 1
  Members:
  10.0.0.4
  10.0.0.3
  ---

  * terminate VM1  (nova delete VM1) 
  expected: the conntrack entry showed above is deleted.
  actual: not deleted
  ---
  host-2:$ sudo conntrack -L |grep 10.0
  tcp      6 431986 ESTABLISHED src=10.0.0.3 dst=10.0.0.4 sport=45074 dport=22 src=10.0.0.4 dst=10.0.0.3 sport=22 dport=45074 [ASSURED] mark=0 zone=1 use=1

  host-2:$ sudo ipset list
  Name: NIPv492469920-ef76-44af-98c7-
  Type: hash:net
  Revision: 4
  Header: family inet hashsize 1024 maxelem 65536
  Size in memory: 16824
  References: 1
  Members:
  10.0.0.4
  ---

  Applied:
  liberty, mitaka, master

  Investigation:
  summary - devices_with_updated_sg_members is consumed by remove_devices_filter unintentionally.
  * when ovs-agent receives security_groups_member_updated RPC call, 
    sg_ids and affected devices are registered to self.firewall.devices_with_updated_sg_members.
    (original intention is that it is handled when refresh_firewall is called. but...)
  * in the main loop of ovs-agent process_deleted_ports is executed before process_network_ports.
    process_deleted_ports calls self.sg_agent.remove_devices_filter.
    if there is any deleted port, 
    remove_devices_filter
    -> defer_apply_off
    -> _remove_conntrack_entries_from_sg_update
    -> _clean_deleted_remote_sg_members_conntrack_entries
    is called. 
    at this time pre_sg_members and sg_members are same since no port info is 
    updated in remove_devices_filter. so no conntrack entry is removed. 
    but nonetheless devices_with_updated_sg_members is cleared !!
  * afterwards
    process_network_ports
    -> setup_port_filters -> refresh_firewall -> defer_apply_off
    ...-> _clean_deleted_remote_sg_members_conntrack_entries
    is called.
    at this time pre_sg_members and sg_members are different since port info was updated.
    but no conntrack entry is removed since devices_with_updated_sg_members was cleared.

  Note:
  deleting conntrack entries was introduced by https://bugs.launchpad.net/neutron/+bug/1335375.
  note that conntrack zone was per network at this time. (per port now)
  The fix of 1335375 is complicated and I wonder it is incomplete (ex. no care of egress & remote-group rule).
  How about simplify to just record affected ports and do "conntrack -D -w <port's zone id>" 
  for affected ports ?

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1580377/+subscriptions