← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1970606] Re: Live migration packet loss increasing as the number of security group rules increases

 

Hello:

Let me first confirm that you are using Wallaby and OVS backend.

In the OVS backend, there are you type of plugs: native and hybrid. The
native plug is used by default and can be used with the OVS native
firewall. The hybrid plug is used when the OVS iptables firewall is
used.

When using the hybrid plug, the TAP port is created when Nova/os-vif
creates the L1 port. This TAP port is connected to the linux bridge
where the iptables rules will be set. Neutron OVS agent has time to set
the OVS rules (fewer ones) and when the VM is unpaused in the
destination host, there is no disruption (or the time is shorter). You
can switch to iptables FW if the time disruption is critical for your
operation.

When using native plug, the port is created but not the TAP port. That
means there is no ofport and the OVS OF rules can't be set. It is at the
very last time, when the VM is unpaused, when libvirt creates the TAP
port. At this point the OVS agent starts applying the OVS OF rules. The
more rules you have, the bigger could be the time gap.

In Neutron Wallaby you can use "live_migration_events" [1] (removed in
Zed, now is True by default). That needs the Nova patch [2], that was
merged in this release. Check first if Nova has it. That will reduce the
live migration disruption, but won't remove it all.

In Neutron master you can use "openflow_processed_per_port" [3]. This
option will allow the OVS agent to write all OF rules related to a
single port in one single transaction. That should reduce the disruption
too.

In any case, Neutron does not have a SLA for the live-migration network
disruption time; we provide a best effort promise but nothing else.

Regards.

[1]https://review.opendev.org/c/openstack/neutron/+/766277
[2]https://review.opendev.org/c/openstack/nova/+/767368
[3]https://review.opendev.org/c/openstack/neutron/+/806246


** Changed in: neutron
       Status: Confirmed => Won't Fix

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1970606

Title:
  Live migration packet loss increasing as the number of security group
  rules increases

Status in neutron:
  Won't Fix

Bug description:
  Hi,

  We lose too many packets during live migration. (After
  post_live_migration starts)

  After investigation we have recognized that it is related with the
  number of security group rules which are applied to instance.

  We are loosing 26 ping if there exist 90 security rules applied to
  instance. (Security group count does not matter, 1 group 90 rules or 3
  group with 30 rules)

  After detaching some rules from instance and let the instance have
  only 4 security group rules and then tried to migrate again. In that
  case we are only loosing 3 pings.

  Do you have any idea? If this is caused by migrating the ovs flows,
  than is there any solution?

  Environment Details:
   OpenStack Wallaby Cluster installed via kolla-ansible to Ubuntu 20.04.2 LTS Hosts. (Kernel:5.4.0-90-generic)
   There exist 5 controller+network node.
   "neutron-openvswitch-agent", "neutron-l3-agent" and "neutron-server" version is "18.1.2.dev118"
   OpenvSwitch used in DVR mode with router HA configured. (l3_ha = true)
   We are using a single centralized neutron router for connecting all tenant networks to provider network.
   We are using bgp_dragent to announce unique tenant networks.
   Tenant network type: vxlan
   External network type: vlan

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1970606/+subscriptions



References