yahoo-eng-team team mailing list archive

[Yi Yang <1866445@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

In Openstack Queens release, we noticed a very serious issue, br-int
bridge in one compute node can't learn MAC addresses of VMs in other
compute nodes, so after launched many VMs, VM-to-VM network performance
will decrease linearly, especially ovs will broadcast packets because it
doesn't learn target VM MAC address, other VMs in same subnet in same
compute node can receive these broadcast packets, therefore, the
corresponding vhost kernel threads are receiving these packets and
wasting CPU resources. More VMs, more serious the issue, worse the
performance, no matter UDP or TCP performance.

We have checked several Queens deployments, they have same issues, but
Openstack Rocky doesn't have this issue. Here is the flow I dumped:

recirc_id(0),in_port(12),eth(src=fa:16:3e:49:26:51,dst=fa:16:3e:a7:0a:3a),eth_type(0x0800),ipv4(tos=0/0x3,frag=no),
packets:11012944, bytes:726983412, used:0.000s, flags:SP.,
actions:push_vlan(vid=1,pcp=0),2,set(tunnel(tun_id=0x49,src=10.3.2.17,dst=10.3.2.16,ttl=64,tp_dst=4789,flags(df|key))),pop_vlan,9,8,11,13,14,15,16,17,18,19

MAC address of target VM wasn't learnt by br-int 
$ sudo ovs-appctl fdb/show br-int | grep "fa:16:3e:a7:0a:3a"

By the way, we used linuxbridge for security group.

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: queens

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1866445

Title:
  br-int bridge in one compute can't learn MAC addresses of VMs in other
  compute nodes

Status in neutron:
  New

Bug description:
  In Openstack Queens release, we noticed a very serious issue, br-int
  bridge in one compute node can't learn MAC addresses of VMs in other
  compute nodes, so after launched many VMs, VM-to-VM network
  performance will decrease linearly, especially ovs will broadcast
  packets because it doesn't learn target VM MAC address, other VMs in
  same subnet in same compute node can receive these broadcast packets,
  therefore, the corresponding vhost kernel threads are receiving these
  packets and wasting CPU resources. More VMs, more serious the issue,
  worse the performance, no matter UDP or TCP performance.

  We have checked several Queens deployments, they have same issues, but
  Openstack Rocky doesn't have this issue. Here is the flow I dumped:

  recirc_id(0),in_port(12),eth(src=fa:16:3e:49:26:51,dst=fa:16:3e:a7:0a:3a),eth_type(0x0800),ipv4(tos=0/0x3,frag=no),
  packets:11012944, bytes:726983412, used:0.000s, flags:SP.,
  actions:push_vlan(vid=1,pcp=0),2,set(tunnel(tun_id=0x49,src=10.3.2.17,dst=10.3.2.16,ttl=64,tp_dst=4789,flags(df|key))),pop_vlan,9,8,11,13,14,15,16,17,18,19

  MAC address of target VM wasn't learnt by br-int 
  $ sudo ovs-appctl fdb/show br-int | grep "fa:16:3e:a7:0a:3a"

  By the way, we used linuxbridge for security group.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1866445/+subscriptions