yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #69992
[Bug 1738659] [NEW] linux bridge assigns mac address to the wrong port
Public bug reported:
* High level description:
linux bridge assigns mac address to the physical external interface instead of the tap interface, therefore the VM instance that uses the tap interface is not able to communicate over IP. The workaround I have found is to convert the bridge to a hub, by setting ageing to 0 (brctl setageing br-name 0). In this way, the bridge floods all packets on all attached bridge interfaces, and everything starts working.
* Pre-conditions:
I have an openstack pike running in latest centos 7 release (7.4.1708). Neutron was manually installed as described in the neutron installation guide at https://docs.openstack.org/neutron/latest/install/install-rdo.html. I have configured neutron for Network Option 2 (self service networks), however the setup I am testing here is an external flat provider network with a single cirros VM instance attached directly to it (without any router in between). The openstack environment is made of two nodes: a controller and a compute. The neutron package versions is 11.0.2-2.el7 (latest in centos 7), the bridge-utils version is 1.5-9.el7 and the kernel version is 3.10.0-693.11.1.el7.x86_64. I have tested this with cirros image cirros-0.4.0-x86_64-disk.img and cirros-0.3.5-x86_64-disk.img.
# rpm -qa | grep neutron-linuxbridge
openstack-neutron-linuxbridge-11.0.2-2.el7.noarch
# rpm -qf /usr/sbin/brctl
bridge-utils-1.5-9.el7.x86_64
# uname -a
Linux compute1 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Bridging is configured like below in both controller and compute:
ml2_conf.ini:
[ml2_type_flat]
flat_networks = physnet1
linuxbridge_agent.ini:
[linux_bridge]
physical_interface_mappings = physnet1:bond2
* Step-by-step reproduction steps:
This is how I created the provider network:
openstack network create \
--share \
--external \
--provider-physical-network physnet1 \
--provider-network-type flat \
ExtNet1
This is how I create the provider subnet:
openstack subnet create \
--network ExtNet1 \
--allocation-pool start=10.20.21.96,end=$10.20.21.127 \
--dns-nameserver 10.20.21.1 \
--gateway 10.20.21.1 \
--subnet-range 10.20.21.0/24 \
ExtSubnet1
This is how I launch a cirros instance and attach it to the provider network:
openstack server create \
--flavor m1.nano \
--image cirros-0.4.0-x86_64-disk.img \
--nic net-id=$(openstack network list | grep ExtNet1 | cut -d\ -f 2) \
--security-group default \
--key-name controller-key \
cirros1
Based on the above, neutron creates in my compute node the following
bridge:
# brctl show
bridge name bridge id STP enabled interfaces
brq75a55ef7-4a 8000.fc15b413e6a3 no bond2
tap44bc34bb-e2
bond2 is the physical interface used for the flat provider network (in
access mode, no vlans) and tap44bc34bb-e2 is the tap interface attached
to my cirros VM instance.
In the bridge, the bond2 is port 2, and the tap tap44bc34bb-e2 interface
is port 1, and both are in forwarding mode.
# brctl showstp brq75a55ef7-4a
brq75a55ef7-4a
<...>
ageing time 300.00
<...>
bond2 (2)
port id 8002 state forwarding
<...>
tap44bc34bb-e2 (1)
port id 8001 state forwarding
<...>
The network flow is like below:
default gw<-->physical switch<-->[bond2 bridge tap]<-->[eth0 cirrosVM]
eth0 mac address is fa:16:3e:cc:dc:ec.
After the cirros VM comes up, it is not able to get an IP address from
the DHCP agent, and there is no IP communication. Therefore I have to
use the console and manually assign the corresponding IP address to the
cirros VM eth0 interface, but still the IP connectivity does not work,
since I cannot ping any external IPs, even in the same 10.20.21.0/24
subnet.
What I have found is that in the bridge forwarding table, the bridge
wrongly assigns the eth0 mac address to the port 2, which is bond2
interface, instead of assigning it to the port 1, which is the tap
interface. This happens only if the arp table in the cirros VM instance
does not contain the mac address of the destination IP I am pinging
(default gw in this case), so the cirros VM sends an arp request
(Request who-has 10.203.219.1 tell 10.203.219.114). See below the eth0
mac address wrongly assigned in the forwarding table to the port 2:
# brctl showmacs brq75a55ef7-4a | grep fa:16:3e:cc:dc:ec
2 fa:16:3e:cc:dc:ec no 0.39
However, since the eth0 mac address is wrongly assigned to the port2,
the arp reply back (Reply 10.203.219.1 is-at 00:17:08:c4:52:80) does not
reach anymore eth0. Using a tcpdump, I can see in the compute node the
arp request going through the tap interface an the bridge, however the
arp reply back does not show up anymore on the tap interface. Since the
arp reply back does not reach the eth0, the arp table in the cirros VM
does not contain the proper entries, and therefore the IP communication
(ping in my case) does not work.
The strange thing is that after a while, for apparently no reason, a
single arp reply back packet gets through the bridge and the tap
interface, and the arp table gets updated with correct mac address in
the cirros VM instance, and I am able to ping that IP address.
If I manually add the mac address of the destination IP I am pining into
the cirros VM instance arp table, and there is no arp request sent, just
icmp packets going out, then the bridge correctly assigns the eth0 mac
address to the port 1, which is the tap interface, and everything starts
working fine. See below the eth0 mac address correctly assigned in the
forwarding table to the port 1:
# brctl showmacs brq75a55ef7-4a | grep fa:16:3e:cc:dc:ec
1 fa:16:3e:cc:dc:ec no 0.09
The only workaround I have found to this issue is to configure the bridge ageing time to zero, converting the bridge into a hub so that it floods all packets on all ports.
It is also strange that in the bridge forwarding table, the bond0 and tap interface mac addresses are shown twice, as seen below:
# brctl showmacs brq75a55ef7-4a | grep yes
2 fc:15:b4:13:e6:a3 yes 0.00
2 fc:15:b4:13:e6:a3 yes 0.00
1 fe:16:3e:cc:dc:ec yes 0.00
1 fe:16:3e:cc:dc:ec yes 0.00
I have tested this in an all-in-one openstack installation running in a
vmware centos 7 VM, that has a single interface configured in
promiscuous mode in the vmware portgroup (therefore no interface bonding
like in the previous setup), and the same issue happens, therefore I do
not think this is caused by the bonding interface.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1738659
Title:
linux bridge assigns mac address to the wrong port
Status in neutron:
New
Bug description:
* High level description:
linux bridge assigns mac address to the physical external interface instead of the tap interface, therefore the VM instance that uses the tap interface is not able to communicate over IP. The workaround I have found is to convert the bridge to a hub, by setting ageing to 0 (brctl setageing br-name 0). In this way, the bridge floods all packets on all attached bridge interfaces, and everything starts working.
* Pre-conditions:
I have an openstack pike running in latest centos 7 release (7.4.1708). Neutron was manually installed as described in the neutron installation guide at https://docs.openstack.org/neutron/latest/install/install-rdo.html. I have configured neutron for Network Option 2 (self service networks), however the setup I am testing here is an external flat provider network with a single cirros VM instance attached directly to it (without any router in between). The openstack environment is made of two nodes: a controller and a compute. The neutron package versions is 11.0.2-2.el7 (latest in centos 7), the bridge-utils version is 1.5-9.el7 and the kernel version is 3.10.0-693.11.1.el7.x86_64. I have tested this with cirros image cirros-0.4.0-x86_64-disk.img and cirros-0.3.5-x86_64-disk.img.
# rpm -qa | grep neutron-linuxbridge
openstack-neutron-linuxbridge-11.0.2-2.el7.noarch
# rpm -qf /usr/sbin/brctl
bridge-utils-1.5-9.el7.x86_64
# uname -a
Linux compute1 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Bridging is configured like below in both controller and compute:
ml2_conf.ini:
[ml2_type_flat]
flat_networks = physnet1
linuxbridge_agent.ini:
[linux_bridge]
physical_interface_mappings = physnet1:bond2
* Step-by-step reproduction steps:
This is how I created the provider network:
openstack network create \
--share \
--external \
--provider-physical-network physnet1 \
--provider-network-type flat \
ExtNet1
This is how I create the provider subnet:
openstack subnet create \
--network ExtNet1 \
--allocation-pool start=10.20.21.96,end=$10.20.21.127 \
--dns-nameserver 10.20.21.1 \
--gateway 10.20.21.1 \
--subnet-range 10.20.21.0/24 \
ExtSubnet1
This is how I launch a cirros instance and attach it to the provider network:
openstack server create \
--flavor m1.nano \
--image cirros-0.4.0-x86_64-disk.img \
--nic net-id=$(openstack network list | grep ExtNet1 | cut -d\ -f 2) \
--security-group default \
--key-name controller-key \
cirros1
Based on the above, neutron creates in my compute node the following
bridge:
# brctl show
bridge name bridge id STP enabled interfaces
brq75a55ef7-4a 8000.fc15b413e6a3 no bond2
tap44bc34bb-e2
bond2 is the physical interface used for the flat provider network (in
access mode, no vlans) and tap44bc34bb-e2 is the tap interface
attached to my cirros VM instance.
In the bridge, the bond2 is port 2, and the tap tap44bc34bb-e2
interface is port 1, and both are in forwarding mode.
# brctl showstp brq75a55ef7-4a
brq75a55ef7-4a
<...>
ageing time 300.00
<...>
bond2 (2)
port id 8002 state forwarding
<...>
tap44bc34bb-e2 (1)
port id 8001 state forwarding
<...>
The network flow is like below:
default gw<-->physical switch<-->[bond2 bridge tap]<-->[eth0 cirrosVM]
eth0 mac address is fa:16:3e:cc:dc:ec.
After the cirros VM comes up, it is not able to get an IP address from
the DHCP agent, and there is no IP communication. Therefore I have to
use the console and manually assign the corresponding IP address to
the cirros VM eth0 interface, but still the IP connectivity does not
work, since I cannot ping any external IPs, even in the same
10.20.21.0/24 subnet.
What I have found is that in the bridge forwarding table, the bridge
wrongly assigns the eth0 mac address to the port 2, which is bond2
interface, instead of assigning it to the port 1, which is the tap
interface. This happens only if the arp table in the cirros VM
instance does not contain the mac address of the destination IP I am
pinging (default gw in this case), so the cirros VM sends an arp
request (Request who-has 10.203.219.1 tell 10.203.219.114). See below
the eth0 mac address wrongly assigned in the forwarding table to the
port 2:
# brctl showmacs brq75a55ef7-4a | grep fa:16:3e:cc:dc:ec
2 fa:16:3e:cc:dc:ec no 0.39
However, since the eth0 mac address is wrongly assigned to the port2,
the arp reply back (Reply 10.203.219.1 is-at 00:17:08:c4:52:80) does
not reach anymore eth0. Using a tcpdump, I can see in the compute node
the arp request going through the tap interface an the bridge, however
the arp reply back does not show up anymore on the tap interface.
Since the arp reply back does not reach the eth0, the arp table in the
cirros VM does not contain the proper entries, and therefore the IP
communication (ping in my case) does not work.
The strange thing is that after a while, for apparently no reason, a
single arp reply back packet gets through the bridge and the tap
interface, and the arp table gets updated with correct mac address in
the cirros VM instance, and I am able to ping that IP address.
If I manually add the mac address of the destination IP I am pining
into the cirros VM instance arp table, and there is no arp request
sent, just icmp packets going out, then the bridge correctly assigns
the eth0 mac address to the port 1, which is the tap interface, and
everything starts working fine. See below the eth0 mac address
correctly assigned in the forwarding table to the port 1:
# brctl showmacs brq75a55ef7-4a | grep fa:16:3e:cc:dc:ec
1 fa:16:3e:cc:dc:ec no 0.09
The only workaround I have found to this issue is to configure the bridge ageing time to zero, converting the bridge into a hub so that it floods all packets on all ports.
It is also strange that in the bridge forwarding table, the bond0 and tap interface mac addresses are shown twice, as seen below:
# brctl showmacs brq75a55ef7-4a | grep yes
2 fc:15:b4:13:e6:a3 yes 0.00
2 fc:15:b4:13:e6:a3 yes 0.00
1 fe:16:3e:cc:dc:ec yes 0.00
1 fe:16:3e:cc:dc:ec yes 0.00
I have tested this in an all-in-one openstack installation running in
a vmware centos 7 VM, that has a single interface configured in
promiscuous mode in the vmware portgroup (therefore no interface
bonding like in the previous setup), and the same issue happens,
therefore I do not think this is caused by the bonding interface.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1738659/+subscriptions
Follow ups
