yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #45852
[Bug 1542032] [NEW] IP reassembly issue on the Linux bridges in Openstack
Public bug reported:
Hi,
Sorry for text diagram. It does not look very well on this screen.
Please, copy paste in a decent fixed width text editor.
Thanks,
Claude.
Title: IP reassembly issue on the Linux bridges in Openstack
------------------------------------------------------------
Summary: When the security groups and the Neutron firewall are active in
Openstack, each and every VM virtual network interfaces (VNIC) is
isolated in a Linux bridge and IP reassembly must be performed in order
to allow firewall inspection of the traffic. The reassembled traffic
sometimes exceed the capacity of the physical interfaces and the traffic
is not forwarded properly.
Linux bridge diagram:
---------------------
----------| |--------------|
VM | | OVS |
------- | -------------- ------- | ----- ----- | ------------ -------
| TAP |-|-------| QBR bridge |------| QVB |-----|-|QVO| | P |-|----| FW-ADMIN |----| PHY |
------- | -------------- ------- | ----- ----- | ------------ -------
| | |
--------- | |--------------|
Introduction:
-------------
In Openstack, the virtual machine (VM) uses the OpenvSwitch (OVS) for
networking purposes. This is not a mandatory setup but this is a common
setup in Openstack.
When the Neutron firewall and the security groups are active, each VM
VNIC, also called a tap interface, is connected to a Linux bridge. This
is the QBR bridge. The QVB interface enables the network communication
with OVS. The QVB interface interacts with the QVO interface in OVS.
Security analysis is performed on the Linux bridge. In order to perform
adequate traffic inspection, the fragmented traffic has to be re-
assembled. The traffic is then forwarded according to Maximum Transmit
Unit (MTU) of the interfaces in the bridge.
The MTU values on all the interfaces are set to 65000 bytes. This is
where a part of the problem experienced with NFV applications is
observed.
Analysis:
---------
As a real life example, the NFV application uses NFS between VMs. NFS is
a well known feature in Unix environments. This feature provides network
file systems. This is the equivalent of a network drive in the Windows
world.
NFS is known to produce large frames. In this example, the VM1
(169.254.4.242) send a larg NFS write instruction to the VM2. The
example below shows a 5 KB packet. The traffic is fragmented in several
packets as instructed by the VM1 VNIC. This is the desired behavior.
root@node-11:~# tcpdump -e -n -i tap3e79842d-eb host 169.254.1.13
23:46:48.938255 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 1514: 169.254.4.242.3015988240 > 169.254.1.13.2049: 1472 write fh Unknown/01000601B1198A1CB3CC4E1EA3AB0B26017B0AD653620700D59B28C700000000 4863 (4863) bytes @ 229376
23:46:48.938271 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 1514: 169.254.4.242 > 169.254.1.13: ip-proto-17
23:46:48.938279 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 1514: 169.254.4.242 > 169.254.1.13: ip-proto-17
23:46:48.938287 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 590: 169.254.4.242 > 169.254.1.13: ip-proto-17
The same packet is found on the QVB interface in one large frame.
root@node-11:~# tcpdump -e -n -i qvb3e79842d-eb host 169.254.1.13
23:46:48.938322 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4
(0x0800), length 5030: 169.254.4.242.3015988240 > 169.254.1.13.2049:
4988 write fh
Unknown/01000601B1198A1CB3CC4E1EA3AB0B26017B0AD653620700D59B28C700000000
4863 (4863) bytes @ 229376
Such large packets cannot cross physical interfaces without being
fragmented again if jumbo frames support is not active in the network.
Even with jumbo frames, the NFS frame size can easily cross the 9K
barrier. NFS frame size up to 32 KB can be observed with NFS over UDP.
For some reasons, this traffic does not seem to be transmitted properly
between compute hosts in Openstack.
Further investigations have revealed the large frames are leaving the
OVS internal bridge (br-int) in direction of the private bridge (br-prv)
using a patch interface in OVS. Once the traffic has reached this point,
it uses the "P" interface (i.e.: p_eeee51a2-0) to reach another Linux
bridge (br-fw-admin) where the physical interface is connected to. The
"P" interface has its MTU set to 65000 and the the physical interface as
long as the Linux bridge are set to 1500. A tcpdump analysis reveals the
large frames are reaching the "P" interface and the Linux bridge.
However, the traffic is not observed on the physical interface. The
traffic does not use the DF bit.
This is the reason why the VNF application works fine when all the VMs
are located on the same compute host while the NFS application does not
work properly when the VMs are using multiple compute hosts. Somehow,
when a large frame needs to be sent over to another compute host, either
the Linux bridge or the physical interface does not fragment the packet
again properly. The information is dropped and lost.
Remedy:
-------
As a workaround, the bridge-nf-call-iptables kernel parameters can be
used to disable the bridge netfilter feature. The traffic is not re-
assembled and the NFV application works like a charm. However, the
traffic is not inspected by the firewall anymore and the security groups
functions of the other VNFs/VMs are affected. This is a compute host
wide setting and not a per Linux bridge setting.
The modification can be applied in real time but all the other Linux
bridges on the compute host are affected.
root@node-11:~# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1
root@node-11:~# echo "0" > /proc/sys/net/bridge/bridge-nf-call-iptables
root@node-11:~# cat /proc/sys/net/bridge/bridge-nf-call-iptables
0
The sysctl command can also be used to control the bridge-nf-call-
iptables kernel parameter.
Attachments:
------------
Traffic capture traces showing a 22 KB NFS write operation (nfs-
fragment-1frame.cap & nfs-reassembly-1frame.cap)
Expectations:
-------------
- Find why the traffic is not re-fragmented before leaving the compute host
- Fix the issue
- Provide configuration remedy if applicable
Note: ML2 port-security set to False does not help. The anti-spoofing
are removed but IP reassembly is still performed although FW inspection
is not needed if this feature is present.
Printouts on the compute host (Openstack Kilo):
-----------------------------------------------
root@node-12:~# nova show VM-1.15
+--------------------------------------+---------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+---------------------------------------------------------------------------+
| Internal-1 network | 169.254.4.242 |
| Internal-2 network | 30.30.102.4 |
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | node-11.domain.tld |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-11.domain.tld |
| OS-EXT-SRV-ATTR:instance_name | instance-000000cc |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-01-13T21:14:36.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | True |
| created | 2016-01-13T21:13:58Z |
| flavor | 2vcpu_2048MBmem_1GBdisk (f0083761-fdb1-48bc-8dfd-86fd894d6832) |
| hostId | dab453da6b0bd05902f3d80f6df83d108cfe9704e3d3c0cc903e7628 |
| id | b515db00-067d-4d9a-86be-9dea03c14d03 |
| image | pxeboot_cxp9025898_2r5b03 (0b67c2b1-2370-4b23-91f1-04236b5bba8e) |
| key_name | - |
| metadata | {} |
| name | VM-1.15 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | 36d1650d2c7f47d4be35a46f3bb6a28e |
| updated | 2016-01-13T21:14:37Z |
| user_id | 928a6b5ff95341f5857c5161df7b6ca1 |
+--------------------------------------+---------------------------------------------------------------------------+
root@node-11:~# brctl show
bridge name bridge id STP enabled interfaces
br-ex 8000.2c44fd7c96cc no eth0.35
p_ff798dba-0
br-fw-admin 8000.2c44fd7c96cc no eth0
p_eeee51a2-0
br-mgmt 8000.2c44fd7c96cc no eth0.1526
br-storage 8000.2c44fd7c96cc no eth0.1525
qbr07abdc1e-38 8000.0e00e0133aec no qvb07abdc1e-38
tap07abdc1e-38
qbr101a4853-a9 8000.66349b3bf77d no qvb101a4853-a9
tap101a4853-a9
qbr1e3b62fd-80 8000.d6c7c2e452ac no qvb1e3b62fd-80
tap1e3b62fd-80
qbr26379086-40 8000.1a87ae64580e no qvb26379086-40
tap26379086-40
qbr2871b06a-fb 8000.b638f3116d76 no qvb2871b06a-fb
tap2871b06a-fb
qbr29c06538-34 8000.ba1c5aac2726 no qvb29c06538-34
tap29c06538-34
qbr2efbc02d-33 8000.32e23aa5404e no qvb2efbc02d-33
tap2efbc02d-33
qbr3298eeb5-a1 8000.667029f958ec no qvb3298eeb5-a1
tap3298eeb5-a1
qbr3e79842d-eb 8000.e2d3c6aea326 no qvb3e79842d-eb
tap3e79842d-eb
qbr4805182f-0b 8000.9e3bf559e7c1 no qvb4805182f-0b
tap4805182f-0b
qbr5160349f-e7 8000.d263b9e4f324 no qvb5160349f-e7
tap5160349f-e7
qbr534c601a-0c 8000.ca0079ee8e55 no qvb534c601a-0c
tap534c601a-0c
qbr622ef3b6-a0 8000.625bd7a53dd5 no qvb622ef3b6-a0
tap622ef3b6-a0
qbr960d7784-82 8000.0642984683ea no qvb960d7784-82
tap960d7784-82
qbr99faeb13-17 8000.a6476340bb75 no qvb99faeb13-17
tap99faeb13-17
qbra80a8610-ef 8000.3af49b35beff no qvba80a8610-ef
tapa80a8610-ef
qbrab3661cd-b2 8000.d6dcaee6a0e7 no qvbab3661cd-b2
tapab3661cd-b2
qbrabbfad8e-05 8000.4e0f384dbfde no qvbabbfad8e-05
tapabbfad8e-05
qbrb9bd0dcd-0c 8000.2a4cf0aac6ca no qvbb9bd0dcd-0c
tapb9bd0dcd-0c
qbrc3a88d15-08 8000.da9fcf716879 no qvbc3a88d15-08
tapc3a88d15-08
qbrcf4d2014-ea 8000.063f92ac020e no qvbcf4d2014-ea
tapcf4d2014-ea
qbrd15b94e7-05 8000.5a8a3d70a79d no qvbd15b94e7-05
tapd15b94e7-05
qbrd3c76f84-6f 8000.66039e089f00 no qvbd3c76f84-6f
tapd3c76f84-6f
qbrd9d1a7c6-e2 8000.02f220117f85 no qvbd9d1a7c6-e2
tapd9d1a7c6-e2
qbrdd069c93-ad 8000.a6e25b3b1a82 no qvbdd069c93-ad
tapdd069c93-ad
qbre3ea8b73-13 8000.0e963b47dbc9 no qvbe3ea8b73-13
tape3ea8b73-13
qbree5d29b2-75 8000.d257b819b97a no qvbee5d29b2-75
tapee5d29b2-75
qbrfdd2d84e-e4 8000.02c712bd61bb no qvbfdd2d84e-e4
tapfdd2d84e-e4
root@node-11:~# virsh dumpxml instance-000000cc
<domain type='kvm' id='131'>
<name>instance-000000cc</name>
<uuid>b515db00-067d-4d9a-86be-9dea03c14d03</uuid>
<metadata>
<nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
<nova:package version="2015.1.1"/>
<nova:name>VM-1.15</nova:name>
<nova:creationTime>2016-01-13 21:14:29</nova:creationTime>
<nova:flavor name="2vcpu_2048MBmem_1GBdisk">
<nova:memory>2048</nova:memory>
<nova:disk>1</nova:disk>
<nova:swap>0</nova:swap>
<nova:ephemeral>0</nova:ephemeral>
<nova:vcpus>2</nova:vcpus>
</nova:flavor>
<nova:owner>
<nova:user uuid="928a6b5ff95341f5857c5161df7b6ca1">vepc</nova:user>
<nova:project uuid="36d1650d2c7f47d4be35a46f3bb6a28e">vEPC</nova:project>
</nova:owner>
<nova:root type="image" uuid="0b67c2b1-2370-4b23-91f1-04236b5bba8e"/>
</nova:instance>
</metadata>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>2</vcpu>
<cputune>
<shares>2048</shares>
</cputune>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>OpenStack Foundation</entry>
<entry name='product'>OpenStack Nova</entry>
<entry name='version'>2015.1.1</entry>
<entry name='serial'>99fa98c8-e7ff-4ece-9155-3a0480f50bfd</entry>
<entry name='uuid'>b515db00-067d-4d9a-86be-9dea03c14d03</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
<topology sockets='2' cores='1' threads='1'/>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/disk'/>
<backingStore type='file' index='1'>
<format type='raw'/>
<source file='/var/lib/nova/instances/_base/5bea60e3738cbc5c2604ec84ce6a1ec6e1debfe6'/>
<backingStore/>
</backingStore>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/disk.config'/>
<backingStore/>
<target dev='vdz' bus='virtio'/>
<alias name='virtio-disk25'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>
<controller type='usb' index='0'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<interface type='bridge'>
<mac address='00:80:37:0e:0f:12'/>
<source bridge='qbr3e79842d-eb'/>
<target dev='tap3e79842d-eb'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='00:80:37:0e:0f:12'/>
<source bridge='qbr960d7784-82'/>
<target dev='tap960d7784-82'/>
<model type='virtio'/>
<alias name='net1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
<serial type='file'>
<source path='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/console.log'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<serial type='pty'>
<source path='/dev/pts/6'/>
<target port='1'/>
<alias name='serial1'/>
</serial>
<console type='file'>
<source path='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/console.log'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='tablet' bus='usb'>
<alias name='input0'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='5902' autoport='yes' listen='0.0.0.0' keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
<stats period='10'/>
</memballoon>
</devices>
</domain>
root@node-11:~# ifconfig qbr3e79842d-eb
qbr3e79842d-eb Link encap:Ethernet HWaddr e2:d3:c6:ae:a3:26
inet6 addr: fe80::897:aeff:fee6:5e1b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65000 Metric:1
RX packets:52495 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2529458 (2.5 MB) TX bytes:648 (648.0 B)
root@node-11:~# ifconfig qvb3e79842d-eb
qvb3e79842d-eb Link encap:Ethernet HWaddr e2:d3:c6:ae:a3:26
inet6 addr: fe80::e0d3:c6ff:feae:a326/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:65000 Metric:1
RX packets:1028373 errors:0 dropped:0 overruns:0 frame:0
TX packets:929673 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:600674132 (600.6 MB) TX bytes:429962708 (429.9 MB)
root@node-11:~# ifconfig tap3e79842d-eb
tap3e79842d-eb Link encap:Ethernet HWaddr fe:80:37:0e:0f:12
inet6 addr: fe80::fc80:37ff:fe0e:f12/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65000 Metric:1
RX packets:967910 errors:0 dropped:0 overruns:0 frame:0
TX packets:1028334 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:431302055 (431.3 MB) TX bytes:600737400 (600.7 MB)
root@node-11:~# brctl show qbr3e79842d-eb
bridge name bridge id STP enabled interfaces
qbr3e79842d-eb 8000.e2d3c6aea326 no qvb3e79842d-eb
tap3e79842d-eb
root@node-11:~# ovs-vsctl show
cd41c9a1-d476-4b48-9d5c-e4c5f18afba5
Bridge br-floating
Port "p_ff798dba-0"
Interface "p_ff798dba-0"
type: internal
Port br-floating
Interface br-floating
type: internal
Bridge br-int
fail_mode: secure
Port "qvocf4d2014-ea"
tag: 122
Interface "qvocf4d2014-ea"
Port "qvo99faeb13-17"
tag: 124
Interface "qvo99faeb13-17"
Port "qvo29c06538-34"
tag: 123
Interface "qvo29c06538-34"
Port "qvoabbfad8e-05"
tag: 123
Interface "qvoabbfad8e-05"
Port "qvoab3661cd-b2"
tag: 113
Interface "qvoab3661cd-b2"
Port "qvo534c601a-0c"
tag: 112
Interface "qvo534c601a-0c"
Port "qvo07abdc1e-38"
tag: 112
Interface "qvo07abdc1e-38"
Port "qvo622ef3b6-a0"
tag: 112
Interface "qvo622ef3b6-a0"
Port "qvodd069c93-ad"
tag: 121
Interface "qvodd069c93-ad"
Port "qvob9bd0dcd-0c"
tag: 113
Interface "qvob9bd0dcd-0c"
Port "qvo101a4853-a9"
tag: 113
Interface "qvo101a4853-a9"
Port "qvofdd2d84e-e4"
tag: 115
Interface "qvofdd2d84e-e4"
Port "qvo3e79842d-eb"
tag: 112
Interface "qvo3e79842d-eb"
Port "qvod3c76f84-6f"
tag: 113
Interface "qvod3c76f84-6f"
Port "qvod9d1a7c6-e2"
tag: 121
Interface "qvod9d1a7c6-e2"
Port "qvo1e3b62fd-80"
tag: 113
Interface "qvo1e3b62fd-80"
Port "qvoc3a88d15-08"
tag: 114
Interface "qvoc3a88d15-08"
Port "qvo26379086-40"
tag: 114
Interface "qvo26379086-40"
Port "qvo2efbc02d-33"
tag: 113
Interface "qvo2efbc02d-33"
Port "qvo4805182f-0b"
tag: 115
Interface "qvo4805182f-0b"
Port "qvo960d7784-82"
tag: 113
Interface "qvo960d7784-82"
Port br-int
Interface br-int
type: internal
Port "qvoa80a8610-ef"
tag: 113
Interface "qvoa80a8610-ef"
Port "qvod15b94e7-05"
tag: 112
Interface "qvod15b94e7-05"
Port int-br-prv
Interface int-br-prv
type: patch
options: {peer=phy-br-prv}
Port "qvo5160349f-e7"
tag: 122
Interface "qvo5160349f-e7"
Port "qvo3298eeb5-a1"
tag: 124
Interface "qvo3298eeb5-a1"
Port "qvoee5d29b2-75"
tag: 112
Interface "qvoee5d29b2-75"
Port "qvoe3ea8b73-13"
tag: 112
Interface "qvoe3ea8b73-13"
Port "qvo2871b06a-fb"
tag: 112
Interface "qvo2871b06a-fb"
Bridge br-prv
Port br-prv
Interface br-prv
type: internal
Port phy-br-prv
Interface phy-br-prv
type: patch
options: {peer=int-br-prv}
Port "p_eeee51a2-0"
Interface "p_eeee51a2-0"
type: internal
ovs_version: "2.3.1"
root@node-11:~# ifconfig qvo3e79842d-eb
qvo3e79842d-eb Link encap:Ethernet HWaddr da:e1:98:c1:6e:cf
inet6 addr: fe80::d8e1:98ff:fec1:6ecf/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:65000 Metric:1
RX packets:931164 errors:0 dropped:0 overruns:0 frame:0
TX packets:1030766 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:430267581 (430.2 MB) TX bytes:601031366 (601.0 MB)
root@node-11:~# ifconfig p_eeee51a2-0
p_eeee51a2-0 Link encap:Ethernet HWaddr 6e:9d:56:fb:62:a5
inet6 addr: fe80::6c9d:56ff:fefb:62a5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65000 Metric:1
RX packets:86297635 errors:0 dropped:0 overruns:0 frame:0
TX packets:143277215 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:66475322925 (66.4 GB) TX bytes:35894211276 (35.8 GB)
root@node-11:~# ifconfig br-fw-admin
br-fw-admin Link encap:Ethernet HWaddr 2c:44:fd:7c:9a:a4
inet addr:10.111.158.103 Bcast:10.111.158.111 Mask:255.255.255.240
inet6 addr: fe80::2e44:fdff:fe7c:9aa4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:61629535 errors:0 dropped:2958811 overruns:0 frame:0
TX packets:842703 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:7658578172 (7.6 GB) TX bytes:313894760 (313.8 MB)
root@node-11:~# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 2c:44:fd:7c:9a:a4
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:184932186 errors:88320 dropped:29585 overruns:0 frame:88323
TX packets:123054385 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:71762107044 (71.7 GB) TX bytes:69565856487 (69.5 GB)
Interrupt:32
root@node-12:~# nova-manage --version
2015.1.1
root@node-12:~# uname -a
Linux node-12.domain.tld 3.13.0-65-generic #105-Ubuntu SMP Mon Sep 21 18:50:58 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@node-12:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty
** Affects: neutron
Importance: Undecided
Status: New
** Attachment added: "NFS packet capture"
https://bugs.launchpad.net/bugs/1542032/+attachment/4564105/+files/nfv-capture.zip
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1542032
Title:
IP reassembly issue on the Linux bridges in Openstack
Status in neutron:
New
Bug description:
Hi,
Sorry for text diagram. It does not look very well on this screen.
Please, copy paste in a decent fixed width text editor.
Thanks,
Claude.
Title: IP reassembly issue on the Linux bridges in Openstack
------------------------------------------------------------
Summary: When the security groups and the Neutron firewall are active
in Openstack, each and every VM virtual network interfaces (VNIC) is
isolated in a Linux bridge and IP reassembly must be performed in
order to allow firewall inspection of the traffic. The reassembled
traffic sometimes exceed the capacity of the physical interfaces and
the traffic is not forwarded properly.
Linux bridge diagram:
---------------------
----------| |--------------|
VM | | OVS |
------- | -------------- ------- | ----- ----- | ------------ -------
| TAP |-|-------| QBR bridge |------| QVB |-----|-|QVO| | P |-|----| FW-ADMIN |----| PHY |
------- | -------------- ------- | ----- ----- | ------------ -------
| | |
--------- | |--------------|
Introduction:
-------------
In Openstack, the virtual machine (VM) uses the OpenvSwitch (OVS) for
networking purposes. This is not a mandatory setup but this is a
common setup in Openstack.
When the Neutron firewall and the security groups are active, each VM
VNIC, also called a tap interface, is connected to a Linux bridge.
This is the QBR bridge. The QVB interface enables the network
communication with OVS. The QVB interface interacts with the QVO
interface in OVS.
Security analysis is performed on the Linux bridge. In order to
perform adequate traffic inspection, the fragmented traffic has to be
re-assembled. The traffic is then forwarded according to Maximum
Transmit Unit (MTU) of the interfaces in the bridge.
The MTU values on all the interfaces are set to 65000 bytes. This is
where a part of the problem experienced with NFV applications is
observed.
Analysis:
---------
As a real life example, the NFV application uses NFS between VMs. NFS
is a well known feature in Unix environments. This feature provides
network file systems. This is the equivalent of a network drive in the
Windows world.
NFS is known to produce large frames. In this example, the VM1
(169.254.4.242) send a larg NFS write instruction to the VM2. The
example below shows a 5 KB packet. The traffic is fragmented in
several packets as instructed by the VM1 VNIC. This is the desired
behavior.
root@node-11:~# tcpdump -e -n -i tap3e79842d-eb host 169.254.1.13
23:46:48.938255 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 1514: 169.254.4.242.3015988240 > 169.254.1.13.2049: 1472 write fh Unknown/01000601B1198A1CB3CC4E1EA3AB0B26017B0AD653620700D59B28C700000000 4863 (4863) bytes @ 229376
23:46:48.938271 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 1514: 169.254.4.242 > 169.254.1.13: ip-proto-17
23:46:48.938279 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 1514: 169.254.4.242 > 169.254.1.13: ip-proto-17
23:46:48.938287 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 590: 169.254.4.242 > 169.254.1.13: ip-proto-17
The same packet is found on the QVB interface in one large frame.
root@node-11:~# tcpdump -e -n -i qvb3e79842d-eb host 169.254.1.13
23:46:48.938322 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4
(0x0800), length 5030: 169.254.4.242.3015988240 > 169.254.1.13.2049:
4988 write fh
Unknown/01000601B1198A1CB3CC4E1EA3AB0B26017B0AD653620700D59B28C700000000
4863 (4863) bytes @ 229376
Such large packets cannot cross physical interfaces without being
fragmented again if jumbo frames support is not active in the network.
Even with jumbo frames, the NFS frame size can easily cross the 9K
barrier. NFS frame size up to 32 KB can be observed with NFS over UDP.
For some reasons, this traffic does not seem to be transmitted
properly between compute hosts in Openstack.
Further investigations have revealed the large frames are leaving the
OVS internal bridge (br-int) in direction of the private bridge (br-
prv) using a patch interface in OVS. Once the traffic has reached this
point, it uses the "P" interface (i.e.: p_eeee51a2-0) to reach another
Linux bridge (br-fw-admin) where the physical interface is connected
to. The "P" interface has its MTU set to 65000 and the the physical
interface as long as the Linux bridge are set to 1500. A tcpdump
analysis reveals the large frames are reaching the "P" interface and
the Linux bridge. However, the traffic is not observed on the physical
interface. The traffic does not use the DF bit.
This is the reason why the VNF application works fine when all the VMs
are located on the same compute host while the NFS application does
not work properly when the VMs are using multiple compute hosts.
Somehow, when a large frame needs to be sent over to another compute
host, either the Linux bridge or the physical interface does not
fragment the packet again properly. The information is dropped and
lost.
Remedy:
-------
As a workaround, the bridge-nf-call-iptables kernel parameters can be
used to disable the bridge netfilter feature. The traffic is not re-
assembled and the NFV application works like a charm. However, the
traffic is not inspected by the firewall anymore and the security
groups functions of the other VNFs/VMs are affected. This is a compute
host wide setting and not a per Linux bridge setting.
The modification can be applied in real time but all the other Linux
bridges on the compute host are affected.
root@node-11:~# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1
root@node-11:~# echo "0" > /proc/sys/net/bridge/bridge-nf-call-
iptables
root@node-11:~# cat /proc/sys/net/bridge/bridge-nf-call-iptables
0
The sysctl command can also be used to control the bridge-nf-call-
iptables kernel parameter.
Attachments:
------------
Traffic capture traces showing a 22 KB NFS write operation (nfs-
fragment-1frame.cap & nfs-reassembly-1frame.cap)
Expectations:
-------------
- Find why the traffic is not re-fragmented before leaving the compute host
- Fix the issue
- Provide configuration remedy if applicable
Note: ML2 port-security set to False does not help. The anti-spoofing
are removed but IP reassembly is still performed although FW
inspection is not needed if this feature is present.
Printouts on the compute host (Openstack Kilo):
-----------------------------------------------
root@node-12:~# nova show VM-1.15
+--------------------------------------+---------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+---------------------------------------------------------------------------+
| Internal-1 network | 169.254.4.242 |
| Internal-2 network | 30.30.102.4 |
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | node-11.domain.tld |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-11.domain.tld |
| OS-EXT-SRV-ATTR:instance_name | instance-000000cc |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-01-13T21:14:36.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | True |
| created | 2016-01-13T21:13:58Z |
| flavor | 2vcpu_2048MBmem_1GBdisk (f0083761-fdb1-48bc-8dfd-86fd894d6832) |
| hostId | dab453da6b0bd05902f3d80f6df83d108cfe9704e3d3c0cc903e7628 |
| id | b515db00-067d-4d9a-86be-9dea03c14d03 |
| image | pxeboot_cxp9025898_2r5b03 (0b67c2b1-2370-4b23-91f1-04236b5bba8e) |
| key_name | - |
| metadata | {} |
| name | VM-1.15 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | 36d1650d2c7f47d4be35a46f3bb6a28e |
| updated | 2016-01-13T21:14:37Z |
| user_id | 928a6b5ff95341f5857c5161df7b6ca1 |
+--------------------------------------+---------------------------------------------------------------------------+
root@node-11:~# brctl show
bridge name bridge id STP enabled interfaces
br-ex 8000.2c44fd7c96cc no eth0.35
p_ff798dba-0
br-fw-admin 8000.2c44fd7c96cc no eth0
p_eeee51a2-0
br-mgmt 8000.2c44fd7c96cc no eth0.1526
br-storage 8000.2c44fd7c96cc no eth0.1525
qbr07abdc1e-38 8000.0e00e0133aec no qvb07abdc1e-38
tap07abdc1e-38
qbr101a4853-a9 8000.66349b3bf77d no qvb101a4853-a9
tap101a4853-a9
qbr1e3b62fd-80 8000.d6c7c2e452ac no qvb1e3b62fd-80
tap1e3b62fd-80
qbr26379086-40 8000.1a87ae64580e no qvb26379086-40
tap26379086-40
qbr2871b06a-fb 8000.b638f3116d76 no qvb2871b06a-fb
tap2871b06a-fb
qbr29c06538-34 8000.ba1c5aac2726 no qvb29c06538-34
tap29c06538-34
qbr2efbc02d-33 8000.32e23aa5404e no qvb2efbc02d-33
tap2efbc02d-33
qbr3298eeb5-a1 8000.667029f958ec no qvb3298eeb5-a1
tap3298eeb5-a1
qbr3e79842d-eb 8000.e2d3c6aea326 no qvb3e79842d-eb
tap3e79842d-eb
qbr4805182f-0b 8000.9e3bf559e7c1 no qvb4805182f-0b
tap4805182f-0b
qbr5160349f-e7 8000.d263b9e4f324 no qvb5160349f-e7
tap5160349f-e7
qbr534c601a-0c 8000.ca0079ee8e55 no qvb534c601a-0c
tap534c601a-0c
qbr622ef3b6-a0 8000.625bd7a53dd5 no qvb622ef3b6-a0
tap622ef3b6-a0
qbr960d7784-82 8000.0642984683ea no qvb960d7784-82
tap960d7784-82
qbr99faeb13-17 8000.a6476340bb75 no qvb99faeb13-17
tap99faeb13-17
qbra80a8610-ef 8000.3af49b35beff no qvba80a8610-ef
tapa80a8610-ef
qbrab3661cd-b2 8000.d6dcaee6a0e7 no qvbab3661cd-b2
tapab3661cd-b2
qbrabbfad8e-05 8000.4e0f384dbfde no qvbabbfad8e-05
tapabbfad8e-05
qbrb9bd0dcd-0c 8000.2a4cf0aac6ca no qvbb9bd0dcd-0c
tapb9bd0dcd-0c
qbrc3a88d15-08 8000.da9fcf716879 no qvbc3a88d15-08
tapc3a88d15-08
qbrcf4d2014-ea 8000.063f92ac020e no qvbcf4d2014-ea
tapcf4d2014-ea
qbrd15b94e7-05 8000.5a8a3d70a79d no qvbd15b94e7-05
tapd15b94e7-05
qbrd3c76f84-6f 8000.66039e089f00 no qvbd3c76f84-6f
tapd3c76f84-6f
qbrd9d1a7c6-e2 8000.02f220117f85 no qvbd9d1a7c6-e2
tapd9d1a7c6-e2
qbrdd069c93-ad 8000.a6e25b3b1a82 no qvbdd069c93-ad
tapdd069c93-ad
qbre3ea8b73-13 8000.0e963b47dbc9 no qvbe3ea8b73-13
tape3ea8b73-13
qbree5d29b2-75 8000.d257b819b97a no qvbee5d29b2-75
tapee5d29b2-75
qbrfdd2d84e-e4 8000.02c712bd61bb no qvbfdd2d84e-e4
tapfdd2d84e-e4
root@node-11:~# virsh dumpxml instance-000000cc
<domain type='kvm' id='131'>
<name>instance-000000cc</name>
<uuid>b515db00-067d-4d9a-86be-9dea03c14d03</uuid>
<metadata>
<nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
<nova:package version="2015.1.1"/>
<nova:name>VM-1.15</nova:name>
<nova:creationTime>2016-01-13 21:14:29</nova:creationTime>
<nova:flavor name="2vcpu_2048MBmem_1GBdisk">
<nova:memory>2048</nova:memory>
<nova:disk>1</nova:disk>
<nova:swap>0</nova:swap>
<nova:ephemeral>0</nova:ephemeral>
<nova:vcpus>2</nova:vcpus>
</nova:flavor>
<nova:owner>
<nova:user uuid="928a6b5ff95341f5857c5161df7b6ca1">vepc</nova:user>
<nova:project uuid="36d1650d2c7f47d4be35a46f3bb6a28e">vEPC</nova:project>
</nova:owner>
<nova:root type="image" uuid="0b67c2b1-2370-4b23-91f1-04236b5bba8e"/>
</nova:instance>
</metadata>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>2</vcpu>
<cputune>
<shares>2048</shares>
</cputune>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>OpenStack Foundation</entry>
<entry name='product'>OpenStack Nova</entry>
<entry name='version'>2015.1.1</entry>
<entry name='serial'>99fa98c8-e7ff-4ece-9155-3a0480f50bfd</entry>
<entry name='uuid'>b515db00-067d-4d9a-86be-9dea03c14d03</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
<topology sockets='2' cores='1' threads='1'/>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/disk'/>
<backingStore type='file' index='1'>
<format type='raw'/>
<source file='/var/lib/nova/instances/_base/5bea60e3738cbc5c2604ec84ce6a1ec6e1debfe6'/>
<backingStore/>
</backingStore>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/disk.config'/>
<backingStore/>
<target dev='vdz' bus='virtio'/>
<alias name='virtio-disk25'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>
<controller type='usb' index='0'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<interface type='bridge'>
<mac address='00:80:37:0e:0f:12'/>
<source bridge='qbr3e79842d-eb'/>
<target dev='tap3e79842d-eb'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='00:80:37:0e:0f:12'/>
<source bridge='qbr960d7784-82'/>
<target dev='tap960d7784-82'/>
<model type='virtio'/>
<alias name='net1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
<serial type='file'>
<source path='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/console.log'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<serial type='pty'>
<source path='/dev/pts/6'/>
<target port='1'/>
<alias name='serial1'/>
</serial>
<console type='file'>
<source path='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/console.log'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='tablet' bus='usb'>
<alias name='input0'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='5902' autoport='yes' listen='0.0.0.0' keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
<stats period='10'/>
</memballoon>
</devices>
</domain>
root@node-11:~# ifconfig qbr3e79842d-eb
qbr3e79842d-eb Link encap:Ethernet HWaddr e2:d3:c6:ae:a3:26
inet6 addr: fe80::897:aeff:fee6:5e1b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65000 Metric:1
RX packets:52495 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2529458 (2.5 MB) TX bytes:648 (648.0 B)
root@node-11:~# ifconfig qvb3e79842d-eb
qvb3e79842d-eb Link encap:Ethernet HWaddr e2:d3:c6:ae:a3:26
inet6 addr: fe80::e0d3:c6ff:feae:a326/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:65000 Metric:1
RX packets:1028373 errors:0 dropped:0 overruns:0 frame:0
TX packets:929673 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:600674132 (600.6 MB) TX bytes:429962708 (429.9 MB)
root@node-11:~# ifconfig tap3e79842d-eb
tap3e79842d-eb Link encap:Ethernet HWaddr fe:80:37:0e:0f:12
inet6 addr: fe80::fc80:37ff:fe0e:f12/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65000 Metric:1
RX packets:967910 errors:0 dropped:0 overruns:0 frame:0
TX packets:1028334 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:431302055 (431.3 MB) TX bytes:600737400 (600.7 MB)
root@node-11:~# brctl show qbr3e79842d-eb
bridge name bridge id STP enabled interfaces
qbr3e79842d-eb 8000.e2d3c6aea326 no qvb3e79842d-eb
tap3e79842d-eb
root@node-11:~# ovs-vsctl show
cd41c9a1-d476-4b48-9d5c-e4c5f18afba5
Bridge br-floating
Port "p_ff798dba-0"
Interface "p_ff798dba-0"
type: internal
Port br-floating
Interface br-floating
type: internal
Bridge br-int
fail_mode: secure
Port "qvocf4d2014-ea"
tag: 122
Interface "qvocf4d2014-ea"
Port "qvo99faeb13-17"
tag: 124
Interface "qvo99faeb13-17"
Port "qvo29c06538-34"
tag: 123
Interface "qvo29c06538-34"
Port "qvoabbfad8e-05"
tag: 123
Interface "qvoabbfad8e-05"
Port "qvoab3661cd-b2"
tag: 113
Interface "qvoab3661cd-b2"
Port "qvo534c601a-0c"
tag: 112
Interface "qvo534c601a-0c"
Port "qvo07abdc1e-38"
tag: 112
Interface "qvo07abdc1e-38"
Port "qvo622ef3b6-a0"
tag: 112
Interface "qvo622ef3b6-a0"
Port "qvodd069c93-ad"
tag: 121
Interface "qvodd069c93-ad"
Port "qvob9bd0dcd-0c"
tag: 113
Interface "qvob9bd0dcd-0c"
Port "qvo101a4853-a9"
tag: 113
Interface "qvo101a4853-a9"
Port "qvofdd2d84e-e4"
tag: 115
Interface "qvofdd2d84e-e4"
Port "qvo3e79842d-eb"
tag: 112
Interface "qvo3e79842d-eb"
Port "qvod3c76f84-6f"
tag: 113
Interface "qvod3c76f84-6f"
Port "qvod9d1a7c6-e2"
tag: 121
Interface "qvod9d1a7c6-e2"
Port "qvo1e3b62fd-80"
tag: 113
Interface "qvo1e3b62fd-80"
Port "qvoc3a88d15-08"
tag: 114
Interface "qvoc3a88d15-08"
Port "qvo26379086-40"
tag: 114
Interface "qvo26379086-40"
Port "qvo2efbc02d-33"
tag: 113
Interface "qvo2efbc02d-33"
Port "qvo4805182f-0b"
tag: 115
Interface "qvo4805182f-0b"
Port "qvo960d7784-82"
tag: 113
Interface "qvo960d7784-82"
Port br-int
Interface br-int
type: internal
Port "qvoa80a8610-ef"
tag: 113
Interface "qvoa80a8610-ef"
Port "qvod15b94e7-05"
tag: 112
Interface "qvod15b94e7-05"
Port int-br-prv
Interface int-br-prv
type: patch
options: {peer=phy-br-prv}
Port "qvo5160349f-e7"
tag: 122
Interface "qvo5160349f-e7"
Port "qvo3298eeb5-a1"
tag: 124
Interface "qvo3298eeb5-a1"
Port "qvoee5d29b2-75"
tag: 112
Interface "qvoee5d29b2-75"
Port "qvoe3ea8b73-13"
tag: 112
Interface "qvoe3ea8b73-13"
Port "qvo2871b06a-fb"
tag: 112
Interface "qvo2871b06a-fb"
Bridge br-prv
Port br-prv
Interface br-prv
type: internal
Port phy-br-prv
Interface phy-br-prv
type: patch
options: {peer=int-br-prv}
Port "p_eeee51a2-0"
Interface "p_eeee51a2-0"
type: internal
ovs_version: "2.3.1"
root@node-11:~# ifconfig qvo3e79842d-eb
qvo3e79842d-eb Link encap:Ethernet HWaddr da:e1:98:c1:6e:cf
inet6 addr: fe80::d8e1:98ff:fec1:6ecf/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:65000 Metric:1
RX packets:931164 errors:0 dropped:0 overruns:0 frame:0
TX packets:1030766 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:430267581 (430.2 MB) TX bytes:601031366 (601.0 MB)
root@node-11:~# ifconfig p_eeee51a2-0
p_eeee51a2-0 Link encap:Ethernet HWaddr 6e:9d:56:fb:62:a5
inet6 addr: fe80::6c9d:56ff:fefb:62a5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65000 Metric:1
RX packets:86297635 errors:0 dropped:0 overruns:0 frame:0
TX packets:143277215 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:66475322925 (66.4 GB) TX bytes:35894211276 (35.8 GB)
root@node-11:~# ifconfig br-fw-admin
br-fw-admin Link encap:Ethernet HWaddr 2c:44:fd:7c:9a:a4
inet addr:10.111.158.103 Bcast:10.111.158.111 Mask:255.255.255.240
inet6 addr: fe80::2e44:fdff:fe7c:9aa4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:61629535 errors:0 dropped:2958811 overruns:0 frame:0
TX packets:842703 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:7658578172 (7.6 GB) TX bytes:313894760 (313.8 MB)
root@node-11:~# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 2c:44:fd:7c:9a:a4
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:184932186 errors:88320 dropped:29585 overruns:0 frame:88323
TX packets:123054385 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:71762107044 (71.7 GB) TX bytes:69565856487 (69.5 GB)
Interrupt:32
root@node-12:~# nova-manage --version
2015.1.1
root@node-12:~# uname -a
Linux node-12.domain.tld 3.13.0-65-generic #105-Ubuntu SMP Mon Sep 21 18:50:58 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@node-12:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1542032/+subscriptions
Follow ups
