← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1542032] [NEW] IP reassembly issue on the Linux bridges in Openstack

 

Public bug reported:

Hi,

Sorry for text diagram. It does not look very well on this screen.
Please, copy paste in a decent fixed width text editor.

Thanks,

Claude.


Title: IP reassembly issue on the Linux bridges in Openstack
------------------------------------------------------------

Summary: When the security groups and the Neutron firewall are active in
Openstack, each and every VM virtual network interfaces (VNIC) is
isolated in a Linux bridge and IP reassembly must be performed in order
to allow firewall inspection of the traffic. The reassembled traffic
sometimes exceed the capacity of the physical interfaces and the traffic
is not forwarded properly.

Linux bridge diagram:
---------------------

----------|                                       |--------------|
   VM     |                                       |      OVS     |
  ------- |       --------------      -------     | -----  ----- |    ------------    -------
  | TAP |-|-------| QBR bridge |------| QVB |-----|-|QVO|  | P |-|----| FW-ADMIN |----| PHY |
  ------- |       --------------      -------     | -----  ----- |    ------------    -------
          |                                       |              |
--------- |                                       |--------------|

Introduction:
-------------

In Openstack, the virtual machine (VM) uses the OpenvSwitch (OVS) for
networking purposes. This is not a mandatory setup but this is a common
setup in Openstack.

When the Neutron firewall and the security groups are active, each VM
VNIC, also called a tap interface, is connected to a Linux bridge. This
is the QBR bridge. The QVB interface enables the network communication
with OVS. The QVB interface interacts with the QVO interface in OVS.

Security analysis is performed on the Linux bridge. In order to perform
adequate traffic inspection, the fragmented traffic has to be re-
assembled. The traffic is then forwarded according to Maximum Transmit
Unit (MTU) of the interfaces in the bridge.

The MTU values on all the interfaces are set to 65000 bytes. This is
where a part of the problem experienced with NFV applications is
observed.

Analysis:
---------

As a real life example, the NFV application uses NFS between VMs. NFS is
a well known feature in Unix environments. This feature provides network
file systems. This is the equivalent of a network drive in the Windows
world.

NFS is known to produce large frames. In this example, the VM1
(169.254.4.242) send a larg NFS write instruction to the VM2. The
example below shows a 5 KB packet. The traffic is fragmented in several
packets as instructed by the VM1 VNIC. This is the desired behavior.

root@node-11:~# tcpdump -e -n -i tap3e79842d-eb host 169.254.1.13

23:46:48.938255 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 1514: 169.254.4.242.3015988240 > 169.254.1.13.2049: 1472 write fh Unknown/01000601B1198A1CB3CC4E1EA3AB0B26017B0AD653620700D59B28C700000000 4863 (4863) bytes @ 229376
23:46:48.938271 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 1514: 169.254.4.242 > 169.254.1.13: ip-proto-17
23:46:48.938279 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 1514: 169.254.4.242 > 169.254.1.13: ip-proto-17
23:46:48.938287 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 590: 169.254.4.242 > 169.254.1.13: ip-proto-17

The same packet is found on the QVB interface in one large frame.

root@node-11:~# tcpdump -e -n -i qvb3e79842d-eb host 169.254.1.13

23:46:48.938322 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4
(0x0800), length 5030: 169.254.4.242.3015988240 > 169.254.1.13.2049:
4988 write fh
Unknown/01000601B1198A1CB3CC4E1EA3AB0B26017B0AD653620700D59B28C700000000
4863 (4863) bytes @ 229376

Such large packets cannot cross physical interfaces without being
fragmented again if jumbo frames support is not active in the network.
Even with jumbo frames, the NFS frame size can easily cross the 9K
barrier. NFS frame size up to 32 KB can be observed with NFS over UDP.

For some reasons, this traffic does not seem to be transmitted properly
between compute hosts in Openstack.

Further investigations have revealed the large frames are leaving the
OVS internal bridge (br-int) in direction of the private bridge (br-prv)
using a patch interface in OVS. Once the traffic has reached this point,
it uses the "P" interface (i.e.: p_eeee51a2-0) to reach another Linux
bridge (br-fw-admin) where the physical interface is connected to. The
"P" interface has its MTU set to 65000 and the the physical interface as
long as the Linux bridge are set to 1500. A tcpdump analysis reveals the
large frames are reaching the "P" interface and the Linux bridge.
However, the traffic is not observed on the physical interface. The
traffic does not use the DF bit.

This is the reason why the VNF application works fine when all the VMs
are located on the same compute host while the NFS application does not
work properly when the VMs are using multiple compute hosts. Somehow,
when a large frame needs to be sent over to another compute host, either
the Linux bridge or the physical interface does not fragment the packet
again properly. The information is dropped and lost.

Remedy:
-------

As a workaround, the bridge-nf-call-iptables kernel parameters can be
used to disable the bridge netfilter feature. The traffic is not re-
assembled and the NFV application works like a charm. However, the
traffic is not inspected by the firewall anymore and the security groups
functions of the other VNFs/VMs are affected. This is a compute host
wide setting and not a per Linux bridge setting.

The modification can be applied in real time but all the other Linux
bridges on the compute host are affected.

root@node-11:~# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1

root@node-11:~# echo "0" > /proc/sys/net/bridge/bridge-nf-call-iptables

root@node-11:~# cat /proc/sys/net/bridge/bridge-nf-call-iptables 
0

The sysctl command can also be used to control the bridge-nf-call-
iptables kernel parameter.

Attachments:
------------

Traffic capture traces showing a 22 KB NFS write operation (nfs-
fragment-1frame.cap & nfs-reassembly-1frame.cap)

Expectations:
-------------

- Find why the traffic is not re-fragmented before leaving the compute host
- Fix the issue
- Provide configuration remedy if applicable

Note: ML2 port-security set to False does not help. The anti-spoofing
are removed but IP reassembly is still performed although FW inspection
is not needed if this feature is present.

Printouts on the compute host (Openstack Kilo):
-----------------------------------------------

root@node-12:~# nova show VM-1.15
+--------------------------------------+---------------------------------------------------------------------------+
| Property                             | Value                                                                     |
+--------------------------------------+---------------------------------------------------------------------------+
| Internal-1 network                   | 169.254.4.242                                                             |
| Internal-2 network                   | 30.30.102.4                                                               |
| OS-DCF:diskConfig                    | MANUAL                                                                    |
| OS-EXT-AZ:availability_zone          | nova                                                                      |
| OS-EXT-SRV-ATTR:host                 | node-11.domain.tld                                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | node-11.domain.tld                                                        |
| OS-EXT-SRV-ATTR:instance_name        | instance-000000cc                                                         |
| OS-EXT-STS:power_state               | 1                                                                         |
| OS-EXT-STS:task_state                | -                                                                         |
| OS-EXT-STS:vm_state                  | active                                                                    |
| OS-SRV-USG:launched_at               | 2016-01-13T21:14:36.000000                                                |
| OS-SRV-USG:terminated_at             | -                                                                         |
| accessIPv4                           |                                                                           |
| accessIPv6                           |                                                                           |
| config_drive                         | True                                                                      |
| created                              | 2016-01-13T21:13:58Z                                                      |
| flavor                               | 2vcpu_2048MBmem_1GBdisk (f0083761-fdb1-48bc-8dfd-86fd894d6832)            |
| hostId                               | dab453da6b0bd05902f3d80f6df83d108cfe9704e3d3c0cc903e7628                  |
| id                                   | b515db00-067d-4d9a-86be-9dea03c14d03                                      |
| image                                | pxeboot_cxp9025898_2r5b03 (0b67c2b1-2370-4b23-91f1-04236b5bba8e) |
| key_name                             | -                                                                         |
| metadata                             | {}                                                                        |
| name                                 | VM-1.15                                                                   |
| os-extended-volumes:volumes_attached | []                                                                        |
| progress                             | 0                                                                         |
| security_groups                      | default                                                                   |
| status                               | ACTIVE                                                                    |
| tenant_id                            | 36d1650d2c7f47d4be35a46f3bb6a28e                                          |
| updated                              | 2016-01-13T21:14:37Z                                                      |
| user_id                              | 928a6b5ff95341f5857c5161df7b6ca1                                          |
+--------------------------------------+---------------------------------------------------------------------------+


root@node-11:~# brctl show
bridge name	bridge id		STP enabled	interfaces
br-ex		8000.2c44fd7c96cc	no		eth0.35
							p_ff798dba-0
br-fw-admin		8000.2c44fd7c96cc	no		eth0
							p_eeee51a2-0
br-mgmt		8000.2c44fd7c96cc	no		eth0.1526
br-storage		8000.2c44fd7c96cc	no		eth0.1525
qbr07abdc1e-38		8000.0e00e0133aec	no		qvb07abdc1e-38
							tap07abdc1e-38
qbr101a4853-a9		8000.66349b3bf77d	no		qvb101a4853-a9
							tap101a4853-a9
qbr1e3b62fd-80		8000.d6c7c2e452ac	no		qvb1e3b62fd-80
							tap1e3b62fd-80
qbr26379086-40		8000.1a87ae64580e	no		qvb26379086-40
							tap26379086-40
qbr2871b06a-fb		8000.b638f3116d76	no		qvb2871b06a-fb
							tap2871b06a-fb
qbr29c06538-34		8000.ba1c5aac2726	no		qvb29c06538-34
							tap29c06538-34
qbr2efbc02d-33		8000.32e23aa5404e	no		qvb2efbc02d-33
							tap2efbc02d-33
qbr3298eeb5-a1		8000.667029f958ec	no		qvb3298eeb5-a1
							tap3298eeb5-a1
qbr3e79842d-eb		8000.e2d3c6aea326	no		qvb3e79842d-eb
							tap3e79842d-eb
qbr4805182f-0b		8000.9e3bf559e7c1	no		qvb4805182f-0b
							tap4805182f-0b
qbr5160349f-e7		8000.d263b9e4f324	no		qvb5160349f-e7
							tap5160349f-e7
qbr534c601a-0c		8000.ca0079ee8e55	no		qvb534c601a-0c
							tap534c601a-0c
qbr622ef3b6-a0		8000.625bd7a53dd5	no		qvb622ef3b6-a0
							tap622ef3b6-a0
qbr960d7784-82		8000.0642984683ea	no		qvb960d7784-82
							tap960d7784-82
qbr99faeb13-17		8000.a6476340bb75	no		qvb99faeb13-17
							tap99faeb13-17
qbra80a8610-ef		8000.3af49b35beff	no		qvba80a8610-ef
							tapa80a8610-ef
qbrab3661cd-b2		8000.d6dcaee6a0e7	no		qvbab3661cd-b2
							tapab3661cd-b2
qbrabbfad8e-05		8000.4e0f384dbfde	no		qvbabbfad8e-05
							tapabbfad8e-05
qbrb9bd0dcd-0c		8000.2a4cf0aac6ca	no		qvbb9bd0dcd-0c
							tapb9bd0dcd-0c
qbrc3a88d15-08		8000.da9fcf716879	no		qvbc3a88d15-08
							tapc3a88d15-08
qbrcf4d2014-ea		8000.063f92ac020e	no		qvbcf4d2014-ea
							tapcf4d2014-ea
qbrd15b94e7-05		8000.5a8a3d70a79d	no		qvbd15b94e7-05
							tapd15b94e7-05
qbrd3c76f84-6f		8000.66039e089f00	no		qvbd3c76f84-6f
							tapd3c76f84-6f
qbrd9d1a7c6-e2		8000.02f220117f85	no		qvbd9d1a7c6-e2
							tapd9d1a7c6-e2
qbrdd069c93-ad		8000.a6e25b3b1a82	no		qvbdd069c93-ad
							tapdd069c93-ad
qbre3ea8b73-13		8000.0e963b47dbc9	no		qvbe3ea8b73-13
							tape3ea8b73-13
qbree5d29b2-75		8000.d257b819b97a	no		qvbee5d29b2-75
							tapee5d29b2-75
qbrfdd2d84e-e4		8000.02c712bd61bb	no		qvbfdd2d84e-e4
							tapfdd2d84e-e4
root@node-11:~# virsh dumpxml instance-000000cc
<domain type='kvm' id='131'>
  <name>instance-000000cc</name>
  <uuid>b515db00-067d-4d9a-86be-9dea03c14d03</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
      <nova:package version="2015.1.1"/>
      <nova:name>VM-1.15</nova:name>
      <nova:creationTime>2016-01-13 21:14:29</nova:creationTime>
      <nova:flavor name="2vcpu_2048MBmem_1GBdisk">
        <nova:memory>2048</nova:memory>
        <nova:disk>1</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>2</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="928a6b5ff95341f5857c5161df7b6ca1">vepc</nova:user>
        <nova:project uuid="36d1650d2c7f47d4be35a46f3bb6a28e">vEPC</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="0b67c2b1-2370-4b23-91f1-04236b5bba8e"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <shares>2048</shares>
  </cputune>
    <sysinfo type='smbios'>
      <system>
        <entry name='manufacturer'>OpenStack Foundation</entry>
        <entry name='product'>OpenStack Nova</entry>
        <entry name='version'>2015.1.1</entry>
        <entry name='serial'>99fa98c8-e7ff-4ece-9155-3a0480f50bfd</entry>
        <entry name='uuid'>b515db00-067d-4d9a-86be-9dea03c14d03</entry>
      </system>
    </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='2' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/disk'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/var/lib/nova/instances/_base/5bea60e3738cbc5c2604ec84ce6a1ec6e1debfe6'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/disk.config'/>
      <backingStore/>
      <target dev='vdz' bus='virtio'/>
      <alias name='virtio-disk25'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='bridge'>
      <mac address='00:80:37:0e:0f:12'/>
      <source bridge='qbr3e79842d-eb'/>
      <target dev='tap3e79842d-eb'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='00:80:37:0e:0f:12'/>
      <source bridge='qbr960d7784-82'/>
      <target dev='tap960d7784-82'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/6'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5902' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
      <stats period='10'/>
    </memballoon>
  </devices>
</domain>

root@node-11:~# ifconfig qbr3e79842d-eb
qbr3e79842d-eb Link encap:Ethernet  HWaddr e2:d3:c6:ae:a3:26  
          inet6 addr: fe80::897:aeff:fee6:5e1b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:65000  Metric:1
          RX packets:52495 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2529458 (2.5 MB)  TX bytes:648 (648.0 B)

root@node-11:~# ifconfig qvb3e79842d-eb
qvb3e79842d-eb Link encap:Ethernet  HWaddr e2:d3:c6:ae:a3:26  
          inet6 addr: fe80::e0d3:c6ff:feae:a326/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:65000  Metric:1
          RX packets:1028373 errors:0 dropped:0 overruns:0 frame:0
          TX packets:929673 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:600674132 (600.6 MB)  TX bytes:429962708 (429.9 MB)

root@node-11:~# ifconfig tap3e79842d-eb
tap3e79842d-eb Link encap:Ethernet  HWaddr fe:80:37:0e:0f:12  
          inet6 addr: fe80::fc80:37ff:fe0e:f12/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:65000  Metric:1
          RX packets:967910 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1028334 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:431302055 (431.3 MB)  TX bytes:600737400 (600.7 MB)

root@node-11:~# brctl show qbr3e79842d-eb
bridge name	bridge id		STP enabled	interfaces
qbr3e79842d-eb		8000.e2d3c6aea326	no		qvb3e79842d-eb
							tap3e79842d-eb
root@node-11:~# ovs-vsctl show
cd41c9a1-d476-4b48-9d5c-e4c5f18afba5
    Bridge br-floating
        Port "p_ff798dba-0"
            Interface "p_ff798dba-0"
                type: internal
        Port br-floating
            Interface br-floating
                type: internal
    Bridge br-int
        fail_mode: secure
        Port "qvocf4d2014-ea"
            tag: 122
            Interface "qvocf4d2014-ea"
        Port "qvo99faeb13-17"
            tag: 124
            Interface "qvo99faeb13-17"
        Port "qvo29c06538-34"
            tag: 123
            Interface "qvo29c06538-34"
        Port "qvoabbfad8e-05"
            tag: 123
            Interface "qvoabbfad8e-05"
        Port "qvoab3661cd-b2"
            tag: 113
            Interface "qvoab3661cd-b2"
        Port "qvo534c601a-0c"
            tag: 112
            Interface "qvo534c601a-0c"
        Port "qvo07abdc1e-38"
            tag: 112
            Interface "qvo07abdc1e-38"
        Port "qvo622ef3b6-a0"
            tag: 112
            Interface "qvo622ef3b6-a0"
        Port "qvodd069c93-ad"
            tag: 121
            Interface "qvodd069c93-ad"
        Port "qvob9bd0dcd-0c"
            tag: 113
            Interface "qvob9bd0dcd-0c"
        Port "qvo101a4853-a9"
            tag: 113
            Interface "qvo101a4853-a9"
        Port "qvofdd2d84e-e4"
            tag: 115
            Interface "qvofdd2d84e-e4"
        Port "qvo3e79842d-eb"
            tag: 112
            Interface "qvo3e79842d-eb"
        Port "qvod3c76f84-6f"
            tag: 113
            Interface "qvod3c76f84-6f"
        Port "qvod9d1a7c6-e2"
            tag: 121
            Interface "qvod9d1a7c6-e2"
        Port "qvo1e3b62fd-80"
            tag: 113
            Interface "qvo1e3b62fd-80"
        Port "qvoc3a88d15-08"
            tag: 114
            Interface "qvoc3a88d15-08"
        Port "qvo26379086-40"
            tag: 114
            Interface "qvo26379086-40"
        Port "qvo2efbc02d-33"
            tag: 113
            Interface "qvo2efbc02d-33"
        Port "qvo4805182f-0b"
            tag: 115
            Interface "qvo4805182f-0b"
        Port "qvo960d7784-82"
            tag: 113
            Interface "qvo960d7784-82"
        Port br-int
            Interface br-int
                type: internal
        Port "qvoa80a8610-ef"
            tag: 113
            Interface "qvoa80a8610-ef"
        Port "qvod15b94e7-05"
            tag: 112
            Interface "qvod15b94e7-05"
        Port int-br-prv
            Interface int-br-prv
                type: patch
                options: {peer=phy-br-prv}
        Port "qvo5160349f-e7"
            tag: 122
            Interface "qvo5160349f-e7"
        Port "qvo3298eeb5-a1"
            tag: 124
            Interface "qvo3298eeb5-a1"
        Port "qvoee5d29b2-75"
            tag: 112
            Interface "qvoee5d29b2-75"
        Port "qvoe3ea8b73-13"
            tag: 112
            Interface "qvoe3ea8b73-13"
        Port "qvo2871b06a-fb"
            tag: 112
            Interface "qvo2871b06a-fb"
    Bridge br-prv
        Port br-prv
            Interface br-prv
                type: internal
        Port phy-br-prv
            Interface phy-br-prv
                type: patch
                options: {peer=int-br-prv}
        Port "p_eeee51a2-0"
            Interface "p_eeee51a2-0"
                type: internal
    ovs_version: "2.3.1"
root@node-11:~# ifconfig qvo3e79842d-eb
qvo3e79842d-eb Link encap:Ethernet  HWaddr da:e1:98:c1:6e:cf  
          inet6 addr: fe80::d8e1:98ff:fec1:6ecf/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:65000  Metric:1
          RX packets:931164 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1030766 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:430267581 (430.2 MB)  TX bytes:601031366 (601.0 MB)

root@node-11:~# ifconfig p_eeee51a2-0
p_eeee51a2-0 Link encap:Ethernet  HWaddr 6e:9d:56:fb:62:a5  
          inet6 addr: fe80::6c9d:56ff:fefb:62a5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:65000  Metric:1
          RX packets:86297635 errors:0 dropped:0 overruns:0 frame:0
          TX packets:143277215 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:66475322925 (66.4 GB)  TX bytes:35894211276 (35.8 GB)

root@node-11:~# ifconfig br-fw-admin
br-fw-admin Link encap:Ethernet  HWaddr 2c:44:fd:7c:9a:a4  
          inet addr:10.111.158.103  Bcast:10.111.158.111  Mask:255.255.255.240
          inet6 addr: fe80::2e44:fdff:fe7c:9aa4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:61629535 errors:0 dropped:2958811 overruns:0 frame:0
          TX packets:842703 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:7658578172 (7.6 GB)  TX bytes:313894760 (313.8 MB)

root@node-11:~# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 2c:44:fd:7c:9a:a4  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:184932186 errors:88320 dropped:29585 overruns:0 frame:88323
          TX packets:123054385 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:71762107044 (71.7 GB)  TX bytes:69565856487 (69.5 GB)
          Interrupt:32 

root@node-12:~# nova-manage  --version
2015.1.1
root@node-12:~# uname -a
Linux node-12.domain.tld 3.13.0-65-generic #105-Ubuntu SMP Mon Sep 21 18:50:58 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@node-12:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.3 LTS
Release:	14.04
Codename:	trusty

** Affects: neutron
     Importance: Undecided
         Status: New

** Attachment added: "NFS packet capture"
   https://bugs.launchpad.net/bugs/1542032/+attachment/4564105/+files/nfv-capture.zip

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1542032

Title:
  IP reassembly issue on the Linux bridges in Openstack

Status in neutron:
  New

Bug description:
  Hi,

  Sorry for text diagram. It does not look very well on this screen.
  Please, copy paste in a decent fixed width text editor.

  Thanks,

  Claude.

  
  Title: IP reassembly issue on the Linux bridges in Openstack
  ------------------------------------------------------------

  Summary: When the security groups and the Neutron firewall are active
  in Openstack, each and every VM virtual network interfaces (VNIC) is
  isolated in a Linux bridge and IP reassembly must be performed in
  order to allow firewall inspection of the traffic. The reassembled
  traffic sometimes exceed the capacity of the physical interfaces and
  the traffic is not forwarded properly.

  Linux bridge diagram:
  ---------------------

  ----------|                                       |--------------|
     VM     |                                       |      OVS     |
    ------- |       --------------      -------     | -----  ----- |    ------------    -------
    | TAP |-|-------| QBR bridge |------| QVB |-----|-|QVO|  | P |-|----| FW-ADMIN |----| PHY |
    ------- |       --------------      -------     | -----  ----- |    ------------    -------
            |                                       |              |
  --------- |                                       |--------------|

  Introduction:
  -------------

  In Openstack, the virtual machine (VM) uses the OpenvSwitch (OVS) for
  networking purposes. This is not a mandatory setup but this is a
  common setup in Openstack.

  When the Neutron firewall and the security groups are active, each VM
  VNIC, also called a tap interface, is connected to a Linux bridge.
  This is the QBR bridge. The QVB interface enables the network
  communication with OVS. The QVB interface interacts with the QVO
  interface in OVS.

  Security analysis is performed on the Linux bridge. In order to
  perform adequate traffic inspection, the fragmented traffic has to be
  re-assembled. The traffic is then forwarded according to Maximum
  Transmit Unit (MTU) of the interfaces in the bridge.

  The MTU values on all the interfaces are set to 65000 bytes. This is
  where a part of the problem experienced with NFV applications is
  observed.

  Analysis:
  ---------

  As a real life example, the NFV application uses NFS between VMs. NFS
  is a well known feature in Unix environments. This feature provides
  network file systems. This is the equivalent of a network drive in the
  Windows world.

  NFS is known to produce large frames. In this example, the VM1
  (169.254.4.242) send a larg NFS write instruction to the VM2. The
  example below shows a 5 KB packet. The traffic is fragmented in
  several packets as instructed by the VM1 VNIC. This is the desired
  behavior.

  root@node-11:~# tcpdump -e -n -i tap3e79842d-eb host 169.254.1.13

  23:46:48.938255 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 1514: 169.254.4.242.3015988240 > 169.254.1.13.2049: 1472 write fh Unknown/01000601B1198A1CB3CC4E1EA3AB0B26017B0AD653620700D59B28C700000000 4863 (4863) bytes @ 229376
  23:46:48.938271 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 1514: 169.254.4.242 > 169.254.1.13: ip-proto-17
  23:46:48.938279 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 1514: 169.254.4.242 > 169.254.1.13: ip-proto-17
  23:46:48.938287 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4 (0x0800), length 590: 169.254.4.242 > 169.254.1.13: ip-proto-17

  The same packet is found on the QVB interface in one large frame.

  root@node-11:~# tcpdump -e -n -i qvb3e79842d-eb host 169.254.1.13

  23:46:48.938322 00:80:37:0e:0f:12 > 00:80:37:0e:0b:12, ethertype IPv4
  (0x0800), length 5030: 169.254.4.242.3015988240 > 169.254.1.13.2049:
  4988 write fh
  Unknown/01000601B1198A1CB3CC4E1EA3AB0B26017B0AD653620700D59B28C700000000
  4863 (4863) bytes @ 229376

  Such large packets cannot cross physical interfaces without being
  fragmented again if jumbo frames support is not active in the network.
  Even with jumbo frames, the NFS frame size can easily cross the 9K
  barrier. NFS frame size up to 32 KB can be observed with NFS over UDP.

  For some reasons, this traffic does not seem to be transmitted
  properly between compute hosts in Openstack.

  Further investigations have revealed the large frames are leaving the
  OVS internal bridge (br-int) in direction of the private bridge (br-
  prv) using a patch interface in OVS. Once the traffic has reached this
  point, it uses the "P" interface (i.e.: p_eeee51a2-0) to reach another
  Linux bridge (br-fw-admin) where the physical interface is connected
  to. The "P" interface has its MTU set to 65000 and the the physical
  interface as long as the Linux bridge are set to 1500. A tcpdump
  analysis reveals the large frames are reaching the "P" interface and
  the Linux bridge. However, the traffic is not observed on the physical
  interface. The traffic does not use the DF bit.

  This is the reason why the VNF application works fine when all the VMs
  are located on the same compute host while the NFS application does
  not work properly when the VMs are using multiple compute hosts.
  Somehow, when a large frame needs to be sent over to another compute
  host, either the Linux bridge or the physical interface does not
  fragment the packet again properly. The information is dropped and
  lost.

  Remedy:
  -------

  As a workaround, the bridge-nf-call-iptables kernel parameters can be
  used to disable the bridge netfilter feature. The traffic is not re-
  assembled and the NFV application works like a charm. However, the
  traffic is not inspected by the firewall anymore and the security
  groups functions of the other VNFs/VMs are affected. This is a compute
  host wide setting and not a per Linux bridge setting.

  The modification can be applied in real time but all the other Linux
  bridges on the compute host are affected.

  root@node-11:~# cat /proc/sys/net/bridge/bridge-nf-call-iptables
  1

  root@node-11:~# echo "0" > /proc/sys/net/bridge/bridge-nf-call-
  iptables

  root@node-11:~# cat /proc/sys/net/bridge/bridge-nf-call-iptables 
  0

  The sysctl command can also be used to control the bridge-nf-call-
  iptables kernel parameter.

  Attachments:
  ------------

  Traffic capture traces showing a 22 KB NFS write operation (nfs-
  fragment-1frame.cap & nfs-reassembly-1frame.cap)

  Expectations:
  -------------

  - Find why the traffic is not re-fragmented before leaving the compute host
  - Fix the issue
  - Provide configuration remedy if applicable

  Note: ML2 port-security set to False does not help. The anti-spoofing
  are removed but IP reassembly is still performed although FW
  inspection is not needed if this feature is present.

  Printouts on the compute host (Openstack Kilo):
  -----------------------------------------------

  root@node-12:~# nova show VM-1.15
  +--------------------------------------+---------------------------------------------------------------------------+
  | Property                             | Value                                                                     |
  +--------------------------------------+---------------------------------------------------------------------------+
  | Internal-1 network                   | 169.254.4.242                                                             |
  | Internal-2 network                   | 30.30.102.4                                                               |
  | OS-DCF:diskConfig                    | MANUAL                                                                    |
  | OS-EXT-AZ:availability_zone          | nova                                                                      |
  | OS-EXT-SRV-ATTR:host                 | node-11.domain.tld                                                        |
  | OS-EXT-SRV-ATTR:hypervisor_hostname  | node-11.domain.tld                                                        |
  | OS-EXT-SRV-ATTR:instance_name        | instance-000000cc                                                         |
  | OS-EXT-STS:power_state               | 1                                                                         |
  | OS-EXT-STS:task_state                | -                                                                         |
  | OS-EXT-STS:vm_state                  | active                                                                    |
  | OS-SRV-USG:launched_at               | 2016-01-13T21:14:36.000000                                                |
  | OS-SRV-USG:terminated_at             | -                                                                         |
  | accessIPv4                           |                                                                           |
  | accessIPv6                           |                                                                           |
  | config_drive                         | True                                                                      |
  | created                              | 2016-01-13T21:13:58Z                                                      |
  | flavor                               | 2vcpu_2048MBmem_1GBdisk (f0083761-fdb1-48bc-8dfd-86fd894d6832)            |
  | hostId                               | dab453da6b0bd05902f3d80f6df83d108cfe9704e3d3c0cc903e7628                  |
  | id                                   | b515db00-067d-4d9a-86be-9dea03c14d03                                      |
  | image                                | pxeboot_cxp9025898_2r5b03 (0b67c2b1-2370-4b23-91f1-04236b5bba8e) |
  | key_name                             | -                                                                         |
  | metadata                             | {}                                                                        |
  | name                                 | VM-1.15                                                                   |
  | os-extended-volumes:volumes_attached | []                                                                        |
  | progress                             | 0                                                                         |
  | security_groups                      | default                                                                   |
  | status                               | ACTIVE                                                                    |
  | tenant_id                            | 36d1650d2c7f47d4be35a46f3bb6a28e                                          |
  | updated                              | 2016-01-13T21:14:37Z                                                      |
  | user_id                              | 928a6b5ff95341f5857c5161df7b6ca1                                          |
  +--------------------------------------+---------------------------------------------------------------------------+

  
  root@node-11:~# brctl show
  bridge name	bridge id		STP enabled	interfaces
  br-ex		8000.2c44fd7c96cc	no		eth0.35
  							p_ff798dba-0
  br-fw-admin		8000.2c44fd7c96cc	no		eth0
  							p_eeee51a2-0
  br-mgmt		8000.2c44fd7c96cc	no		eth0.1526
  br-storage		8000.2c44fd7c96cc	no		eth0.1525
  qbr07abdc1e-38		8000.0e00e0133aec	no		qvb07abdc1e-38
  							tap07abdc1e-38
  qbr101a4853-a9		8000.66349b3bf77d	no		qvb101a4853-a9
  							tap101a4853-a9
  qbr1e3b62fd-80		8000.d6c7c2e452ac	no		qvb1e3b62fd-80
  							tap1e3b62fd-80
  qbr26379086-40		8000.1a87ae64580e	no		qvb26379086-40
  							tap26379086-40
  qbr2871b06a-fb		8000.b638f3116d76	no		qvb2871b06a-fb
  							tap2871b06a-fb
  qbr29c06538-34		8000.ba1c5aac2726	no		qvb29c06538-34
  							tap29c06538-34
  qbr2efbc02d-33		8000.32e23aa5404e	no		qvb2efbc02d-33
  							tap2efbc02d-33
  qbr3298eeb5-a1		8000.667029f958ec	no		qvb3298eeb5-a1
  							tap3298eeb5-a1
  qbr3e79842d-eb		8000.e2d3c6aea326	no		qvb3e79842d-eb
  							tap3e79842d-eb
  qbr4805182f-0b		8000.9e3bf559e7c1	no		qvb4805182f-0b
  							tap4805182f-0b
  qbr5160349f-e7		8000.d263b9e4f324	no		qvb5160349f-e7
  							tap5160349f-e7
  qbr534c601a-0c		8000.ca0079ee8e55	no		qvb534c601a-0c
  							tap534c601a-0c
  qbr622ef3b6-a0		8000.625bd7a53dd5	no		qvb622ef3b6-a0
  							tap622ef3b6-a0
  qbr960d7784-82		8000.0642984683ea	no		qvb960d7784-82
  							tap960d7784-82
  qbr99faeb13-17		8000.a6476340bb75	no		qvb99faeb13-17
  							tap99faeb13-17
  qbra80a8610-ef		8000.3af49b35beff	no		qvba80a8610-ef
  							tapa80a8610-ef
  qbrab3661cd-b2		8000.d6dcaee6a0e7	no		qvbab3661cd-b2
  							tapab3661cd-b2
  qbrabbfad8e-05		8000.4e0f384dbfde	no		qvbabbfad8e-05
  							tapabbfad8e-05
  qbrb9bd0dcd-0c		8000.2a4cf0aac6ca	no		qvbb9bd0dcd-0c
  							tapb9bd0dcd-0c
  qbrc3a88d15-08		8000.da9fcf716879	no		qvbc3a88d15-08
  							tapc3a88d15-08
  qbrcf4d2014-ea		8000.063f92ac020e	no		qvbcf4d2014-ea
  							tapcf4d2014-ea
  qbrd15b94e7-05		8000.5a8a3d70a79d	no		qvbd15b94e7-05
  							tapd15b94e7-05
  qbrd3c76f84-6f		8000.66039e089f00	no		qvbd3c76f84-6f
  							tapd3c76f84-6f
  qbrd9d1a7c6-e2		8000.02f220117f85	no		qvbd9d1a7c6-e2
  							tapd9d1a7c6-e2
  qbrdd069c93-ad		8000.a6e25b3b1a82	no		qvbdd069c93-ad
  							tapdd069c93-ad
  qbre3ea8b73-13		8000.0e963b47dbc9	no		qvbe3ea8b73-13
  							tape3ea8b73-13
  qbree5d29b2-75		8000.d257b819b97a	no		qvbee5d29b2-75
  							tapee5d29b2-75
  qbrfdd2d84e-e4		8000.02c712bd61bb	no		qvbfdd2d84e-e4
  							tapfdd2d84e-e4
  root@node-11:~# virsh dumpxml instance-000000cc
  <domain type='kvm' id='131'>
    <name>instance-000000cc</name>
    <uuid>b515db00-067d-4d9a-86be-9dea03c14d03</uuid>
    <metadata>
      <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
        <nova:package version="2015.1.1"/>
        <nova:name>VM-1.15</nova:name>
        <nova:creationTime>2016-01-13 21:14:29</nova:creationTime>
        <nova:flavor name="2vcpu_2048MBmem_1GBdisk">
          <nova:memory>2048</nova:memory>
          <nova:disk>1</nova:disk>
          <nova:swap>0</nova:swap>
          <nova:ephemeral>0</nova:ephemeral>
          <nova:vcpus>2</nova:vcpus>
        </nova:flavor>
        <nova:owner>
          <nova:user uuid="928a6b5ff95341f5857c5161df7b6ca1">vepc</nova:user>
          <nova:project uuid="36d1650d2c7f47d4be35a46f3bb6a28e">vEPC</nova:project>
        </nova:owner>
        <nova:root type="image" uuid="0b67c2b1-2370-4b23-91f1-04236b5bba8e"/>
      </nova:instance>
    </metadata>
    <memory unit='KiB'>2097152</memory>
    <currentMemory unit='KiB'>2097152</currentMemory>
    <vcpu placement='static'>2</vcpu>
    <cputune>
      <shares>2048</shares>
    </cputune>
      <sysinfo type='smbios'>
        <system>
          <entry name='manufacturer'>OpenStack Foundation</entry>
          <entry name='product'>OpenStack Nova</entry>
          <entry name='version'>2015.1.1</entry>
          <entry name='serial'>99fa98c8-e7ff-4ece-9155-3a0480f50bfd</entry>
          <entry name='uuid'>b515db00-067d-4d9a-86be-9dea03c14d03</entry>
        </system>
      </sysinfo>
    <os>
      <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
      <boot dev='hd'/>
      <smbios mode='sysinfo'/>
    </os>
    <features>
      <acpi/>
      <apic/>
    </features>
    <cpu mode='host-model'>
      <model fallback='allow'/>
      <topology sockets='2' cores='1' threads='1'/>
    </cpu>
    <clock offset='utc'>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='rtc' tickpolicy='catchup'/>
      <timer name='hpet' present='no'/>
    </clock>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>destroy</on_crash>
    <devices>
      <emulator>/usr/bin/kvm</emulator>
      <disk type='file' device='disk'>
        <driver name='qemu' type='qcow2' cache='none'/>
        <source file='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/disk'/>
        <backingStore type='file' index='1'>
          <format type='raw'/>
          <source file='/var/lib/nova/instances/_base/5bea60e3738cbc5c2604ec84ce6a1ec6e1debfe6'/>
          <backingStore/>
        </backingStore>
        <target dev='vda' bus='virtio'/>
        <alias name='virtio-disk0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
      </disk>
      <disk type='file' device='disk'>
        <driver name='qemu' type='raw' cache='none'/>
        <source file='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/disk.config'/>
        <backingStore/>
        <target dev='vdz' bus='virtio'/>
        <alias name='virtio-disk25'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
      </disk>
      <controller type='usb' index='0'>
        <alias name='usb0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
      </controller>
      <controller type='pci' index='0' model='pci-root'>
        <alias name='pci.0'/>
      </controller>
      <interface type='bridge'>
        <mac address='00:80:37:0e:0f:12'/>
        <source bridge='qbr3e79842d-eb'/>
        <target dev='tap3e79842d-eb'/>
        <model type='virtio'/>
        <alias name='net0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </interface>
      <interface type='bridge'>
        <mac address='00:80:37:0e:0f:12'/>
        <source bridge='qbr960d7784-82'/>
        <target dev='tap960d7784-82'/>
        <model type='virtio'/>
        <alias name='net1'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </interface>
      <serial type='file'>
        <source path='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/console.log'/>
        <target port='0'/>
        <alias name='serial0'/>
      </serial>
      <serial type='pty'>
        <source path='/dev/pts/6'/>
        <target port='1'/>
        <alias name='serial1'/>
      </serial>
      <console type='file'>
        <source path='/var/lib/nova/instances/b515db00-067d-4d9a-86be-9dea03c14d03/console.log'/>
        <target type='serial' port='0'/>
        <alias name='serial0'/>
      </console>
      <input type='tablet' bus='usb'>
        <alias name='input0'/>
      </input>
      <input type='mouse' bus='ps2'/>
      <input type='keyboard' bus='ps2'/>
      <graphics type='vnc' port='5902' autoport='yes' listen='0.0.0.0' keymap='en-us'>
        <listen type='address' address='0.0.0.0'/>
      </graphics>
      <video>
        <model type='cirrus' vram='9216' heads='1'/>
        <alias name='video0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
      </video>
      <memballoon model='virtio'>
        <alias name='balloon0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
        <stats period='10'/>
      </memballoon>
    </devices>
  </domain>

  root@node-11:~# ifconfig qbr3e79842d-eb
  qbr3e79842d-eb Link encap:Ethernet  HWaddr e2:d3:c6:ae:a3:26  
            inet6 addr: fe80::897:aeff:fee6:5e1b/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST  MTU:65000  Metric:1
            RX packets:52495 errors:0 dropped:0 overruns:0 frame:0
            TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0 
            RX bytes:2529458 (2.5 MB)  TX bytes:648 (648.0 B)

  root@node-11:~# ifconfig qvb3e79842d-eb
  qvb3e79842d-eb Link encap:Ethernet  HWaddr e2:d3:c6:ae:a3:26  
            inet6 addr: fe80::e0d3:c6ff:feae:a326/64 Scope:Link
            UP BROADCAST RUNNING PROMISC MULTICAST  MTU:65000  Metric:1
            RX packets:1028373 errors:0 dropped:0 overruns:0 frame:0
            TX packets:929673 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000 
            RX bytes:600674132 (600.6 MB)  TX bytes:429962708 (429.9 MB)

  root@node-11:~# ifconfig tap3e79842d-eb
  tap3e79842d-eb Link encap:Ethernet  HWaddr fe:80:37:0e:0f:12  
            inet6 addr: fe80::fc80:37ff:fe0e:f12/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST  MTU:65000  Metric:1
            RX packets:967910 errors:0 dropped:0 overruns:0 frame:0
            TX packets:1028334 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:500 
            RX bytes:431302055 (431.3 MB)  TX bytes:600737400 (600.7 MB)

  root@node-11:~# brctl show qbr3e79842d-eb
  bridge name	bridge id		STP enabled	interfaces
  qbr3e79842d-eb		8000.e2d3c6aea326	no		qvb3e79842d-eb
  							tap3e79842d-eb
  root@node-11:~# ovs-vsctl show
  cd41c9a1-d476-4b48-9d5c-e4c5f18afba5
      Bridge br-floating
          Port "p_ff798dba-0"
              Interface "p_ff798dba-0"
                  type: internal
          Port br-floating
              Interface br-floating
                  type: internal
      Bridge br-int
          fail_mode: secure
          Port "qvocf4d2014-ea"
              tag: 122
              Interface "qvocf4d2014-ea"
          Port "qvo99faeb13-17"
              tag: 124
              Interface "qvo99faeb13-17"
          Port "qvo29c06538-34"
              tag: 123
              Interface "qvo29c06538-34"
          Port "qvoabbfad8e-05"
              tag: 123
              Interface "qvoabbfad8e-05"
          Port "qvoab3661cd-b2"
              tag: 113
              Interface "qvoab3661cd-b2"
          Port "qvo534c601a-0c"
              tag: 112
              Interface "qvo534c601a-0c"
          Port "qvo07abdc1e-38"
              tag: 112
              Interface "qvo07abdc1e-38"
          Port "qvo622ef3b6-a0"
              tag: 112
              Interface "qvo622ef3b6-a0"
          Port "qvodd069c93-ad"
              tag: 121
              Interface "qvodd069c93-ad"
          Port "qvob9bd0dcd-0c"
              tag: 113
              Interface "qvob9bd0dcd-0c"
          Port "qvo101a4853-a9"
              tag: 113
              Interface "qvo101a4853-a9"
          Port "qvofdd2d84e-e4"
              tag: 115
              Interface "qvofdd2d84e-e4"
          Port "qvo3e79842d-eb"
              tag: 112
              Interface "qvo3e79842d-eb"
          Port "qvod3c76f84-6f"
              tag: 113
              Interface "qvod3c76f84-6f"
          Port "qvod9d1a7c6-e2"
              tag: 121
              Interface "qvod9d1a7c6-e2"
          Port "qvo1e3b62fd-80"
              tag: 113
              Interface "qvo1e3b62fd-80"
          Port "qvoc3a88d15-08"
              tag: 114
              Interface "qvoc3a88d15-08"
          Port "qvo26379086-40"
              tag: 114
              Interface "qvo26379086-40"
          Port "qvo2efbc02d-33"
              tag: 113
              Interface "qvo2efbc02d-33"
          Port "qvo4805182f-0b"
              tag: 115
              Interface "qvo4805182f-0b"
          Port "qvo960d7784-82"
              tag: 113
              Interface "qvo960d7784-82"
          Port br-int
              Interface br-int
                  type: internal
          Port "qvoa80a8610-ef"
              tag: 113
              Interface "qvoa80a8610-ef"
          Port "qvod15b94e7-05"
              tag: 112
              Interface "qvod15b94e7-05"
          Port int-br-prv
              Interface int-br-prv
                  type: patch
                  options: {peer=phy-br-prv}
          Port "qvo5160349f-e7"
              tag: 122
              Interface "qvo5160349f-e7"
          Port "qvo3298eeb5-a1"
              tag: 124
              Interface "qvo3298eeb5-a1"
          Port "qvoee5d29b2-75"
              tag: 112
              Interface "qvoee5d29b2-75"
          Port "qvoe3ea8b73-13"
              tag: 112
              Interface "qvoe3ea8b73-13"
          Port "qvo2871b06a-fb"
              tag: 112
              Interface "qvo2871b06a-fb"
      Bridge br-prv
          Port br-prv
              Interface br-prv
                  type: internal
          Port phy-br-prv
              Interface phy-br-prv
                  type: patch
                  options: {peer=int-br-prv}
          Port "p_eeee51a2-0"
              Interface "p_eeee51a2-0"
                  type: internal
      ovs_version: "2.3.1"
  root@node-11:~# ifconfig qvo3e79842d-eb
  qvo3e79842d-eb Link encap:Ethernet  HWaddr da:e1:98:c1:6e:cf  
            inet6 addr: fe80::d8e1:98ff:fec1:6ecf/64 Scope:Link
            UP BROADCAST RUNNING PROMISC MULTICAST  MTU:65000  Metric:1
            RX packets:931164 errors:0 dropped:0 overruns:0 frame:0
            TX packets:1030766 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000 
            RX bytes:430267581 (430.2 MB)  TX bytes:601031366 (601.0 MB)

  root@node-11:~# ifconfig p_eeee51a2-0
  p_eeee51a2-0 Link encap:Ethernet  HWaddr 6e:9d:56:fb:62:a5  
            inet6 addr: fe80::6c9d:56ff:fefb:62a5/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST  MTU:65000  Metric:1
            RX packets:86297635 errors:0 dropped:0 overruns:0 frame:0
            TX packets:143277215 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0 
            RX bytes:66475322925 (66.4 GB)  TX bytes:35894211276 (35.8 GB)

  root@node-11:~# ifconfig br-fw-admin
  br-fw-admin Link encap:Ethernet  HWaddr 2c:44:fd:7c:9a:a4  
            inet addr:10.111.158.103  Bcast:10.111.158.111  Mask:255.255.255.240
            inet6 addr: fe80::2e44:fdff:fe7c:9aa4/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:61629535 errors:0 dropped:2958811 overruns:0 frame:0
            TX packets:842703 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0 
            RX bytes:7658578172 (7.6 GB)  TX bytes:313894760 (313.8 MB)

  root@node-11:~# ifconfig eth0
  eth0      Link encap:Ethernet  HWaddr 2c:44:fd:7c:9a:a4  
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:184932186 errors:88320 dropped:29585 overruns:0 frame:88323
            TX packets:123054385 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000 
            RX bytes:71762107044 (71.7 GB)  TX bytes:69565856487 (69.5 GB)
            Interrupt:32 

  root@node-12:~# nova-manage  --version
  2015.1.1
  root@node-12:~# uname -a
  Linux node-12.domain.tld 3.13.0-65-generic #105-Ubuntu SMP Mon Sep 21 18:50:58 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
  root@node-12:~# lsb_release -a
  No LSB modules are available.
  Distributor ID:	Ubuntu
  Description:	Ubuntu 14.04.3 LTS
  Release:	14.04
  Codename:	trusty

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1542032/+subscriptions


Follow ups