yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #02159
[Bug 1166921] Re: OVS and namespace l3 routing issue
This all came back to promiscuous ports being denied on the switch in-
case anyone comes across this
** Changed in: quantum
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to quantum.
https://bugs.launchpad.net/bugs/1166921
Title:
OVS and namespace l3 routing issue
Status in OpenStack Quantum (virtual network service):
Invalid
Bug description:
Summary: namespaces are unable to ping upstream gateway
quantum version: git grizzly/stable
os: Ubuntu 12.10
Confing files atttached
kernel: linux ubuntu 3.5.0-26-generic
software:
ovs-vsctl --version
ovs-vsctl (Open vSwitch) 1.4.3
Compiled Feb 18 2013 12:58:42
lsmod:
Module Size Used by
xt_conntrack 12761 1
ip6table_filter 12816 1
ip6_tables 27208 1 ip6table_filter
ipt_REDIRECT 12550 1
iptable_nat 13183 2
nf_nat 25255 2 ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4 14481 4 iptable_nat,nf_nat
nf_conntrack 82634 4 xt_conntrack,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4 12730 1 nf_conntrack_ipv4
xt_tcpudp 12604 2
iptable_filter 12811 2
ip_tables 26996 2 iptable_nat,iptable_filter
x_tables 29757 8
xt_conntrack,ip6table_filter,ip6_tables,ipt_REDIRECT,iptable_nat,xt_tcpudp,iptable_filter,ip_tables
vmxnet 26594 0
vmblock 18758 0
vmsync 13025 0
vmhgfs 63467 0
brcompat 13513 0
openvswitch 84219 4 brcompat
coretemp 13401 0
ppdev 17074 0
vmw_balloon 12674 0
microcode 22804 0
psmouse 95595 0
serio_raw 13216 0
parport_pc 32689 1
vmwgfx 121449 0
mac_hid 13206 0
ttm 83596 1 vmwgfx
vmci 101575 1 vmhgfs
drm 288721 2 vmwgfx,ttm
i2c_piix4 13168 0
shpchp 37109 0
lp 17760 0
parport 46346 3 ppdev,parport_pc,lp
floppy 69453 0
e1000 114937 0
mptspi 22530 2
mptscsih 40290 1 mptspi
mptbase 101920 2 mptspi,mptscsih
Details:
The environment is setup in a multi-node environment with namespacing enabled and the namespaces get created:
# ip netns
qdhcp-f178b2a3-870f-4648-b812-b347637c55e4
qrouter-2fe50115-8678-4f03-93e9-1ada8ac71f8c
The network devices in the namespaces get created and the ips setup:
# ip netns exec qrouter-2fe50115-8678-4f03-93e9-1ada8ac71f8c ifconfig -a
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:13 errors:0 dropped:0 overruns:0 frame:0
TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1108 (1.1 KB) TX bytes:1108 (1.1 KB)
qg-df16381e-50 Link encap:Ethernet HWaddr fa:16:3e:9b:39:47
inet addr:XX.XX.186.2 Bcast:XX.XX.186.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe9b:3947/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11968 errors:0 dropped:0 overruns:0 frame:0
TX packets:67 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:769542 (769.5 KB) TX bytes:3350 (3.3 KB)
qr-7fb2b954-7a Link encap:Ethernet HWaddr fa:16:3e:e8:ad:6d
inet addr:10.0.0.1 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fee8:ad6d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:240 errors:0 dropped:0 overruns:0 frame:0
TX packets:222 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:21176 (21.1 KB) TX bytes:23236 (23.2 KB)
and the routing in the namspace is added:
# ip netns exec qrouter-2fe50115-8678-4f03-93e9-1ada8ac71f8c netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 XX.XX.186.1 0.0.0.0 UG 0 0 0 qg-df16381e-50
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-7fb2b954-7a
XX.XX.186.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-df16381e-50
but the namespace is unable to ping it's gateway:
# ip netns exec qrouter-2fe50115-8678-4f03-93e9-1ada8ac71f8c ping XX.XX.186.1
PING XX.XX.186.1 (XX.XX.186.1) 56(84) bytes of data.
^C
--- XX.XX.186.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1008ms
where as the host can:
# ping XX.XX.186.1
PING XX.XX.186.1 (XX.XX.186.1) 56(84) bytes of data.
64 bytes from XX.XX.186.1: icmp_req=1 ttl=64 time=4.81 ms
64 bytes from XX.XX.186.1: icmp_req=2 ttl=64 time=0.230 ms
^C
--- XX.XX.186.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.230/2.524/4.818/2.294 ms
I have check that it is not a firewall issue on the gateway device and
I do see the following error in the ovs logs
# tail /var/log/openvswitch/ovs-vswitchd.log
Apr 09 19:25:49|01321|netdev_linux|WARN|Dropped 35 log messages in last 15 seconds (most recently, 5 seconds ago) due to excessive rate
Apr 09 19:25:49|01322|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on tap4fc7ba3c-78 device failed: No such device
Apr 09 19:25:58|01323|netdev|WARN|Dropped 209 log messages in last 12 seconds (most recently, 1 seconds ago) due to excessive rate
Apr 09 19:25:58|01324|netdev|WARN|failed to get flags for network device qr-7fb2b954-7a: No such device
Apr 09 19:25:59|01325|netdev_linux|WARN|Dropped 23 log messages in last 10 seconds (most recently, 5 seconds ago) due to excessive rate
Apr 09 19:25:59|01326|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on tap4fc7ba3c-78 device failed: No such device
Apr 09 19:26:09|01327|netdev_linux|WARN|Dropped 23 log messages in last 10 seconds (most recently, 5 seconds ago) due to excessive rate
Apr 09 19:26:09|01328|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on tap4fc7ba3c-78 device failed: No such device
Apr 09 19:26:10|01329|netdev|WARN|Dropped 227 log messages in last 12 seconds (most recently, 1 seconds ago) due to excessive rate
Apr 09 19:26:10|01330|netdev|WARN|failed to get flags for network device qr-7fb2b954-7a: No such device
further info:
ovs-vsctl show
66f89964-9ba5-4755-ac32-021c183b3432
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port "qg-df16381e-50"
Interface "qg-df16381e-50"
type: internal
Port "eth1"
Interface "eth1"
Bridge br-tun
Port "gre-1"
Interface "gre-1"
type: gre
options: {in_key=flow, out_key=flow, remote_ip="10.10.20.53"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal
Bridge br-int
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "tap4fc7ba3c-78"
tag: 1
Interface "tap4fc7ba3c-78"
type: internal
Port br-int
Interface br-int
type: internal
Port "qr-7fb2b954-7a"
tag: 1
Interface "qr-7fb2b954-7a"
type: internal
ovs_version: "1.4.3"
To manage notifications about this bug go to:
https://bugs.launchpad.net/quantum/+bug/1166921/+subscriptions
