openstack team mailing list archive

Thread
Date
Re: Issues with openvswitch agent on Quantum/network node

To: "Sreerekha Shenoy (sresheno)" <sresheno@xxxxxxxxx>
From: Balamurugan V G <balamuruganvg@xxxxxxxxx>
Date: Thu, 20 Jun 2013 18:50:04 +0530
Cc: "openstack@xxxxxxxxxxxxxxxxxxx" <openstack@xxxxxxxxxxxxxxxxxxx>
In-reply-to: <F719C2C7E894104CA53C0F3912FCA2F414E83695@xmb-rcd-x09.cisco.com>
Run a ping in your VM to its network gateway and do the following:

root@openstack-network:~# quantum router-list
+--------------------------------------+----------------------+--------------------------------------------------------+
| id                                   | name                 |
external_gateway_info                                  |
+--------------------------------------+----------------------+--------------------------------------------------------+
| 2abf6d6d-b2ac-4fad-99e5-ebf96e1ce167 | auto_router_1 |
{"network_id": "d7d40dba-6d14-4ba5-89dd-972fde4361ab"} |
+--------------------------------------+----------------------+--------------------------------------------------------+
root@openstack-network:~# ip netns exec
qrouter-2abf6d6d-b2ac-4fad-99e5-ebf96e1ce167 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         1.1.1.1       0.0.0.0         UG    0      0        0
qg-ec4cd5f1-e6
1.1.1.0       0.0.0.0         255.255.255.0   U     0      0        0
qg-ec4cd5f1-e6
2.2.2.0     0.0.0.0         255.255.255.0   U     0      0        0
qr-272f8aec-37
root@openstack-network:~# ip netns exec
qrouter-2abf6d6d-b2ac-4fad-99e5-ebf96e1ce167 tcpdump -i qr-272f8aec-37
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on qr-272f8aec-37, link-type EN10MB (Ethernet), capture size
65535 bytes
^C18:44:26.406660 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008,
seq 56, length 64
18:44:26.406692 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
56, length 64
18:44:27.406690 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
57, length 64
18:44:27.406724 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
57, length 64
18:44:28.406695 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
58, length 64
18:44:28.406721 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
58, length 64
18:44:29.406683 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
59, length 64
18:44:29.406716 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
59, length 64
18:44:30.406668 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
60, length 64
18:44:30.406690 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
60, length 64
18:44:31.406666 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
61, length 64
18:44:31.406701 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
61, length 64
18:44:32.406688 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
62, length 64
18:44:32.406709 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
62, length 64
18:44:32.461101 IP 2.2.2.4.627 > 10.203.15.14.31995: UDP, length 100
18:44:33.406671 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
63, length 64
18:44:33.406704 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
63, length 64
18:44:34.406666 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
64, length 64
18:44:34.406692 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
64, length 64
18:44:35.406674 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
65, length 64
18:44:35.406708 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
65, length 64
18:44:36.406687 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
66, length 64
18:44:36.406719 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
66, length 64
18:44:37.406672 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
67, length 64
18:44:37.406698 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
67, length 64
18:44:38.406669 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
68, length 64
18:44:38.406695 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
68, length 64
18:44:39.406712 IP 2.2.2.4 > 2.2.2.1: ICMP echo request, id 38008, seq
69, length 64
18:44:39.406747 IP 2.2.2.1 > 2.2.2.4: ICMP echo reply, id 38008, seq
69, length 64

29 packets captured
29 packets received by filter
0 packets dropped by kernel
root@openstack-network:~#

2.2.2.0/24 is my VM's network.

You should see that the packets are reaching the the network node from
the compute node where your VM is running. This might help to narrow
down.

Hope this helps. I do not know of any tools as such except general
linux commands to debug the issues.

Regards,
Balu



On Thu, Jun 20, 2013 at 6:36 PM, Sreerekha Shenoy (sresheno)
<sresheno@xxxxxxxxx> wrote:
> Hi,
>
> It’s been a week since we are stuck in getting this setup working
>
> Any pointers on tools / document to move further would be really
> appreciated.
>
>
>
> (currently we are lagging in getting our hands onto tools that can help
> tracing the packets inside OVS to and from VMs….)
>
>
>
> Thanks,
>
> Best Regards,
>
> Rekha
>
>
>
> From: Sreerekha Shenoy (sresheno)
> Sent: Thursday, June 20, 2013 6:19 PM
> To: openstack@xxxxxxxxxxxxxxxxxxx
> Subject: RE: Issues with openvswitch agent on Quantum/network node
>
>
>
> Hi,
>
> What are the good to know tools and networking concepts for me to be able to
> debug connectivity issues inside Openstack VLAN mode of OVS?
>
>
>
> Thanks,
>
> Best Regards,
>
> Rekha
>
>
>
> From: Sreerekha Shenoy (sresheno)
> Sent: Friday, June 14, 2013 2:23 PM
> To: openstack@xxxxxxxxxxxxxxxxxxx
> Subject: Issues with openvswitch agent on Quantum/network node
>
>
>
> Hi,
>
> I have a multi node grizzly setup using the following guide:
>
> https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst
>
> My network topology is as attached and is using GRE tunnel based VLAN.
>
>
>
> I have VMs spawning on compute for cirros images.
>
> However, inter-VM ping works (e.g. 50.50.1.4 can ping 50.50.1.5), but the
> following DON’T work:
>
> a)      VM cannot ping 50.50.1.1
>
> b)      VM cannot ping Compute ETH1 (data n/w) , 10.20.20.53
>
> c)       VM cannot be reachable from IP Namespace either (ip netns exec
> qrouter-xxxx ping 50.50.1.4 does NOT work)
>
>
>
> Apart from the same, my network node has the following issue AFTER a reboot:
>
> ·          On network node the following agent is not running i.e.
> “quantum-plugin-openvswitch-agent”
>
> ·         Logs of this agent indicates it needs the plugin upgrade....dont
> have the precise error log..
>
> o   ovs-vswitchd.log indicates the following:
>
> §  Apr 16 22:08:39|00482|netdev|WARN|Dropped 233 log messages in last 12
> seconds (most recently, 1 seconds ago) due to excessive rate
>
> §  Apr 16 22:08:39|00483|netdev|WARN|failed to get flags for network device
> qr-12091c58-8a: No such device
>
> §  Apr 16 22:08:43|00484|netdev_linux|WARN|Dropped 35 log messages in last
> 15 seconds (most recently, 5 seconds ago) due to excessive rate
>
> ·         The version of OSK-plugin-ovs-agent is 2013.1.1-0ubuntu1
>
> ·         The result of the above being that “br-tun DOES NOT have gre-2 /
> gre-1 port on it” for both compute and network..
>
>
>
> Please let me know what could be done to:
>
> 1.       get the ovs-agent running on compute
>
> 2.       have VMs pingable
>
> ================================================================================================================
>
> compute Node output:
>
> root@computenode:~# ovs-vsctl show
>
> 4aad5da2-861d-4190-a50b-37dfc5ca237a
>
>     Bridge br-tun
>
>         Port patch-int
>
>             Interface patch-int
>
>                 type: patch
>
>                 options: {peer=patch-tun}
>
>         Port br-tun
>
>             Interface br-tun
>
>                 type: internal
>
>     Bridge br-int
>
>         Port "qvo47a58f32-be"
>
>             Interface "qvo47a58f32-be"
>
>         Port "qvo19e7cd4c-c0"
>
>             tag: 1
>
>             Interface "qvo19e7cd4c-c0"
>
>         Port "qvobaa01ac1-69"
>
>             tag: 1
>
>             Interface "qvobaa01ac1-69"
>
>         Port patch-tun
>
>             Interface patch-tun
>
>                 type: patch
>
>                 options: {peer=patch-int}
>
>         Port br-int
>
>             Interface br-int
>
>                 type: internal
>
>     ovs_version: "1.4.0+build0"
>
>
>
> ================================================================================================================
>
>
>
> Network node output:
>
>
>
> root@networknode:~# ovs-vsctl show
>
> 9e17bb8c-d69d-48ab-9333-b22447c0c268
>
>     Bridge br-tun
>
>         Port patch-int
>
>             Interface patch-int
>
>                 type: patch
>
>                 options: {peer=patch-tun}
>
>         Port br-tun
>
>             Interface br-tun
>
>                 type: internal
>
>     Bridge br-int
>
>         Port patch-tun
>
>             Interface patch-tun
>
>                 type: patch
>
>                 options: {peer=patch-int}
>
>         Port br-int
>
>             Interface br-int
>
>                 type: internal
>
>         Port "qr-a7ab3f7c-8e"
>
>             Interface "qr-a7ab3f7c-8e"
>
>                 type: internal
>
>         Port "tap6a322215-f0"
>
>             tag: 1
>
>             Interface "tap6a322215-f0"
>
>                 type: internal
>
>     Bridge br-ex
>
>         Port br-ex
>
>             Interface br-ex
>
>                 type: internal
>
>         Port "qg-1d552cbd-a0"
>
>             Interface "qg-1d552cbd-a0"
>
>                 type: internal
>
>         Port "eth2"
>
>             Interface "eth2"
>
>     ovs_version: "1.4.0+build0"
>
>
>
>
>
> Thanks and Regards,
>
> Rashmi S N
>
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
Follow ups

Re: Issues with openvswitch agent on Quantum/network node
From: Sreerekha Shenoy (sresheno), 2013-06-20
References

Re: Issues with openvswitch agent on Quantum/network node
From: Sreerekha Shenoy (sresheno), 2013-06-20