openstack team mailing list archive

[Greg Chavez <greg.chavez@xxxxxxxxx>]

Here's the last command output you asked for:

root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-ofctl dump-flows br-eth1
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=78793.694s, table=0, n_packets=6, n_bytes=468,
priority=2,in_port=6 actions=drop
 cookie=0x0, duration=78794.033s, table=0, n_packets=17355,
n_bytes=3335788, priority=1 actions=NORMAL


On Wed, Feb 20, 2013 at 10:57 AM, Greg Chavez <greg.chavez@xxxxxxxxx> wrote:

>
> Hey Anil, thanks for responding.  Here's the output:
>
> root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-vsctl show
> 9d9f7949-2b80-40c8-a9e0-6a116200ed96
>     Bridge br-int
>         Port br-int
>             Interface br-int
>                 type: internal
>         Port "int-br-eth1"
>             Interface "int-br-eth1"
>     Bridge "br-eth1"
>         Port "phy-br-eth1"
>             Interface "phy-br-eth1"
>         Port "br-eth1"
>             Interface "br-eth1"
>                 type: internal
>         Port "eth1"
>             Interface "eth1"
>     ovs_version: "1.4.3"
>
> root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-dpctl show
> system@br-eth1:
> lookups: hit:5227 missed:24022 lost:0
> flows: 1
> port 0: br-eth1 (internal)
>  port 1: eth1
> port 6: phy-br-eth1
> system@br-int:
> lookups: hit:2994 missed:13754 lost:0
>  flows: 1
> port 0: br-int (internal)
> port 2: int-br-eth1
>
> root@kvm-cs-sn-10i:/var/lib/nova/instances# brctl show
> bridge name bridge id STP enabled interfaces
> br-eth1 0000.bc305befedd1 no eth1
> phy-br-eth1
> br-int 0000.8ae31e5f7941 no int-br-eth1
> qbr5334a0cb-64 8000.76fb293fe9cf no qvb5334a0cb-64
>  vnet0
>
> root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-ofctl dump-flows br-int
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=75251.156s, table=0, n_packets=16581,
> n_bytes=3186436, priority=2,in_port=2 actions=drop
>  cookie=0x0, duration=75251.527s, table=0, n_packets=0, n_bytes=0,
> priority=1 actions=NORMAL
>
> Thanks!  I have until tomorrow to get this working, then my boss has
> mandating that I try Cloudstack.  Argh, but I'm so close!
>
>
> On Wed, Feb 20, 2013 at 10:29 AM, Anil Vishnoi <vishnoianil@xxxxxxxxx>wrote:
>
>> Hi Greg,
>>
>> Can you paste the output of following command from your compute node.
>>
>> ovs-vsctl show
>> ovs-dpctl show
>> brctl show
>>
>> ovs-ofctl dump-flows br-int
>> ovs-ofctl dump-flows br-eth1
>>
>> Because i think first issue we need to resolve here is why DHCP packet is
>> not leaving your compute host.
>>
>> Anil
>>
>>
>> On Wed, Feb 20, 2013 at 6:48 AM, Greg Chavez <greg.chavez@xxxxxxxxx>wrote:
>>
>>>
>>>
>>> From my perspective, it seems that the OVS bridges are not being brought
>>> up correctly.  As you can see in my earlier post, the integration bridge
>>> (int) and the physical interface bridges (br-ex and br-eth1) are downed.
>>>  I've tried to bring them up in promiscuous mode in the case of br-int, and
>>> with the physical interfaces ported to the bridge in the case of br-ex and
>>> br-eth1.  I've had no luck unfortunately.
>>>
>>> It seems that nothing is getting past br-int.  I can see BOOTP packets
>>> on the VM side of br-int, and I can see VTP packets on the physical side of
>>> br-int.  But that's where it ends.
>>>
>>> For example, when I reboot my VM, I see this:
>>>
>>> root@kvm-cs-sn-10i:/var/lib/nova/instances# tcpdump -i qvo5334a0cb-64
>>>
>>> tcpdump: WARNING: qvo5334a0cb-64: no IPv4 address assigned
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>> listening on qvo5334a0cb-64, link-type EN10MB (Ethernet), capture size
>>> 65535 bytes
>>>
>>> 13:42:08.099061 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>>> Request from fa:16:3e:06:48:09 (oui Unknown), length 280
>>> 13:42:08.101675 IP6 :: > ff02::16: HBH ICMP6, multicast listener report
>>> v2, 1 group record(s), length 28
>>> 13:42:08.161728 IP6 :: > ff02::1:ff06:4809: ICMP6, neighbor
>>> solicitation, who has fe80::f816:3eff:fe06:4809, length 24
>>> 13:42:08.373745 IP6 :: > ff02::16: HBH ICMP6, multicast listener report
>>> v2, 1 group record(s), length 28
>>> 13:42:11.102528 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>>> Request from fa:16:3e:06:48:09 (oui Unknown), length 280
>>> 13:42:14.105850 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>>> Request from fa:16:3e:06:48:09 (oui Unknown), length 280
>>>
>>> But that's as far as it goes.  The dhcp agent never get this.
>>>
>>> I"ve tried deleting and recreating the bridges, rebooting the systems,
>>> but nothing seems to work.  Maybe it's just the right combination of
>>> things.  I don't know.
>>>
>>> Help!
>>>
>>>
>>> On Tue, Feb 19, 2013 at 5:23 AM, Sylvain Bauza <
>>> sylvain.bauza@xxxxxxxxxxxx> wrote:
>>>
>>>>  Hi Greg,
>>>>
>>>> I did have trouble with DHCP assignation (see my previous post in this
>>>> list), which was being fixed by deleting ovs bridges on network node,
>>>> recreating them and restarting OVS plugin and L3/DHCP agents (which were
>>>> all on the same physical node).
>>>> Maybe it helps.
>>>>
>>>> Anyway, when DHCP'ing from your VM (asking for an IP), could you please
>>>> tcpdump :
>>>> 1. your virtual network interface on compute node
>>>> 2. your physical network interface on compute node
>>>> 3. your physical network interface on network node
>>>>
>>>> and see BOOTP/DHCP packets ?
>>>> On the physical layer, you should see GRE packets (provided you
>>>> correctly followed the mentioned guide) encapsulating your BOOTP/DHCP
>>>> packets.
>>>>
>>>> If that's OK, could you please issue the below commands (on the network
>>>> node) :
>>>>  - brctl show
>>>>  - ip a
>>>>  - ovs-vsctl show
>>>>  - route -n
>>>>
>>>> Thanks,
>>>> -Sylvain
>>>>
>>>> Le 19/02/2013 00:55, Greg Chavez a écrit :
>>>>
>>>> Third time I'm replying to my own message.  It seems like the initial
>>>> network state is a problem for many first time openstackers.  Surely
>>>> somewhere would be well to assist me.  I'm running out of time to make this
>>>> work.  Thanks.
>>>>
>>>>
>>>> On Sun, Feb 17, 2013 at 3:08 AM, Greg Chavez <greg.chavez@xxxxxxxxx>wrote:
>>>>
>>>>> I'm replying to my own message because I'm desperate.  My network
>>>>> situation is a mess.  I need to add this as well: my bridge interfaces are
>>>>> all down.  On my compute node:
>>>>>
>>>>>  root@kvm-cs-sn-10i:/var/lib/nova/instances/instance-00000005# ip
>>>>> addr show | grep ^[0-9]
>>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
>>>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
>>>>> qlen 1000
>>>>> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
>>>>> qlen 1000
>>>>> 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
>>>>> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
>>>>> 9: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
>>>>> 10: br-eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
>>>>> 13: phy-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>> pfifo_fast state UP qlen 1000
>>>>> 14: int-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>> pfifo_fast state UP qlen 1000
>>>>> 15: qbre56c5d9e-b6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>> noqueue state UP
>>>>> 16: qvoe56c5d9e-b6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
>>>>> qdisc pfifo_fast state UP qlen 1000
>>>>> 17: qvbe56c5d9e-b6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
>>>>> qdisc pfifo_fast master qbre56c5d9e-b6 state UP qlen 1000
>>>>> 19: qbrb805a9c9-11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>> noqueue state UP
>>>>> 20: qvob805a9c9-11: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
>>>>> qdisc pfifo_fast state UP qlen 1000
>>>>> 21: qvbb805a9c9-11: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
>>>>> qdisc pfifo_fast master qbrb805a9c9-11 state UP qlen 1000
>>>>> 34: qbr2b23c51f-02: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>> noqueue state UP
>>>>> 35: qvo2b23c51f-02: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
>>>>> qdisc pfifo_fast state UP qlen 1000
>>>>> 36: qvb2b23c51f-02: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
>>>>> qdisc pfifo_fast master qbr2b23c51f-02 state UP qlen 1000
>>>>> 37: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>>>>> master qbr2b23c51f-02 state UNKNOWN qlen 500
>>>>>
>>>>>  And on my network node:
>>>>>
>>>>>  root@knet-cs-gen-01i:~# ip addr show | grep ^[0-9]
>>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
>>>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
>>>>> qlen 1000
>>>>> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
>>>>> qlen 1000
>>>>> 4: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq
>>>>> state UP qlen 1000
>>>>> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
>>>>> 6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
>>>>> 7: br-eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
>>>>> 8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>>>>> state UNKNOWN
>>>>> 22: phy-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>> pfifo_fast state UP qlen 1000
>>>>> 23: int-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>> pfifo_fast state UP qlen 1000
>>>>>
>>>>>  I gave br-ex an IP and UP'ed it manually.  I assume this is correct.
>>>>>  By I honestly don't know.
>>>>>
>>>>>  Thanks.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 15, 2013 at 6:54 PM, Greg Chavez <greg.chavez@xxxxxxxxx>wrote:
>>>>>
>>>>>>
>>>>>>  Sigh.  So I abandoned RHEL 6.3, rekicked my systems and set up the
>>>>>> scale-ready installation described in these instructions:
>>>>>>
>>>>>>
>>>>>> https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst
>>>>>>
>>>>>>  Basically:
>>>>>>
>>>>>>  (o) controller node on a mgmt and public net
>>>>>> (o) network node (quantum and openvs) on a mgmt, net-config, and
>>>>>> public net
>>>>>>  (o) compute node is on a mgmt and net-config net
>>>>>>
>>>>>>  Took me just over an hour and ran into only a few easily-fixed
>>>>>> speed bumps.  But the VM networks are totally non-functioning.  VMs launch
>>>>>> but no network traffic can go in or out.
>>>>>>
>>>>>>  I'm particularly befuddled by these problems:
>>>>>>
>>>>>>  ( 1 ) This error in nova-compute:
>>>>>>
>>>>>>  ERROR nova.network.quantumv2 [-] _get_auth_token() failed
>>>>>>
>>>>>>  ( 2 ) No NAT rules on the compute node, which probably explains why
>>>>>> the VMs complain about not finding a network or being able to get metadata
>>>>>> from 169.254.169.254.
>>>>>>
>>>>>>  root@kvm-cs-sn-10i:~# iptables -t nat -S
>>>>>> -P PREROUTING ACCEPT
>>>>>> -P INPUT ACCEPT
>>>>>> -P OUTPUT ACCEPT
>>>>>> -P POSTROUTING ACCEPT
>>>>>> -N nova-api-metadat-OUTPUT
>>>>>> -N nova-api-metadat-POSTROUTING
>>>>>> -N nova-api-metadat-PREROUTING
>>>>>> -N nova-api-metadat-float-snat
>>>>>> -N nova-api-metadat-snat
>>>>>> -N nova-compute-OUTPUT
>>>>>> -N nova-compute-POSTROUTING
>>>>>> -N nova-compute-PREROUTING
>>>>>> -N nova-compute-float-snat
>>>>>> -N nova-compute-snat
>>>>>> -N nova-postrouting-bottom
>>>>>> -A PREROUTING -j nova-api-metadat-PREROUTING
>>>>>> -A PREROUTING -j nova-compute-PREROUTING
>>>>>> -A OUTPUT -j nova-api-metadat-OUTPUT
>>>>>> -A OUTPUT -j nova-compute-OUTPUT
>>>>>> -A POSTROUTING -j nova-api-metadat-POSTROUTING
>>>>>> -A POSTROUTING -j nova-compute-POSTROUTING
>>>>>> -A POSTROUTING -j nova-postrouting-bottom
>>>>>> -A nova-api-metadat-snat -j nova-api-metadat-float-snat
>>>>>> -A nova-compute-snat -j nova-compute-float-snat
>>>>>> -A nova-postrouting-bottom -j nova-api-metadat-snat
>>>>>> -A nova-postrouting-bottom -j nova-compute-snat
>>>>>>
>>>>>>  (3) A lastly, no default secgroup rules, whose function governs...
>>>>>> what exactly?  Connections to the VM's public or private IPs?  I guess I'm
>>>>>> just not sure if this is relevant to my overall problem of ZERO VM network
>>>>>> connectivity.
>>>>>>
>>>>>>  I seek guidance please.  Thanks.
>>>>>>
>>>>>>