openstack team mailing list archive

[Salvatore Orlando <sorlando@xxxxxxxxxx>]

Hi Greg,

I would like to understand a little bit better your setup.
I scrolled through the thread, but I'm not sure if you've already
provided the information I'm asking.

On the compute node, the pair phy-br-eth1/int-br-eth1 is a veth pair
or a OVS patch port?
I'm assuming you're using the OVS plugin (I think I'm not wrong here,
but please correct me in case I am). It might worth knowing if you're
doing vlan or gre-based isolation with it.

Salvatore

On 20 February 2013 17:35, Greg Chavez <greg.chavez@xxxxxxxxx> wrote:
>
> Here's the last command output you asked for:
>
> root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-ofctl dump-flows br-eth1
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=78793.694s, table=0, n_packets=6, n_bytes=468,
> priority=2,in_port=6 actions=drop
>  cookie=0x0, duration=78794.033s, table=0, n_packets=17355, n_bytes=3335788,
> priority=1 actions=NORMAL
>
>
> On Wed, Feb 20, 2013 at 10:57 AM, Greg Chavez <greg.chavez@xxxxxxxxx> wrote:
>>
>>
>> Hey Anil, thanks for responding.  Here's the output:
>>
>> root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-vsctl show
>> 9d9f7949-2b80-40c8-a9e0-6a116200ed96
>>     Bridge br-int
>>         Port br-int
>>             Interface br-int
>>                 type: internal
>>         Port "int-br-eth1"
>>             Interface "int-br-eth1"
>>     Bridge "br-eth1"
>>         Port "phy-br-eth1"
>>             Interface "phy-br-eth1"
>>         Port "br-eth1"
>>             Interface "br-eth1"
>>                 type: internal
>>         Port "eth1"
>>             Interface "eth1"
>>     ovs_version: "1.4.3"
>>
>> root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-dpctl show
>> system@br-eth1:
>> lookups: hit:5227 missed:24022 lost:0
>> flows: 1
>> port 0: br-eth1 (internal)
>> port 1: eth1
>> port 6: phy-br-eth1
>> system@br-int:
>> lookups: hit:2994 missed:13754 lost:0
>> flows: 1
>> port 0: br-int (internal)
>> port 2: int-br-eth1
>>
>> root@kvm-cs-sn-10i:/var/lib/nova/instances# brctl show
>> bridge name bridge id STP enabled interfaces
>> br-eth1 0000.bc305befedd1 no eth1
>> phy-br-eth1
>> br-int 0000.8ae31e5f7941 no int-br-eth1
>> qbr5334a0cb-64 8000.76fb293fe9cf no qvb5334a0cb-64
>> vnet0
>>
>> root@kvm-cs-sn-10i:/var/lib/nova/instances# ovs-ofctl dump-flows br-int
>> NXST_FLOW reply (xid=0x4):
>>  cookie=0x0, duration=75251.156s, table=0, n_packets=16581,
>> n_bytes=3186436, priority=2,in_port=2 actions=drop
>>  cookie=0x0, duration=75251.527s, table=0, n_packets=0, n_bytes=0,
>> priority=1 actions=NORMAL
>>
>> Thanks!  I have until tomorrow to get this working, then my boss has
>> mandating that I try Cloudstack.  Argh, but I'm so close!
>>
>>
>> On Wed, Feb 20, 2013 at 10:29 AM, Anil Vishnoi <vishnoianil@xxxxxxxxx>
>> wrote:
>>>
>>> Hi Greg,
>>>
>>> Can you paste the output of following command from your compute node.
>>>
>>> ovs-vsctl show
>>> ovs-dpctl show
>>> brctl show
>>>
>>> ovs-ofctl dump-flows br-int
>>> ovs-ofctl dump-flows br-eth1
>>>
>>> Because i think first issue we need to resolve here is why DHCP packet is
>>> not leaving your compute host.
>>>
>>> Anil
>>>
>>>
>>> On Wed, Feb 20, 2013 at 6:48 AM, Greg Chavez <greg.chavez@xxxxxxxxx>
>>> wrote:
>>>>
>>>>
>>>>
>>>> From my perspective, it seems that the OVS bridges are not being brought
>>>> up correctly.  As you can see in my earlier post, the integration bridge
>>>> (int) and the physical interface bridges (br-ex and br-eth1) are downed.
>>>> I've tried to bring them up in promiscuous mode in the case of br-int, and
>>>> with the physical interfaces ported to the bridge in the case of br-ex and
>>>> br-eth1.  I've had no luck unfortunately.
>>>>
>>>> It seems that nothing is getting past br-int.  I can see BOOTP packets
>>>> on the VM side of br-int, and I can see VTP packets on the physical side of
>>>> br-int.  But that's where it ends.
>>>>
>>>> For example, when I reboot my VM, I see this:
>>>>
>>>> root@kvm-cs-sn-10i:/var/lib/nova/instances# tcpdump -i qvo5334a0cb-64
>>>>
>>>> tcpdump: WARNING: qvo5334a0cb-64: no IPv4 address assigned
>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>> decode
>>>> listening on qvo5334a0cb-64, link-type EN10MB (Ethernet), capture size
>>>> 65535 bytes
>>>>
>>>> 13:42:08.099061 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>>>> Request from fa:16:3e:06:48:09 (oui Unknown), length 280
>>>> 13:42:08.101675 IP6 :: > ff02::16: HBH ICMP6, multicast listener report
>>>> v2, 1 group record(s), length 28
>>>> 13:42:08.161728 IP6 :: > ff02::1:ff06:4809: ICMP6, neighbor
>>>> solicitation, who has fe80::f816:3eff:fe06:4809, length 24
>>>> 13:42:08.373745 IP6 :: > ff02::16: HBH ICMP6, multicast listener report
>>>> v2, 1 group record(s), length 28
>>>> 13:42:11.102528 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>>>> Request from fa:16:3e:06:48:09 (oui Unknown), length 280
>>>> 13:42:14.105850 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>>>> Request from fa:16:3e:06:48:09 (oui Unknown), length 280
>>>>
>>>> But that's as far as it goes.  The dhcp agent never get this.
>>>>
>>>> I"ve tried deleting and recreating the bridges, rebooting the systems,
>>>> but nothing seems to work.  Maybe it's just the right combination of things.
>>>> I don't know.
>>>>
>>>> Help!
>>>>
>>>>
>>>> On Tue, Feb 19, 2013 at 5:23 AM, Sylvain Bauza
>>>> <sylvain.bauza@xxxxxxxxxxxx> wrote:
>>>>>
>>>>> Hi Greg,
>>>>>
>>>>> I did have trouble with DHCP assignation (see my previous post in this
>>>>> list), which was being fixed by deleting ovs bridges on network node,
>>>>> recreating them and restarting OVS plugin and L3/DHCP agents (which were all
>>>>> on the same physical node).
>>>>> Maybe it helps.
>>>>>
>>>>> Anyway, when DHCP'ing from your VM (asking for an IP), could you please
>>>>> tcpdump :
>>>>> 1. your virtual network interface on compute node
>>>>> 2. your physical network interface on compute node
>>>>> 3. your physical network interface on network node
>>>>>
>>>>> and see BOOTP/DHCP packets ?
>>>>> On the physical layer, you should see GRE packets (provided you
>>>>> correctly followed the mentioned guide) encapsulating your BOOTP/DHCP
>>>>> packets.
>>>>>
>>>>> If that's OK, could you please issue the below commands (on the network
>>>>> node) :
>>>>>  - brctl show
>>>>>  - ip a
>>>>>  - ovs-vsctl show
>>>>>  - route -n
>>>>>
>>>>> Thanks,
>>>>> -Sylvain
>>>>>
>>>>> Le 19/02/2013 00:55, Greg Chavez a écrit :
>>>>>
>>>>> Third time I'm replying to my own message.  It seems like the initial
>>>>> network state is a problem for many first time openstackers.  Surely
>>>>> somewhere would be well to assist me.  I'm running out of time to make this
>>>>> work.  Thanks.
>>>>>
>>>>>
>>>>> On Sun, Feb 17, 2013 at 3:08 AM, Greg Chavez <greg.chavez@xxxxxxxxx>
>>>>> wrote:
>>>>>>
>>>>>> I'm replying to my own message because I'm desperate.  My network
>>>>>> situation is a mess.  I need to add this as well: my bridge interfaces are
>>>>>> all down.  On my compute node:
>>>>>>
>>>>>> root@kvm-cs-sn-10i:/var/lib/nova/instances/instance-00000005# ip addr
>>>>>> show | grep ^[0-9]
>>>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
>>>>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
>>>>>> qlen 1000
>>>>>> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
>>>>>> qlen 1000
>>>>>> 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen
>>>>>> 1000
>>>>>> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen
>>>>>> 1000
>>>>>> 9: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
>>>>>> 10: br-eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
>>>>>> 13: phy-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>>> pfifo_fast state UP qlen 1000
>>>>>> 14: int-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>>> pfifo_fast state UP qlen 1000
>>>>>> 15: qbre56c5d9e-b6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>>> noqueue state UP
>>>>>> 16: qvoe56c5d9e-b6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
>>>>>> qdisc pfifo_fast state UP qlen 1000
>>>>>> 17: qvbe56c5d9e-b6: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
>>>>>> qdisc pfifo_fast master qbre56c5d9e-b6 state UP qlen 1000
>>>>>> 19: qbrb805a9c9-11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>>> noqueue state UP
>>>>>> 20: qvob805a9c9-11: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
>>>>>> qdisc pfifo_fast state UP qlen 1000
>>>>>> 21: qvbb805a9c9-11: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
>>>>>> qdisc pfifo_fast master qbrb805a9c9-11 state UP qlen 1000
>>>>>> 34: qbr2b23c51f-02: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>>> noqueue state UP
>>>>>> 35: qvo2b23c51f-02: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
>>>>>> qdisc pfifo_fast state UP qlen 1000
>>>>>> 36: qvb2b23c51f-02: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
>>>>>> qdisc pfifo_fast master qbr2b23c51f-02 state UP qlen 1000
>>>>>> 37: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>>>>>> master qbr2b23c51f-02 state UNKNOWN qlen 500
>>>>>>
>>>>>> And on my network node:
>>>>>>
>>>>>> root@knet-cs-gen-01i:~# ip addr show | grep ^[0-9]
>>>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
>>>>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
>>>>>> qlen 1000
>>>>>> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
>>>>>> qlen 1000
>>>>>> 4: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq
>>>>>> state UP qlen 1000
>>>>>> 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen
>>>>>> 1000
>>>>>> 6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
>>>>>> 7: br-eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
>>>>>> 8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>>>>>> state UNKNOWN
>>>>>> 22: phy-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>>> pfifo_fast state UP qlen 1000
>>>>>> 23: int-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>>> pfifo_fast state UP qlen 1000
>>>>>>
>>>>>> I gave br-ex an IP and UP'ed it manually.  I assume this is correct.
>>>>>> By I honestly don't know.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 15, 2013 at 6:54 PM, Greg Chavez <greg.chavez@xxxxxxxxx>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Sigh.  So I abandoned RHEL 6.3, rekicked my systems and set up the
>>>>>>> scale-ready installation described in these instructions:
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst
>>>>>>>
>>>>>>> Basically:
>>>>>>>
>>>>>>> (o) controller node on a mgmt and public net
>>>>>>> (o) network node (quantum and openvs) on a mgmt, net-config, and
>>>>>>> public net
>>>>>>> (o) compute node is on a mgmt and net-config net
>>>>>>>
>>>>>>> Took me just over an hour and ran into only a few easily-fixed speed
>>>>>>> bumps.  But the VM networks are totally non-functioning.  VMs launch but no
>>>>>>> network traffic can go in or out.
>>>>>>>
>>>>>>> I'm particularly befuddled by these problems:
>>>>>>>
>>>>>>> ( 1 ) This error in nova-compute:
>>>>>>>
>>>>>>> ERROR nova.network.quantumv2 [-] _get_auth_token() failed
>>>>>>>
>>>>>>> ( 2 ) No NAT rules on the compute node, which probably explains why
>>>>>>> the VMs complain about not finding a network or being able to get metadata
>>>>>>> from 169.254.169.254.
>>>>>>>
>>>>>>> root@kvm-cs-sn-10i:~# iptables -t nat -S
>>>>>>> -P PREROUTING ACCEPT
>>>>>>> -P INPUT ACCEPT
>>>>>>> -P OUTPUT ACCEPT
>>>>>>> -P POSTROUTING ACCEPT
>>>>>>> -N nova-api-metadat-OUTPUT
>>>>>>> -N nova-api-metadat-POSTROUTING
>>>>>>> -N nova-api-metadat-PREROUTING
>>>>>>> -N nova-api-metadat-float-snat
>>>>>>> -N nova-api-metadat-snat
>>>>>>> -N nova-compute-OUTPUT
>>>>>>> -N nova-compute-POSTROUTING
>>>>>>> -N nova-compute-PREROUTING
>>>>>>> -N nova-compute-float-snat
>>>>>>> -N nova-compute-snat
>>>>>>> -N nova-postrouting-bottom
>>>>>>> -A PREROUTING -j nova-api-metadat-PREROUTING
>>>>>>> -A PREROUTING -j nova-compute-PREROUTING
>>>>>>> -A OUTPUT -j nova-api-metadat-OUTPUT
>>>>>>> -A OUTPUT -j nova-compute-OUTPUT
>>>>>>> -A POSTROUTING -j nova-api-metadat-POSTROUTING
>>>>>>> -A POSTROUTING -j nova-compute-POSTROUTING
>>>>>>> -A POSTROUTING -j nova-postrouting-bottom
>>>>>>> -A nova-api-metadat-snat -j nova-api-metadat-float-snat
>>>>>>> -A nova-compute-snat -j nova-compute-float-snat
>>>>>>> -A nova-postrouting-bottom -j nova-api-metadat-snat
>>>>>>> -A nova-postrouting-bottom -j nova-compute-snat
>>>>>>>
>>>>>>> (3) A lastly, no default secgroup rules, whose function governs...
>>>>>>> what exactly?  Connections to the VM's public or private IPs?  I guess I'm
>>>>>>> just not sure if this is relevant to my overall problem of ZERO VM network
>>>>>>> connectivity.
>>>>>>>
>>>>>>> I seek guidance please.  Thanks.
>>>>>>>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>