openstack team mailing list archive

Thread
Date
Rebooted, now can't ping my guest

To: openstack@xxxxxxxxxxxxxxxxxxx
From: The King in Yellow <yellowking@xxxxxxxxx>
Date: Wed, 27 Feb 2013 18:22:52 -0500
I have been working on creating an OpenStack environment according to the Basic
Install <http://docs.openstack.org/folsom/basic-install/content/index.html>doc.
 It was working fine last night!  In order to make sure I didn't mess
anything up, I downed controller/network/compute nodes and cloned them
(they are nested on ESXi 5.0u1).

Upon coming back up, I can't ping my guests.  I'm on the network node,
pinging 10.5.5.3, which is a running guest.  I'm guessing the GRE tunnel
isn't coming between the compute and network node, since the br-*
interfaces down?  (After this, I manually "ip link set up" all br-*
interfaces on both compute and network-- nothing)

I have no experience with either Quantum or Open vSwitch, so I don't know
what this is telling me.  I'm rather at a loss-- can anybody point me in
the right direction here?  I don't see anything in the quantum logs right
now that seems to indicate an error-- openvswitch-agent.log is cycling
through things like the following, though:

2013-02-27 18:19:43    DEBUG [quantum.agent.linux.utils] Running command:
sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl
--timeout=2 get Interface qr-9f9041ce-65 external_ids
2013-02-27 18:19:43    DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap',
'/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get',
'Interface', 'qr-9f9041ce-65', 'external_ids']
Exit code: 0
Stdout: '{attached-mac="fa:16:3e:e2:38:da",
iface-id="9f9041ce-654d-4706-a208-60cf5fca5d90", iface-status=active}\n'
Stderr: ''
2013-02-27 18:19:43    DEBUG [quantum.agent.linux.utils] Running command:
sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl
--timeout=2 get Interface tap45ffdc5f-da external_ids
2013-02-27 18:19:43    DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap',
'/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get',
'Interface', 'tap45ffdc5f-da', 'external_ids']
Exit code: 0
Stdout: '{attached-mac="fa:16:3e:36:2e:54",
iface-id="45ffdc5f-dad9-444a-aff4-3d39b607f828", iface-status=active}\n'
Stderr: ''
2013-02-27 18:19:45    DEBUG [quantum.agent.linux.utils] Running command:
sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl
--timeout=2 list-ports br-int
2013-02-27 18:19:45    DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap',
'/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports',
'br-int']
Exit code: 0
Stdout: 'patch-tun\nqr-9f9041ce-65\ntap45ffdc5f-da\n'
Stderr: ''
2013-02-27 18:19:45    DEBUG [quantum.agent.linux.utils] Running command:
sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ovs-vsctl
--timeout=2 get Interface patch-tun external_ids
2013-02-27 18:19:45    DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap',
'/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get',
'Interface', 'patch-tun', 'external_ids']
Exit code: 0
Stdout: '{}\n'
Stderr: ''


Here are the output of ifconfig -a, ovs-vsctl show, and ovs-cfctl of each
bridge on the network node:

root@os-network:~# ifconfig -a
br-ex     Link encap:Ethernet  HWaddr 00:50:56:81:66:d8
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:23 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1380 (1.3 KB)  TX bytes:0 (0.0 B)

br-int    Link encap:Ethernet  HWaddr 5e:5a:c3:07:44:42
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

br-tun    Link encap:Ethernet  HWaddr 56:2d:9f:6c:ac:4f
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr 00:50:56:81:28:f4
          inet addr:192.168.0.2  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fe81:28f4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:535 errors:0 dropped:10 overruns:0 frame:0
          TX packets:554 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:137612 (137.6 KB)  TX bytes:108783 (108.7 KB)

eth1      Link encap:Ethernet  HWaddr 00:50:56:81:44:e7
          inet addr:10.10.10.1  Bcast:10.10.10.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fe81:44e7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:76 errors:0 dropped:9 overruns:0 frame:0
          TX packets:40 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:14531 (14.5 KB)  TX bytes:7529 (7.5 KB)

eth2      Link encap:Ethernet  HWaddr 00:50:56:81:66:d8
          inet6 addr: fe80::250:56ff:fe81:66d8/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:24 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1440 (1.4 KB)  TX bytes:3496 (3.4 KB)

eth3      Link encap:Ethernet  HWaddr 00:50:56:81:08:97
          inet addr:10.42.39.253  Bcast:10.42.39.255  Mask:255.255.254.0
          inet6 addr: fe80::250:56ff:fe81:897/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:500 errors:0 dropped:0 overruns:0 frame:0
          TX packets:530 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:52288 (52.2 KB)  TX bytes:59695 (59.6 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:231 errors:0 dropped:0 overruns:0 frame:0
          TX packets:231 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:22595 (22.5 KB)  TX bytes:22595 (22.5 KB)

qg-28108125-11 Link encap:Ethernet  HWaddr 6a:5e:96:0e:59:a0
          inet addr:10.42.36.130  Bcast:10.42.37.255  Mask:255.255.254.0
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:23 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1380 (1.3 KB)  TX bytes:0 (0.0 B)

qr-9f9041ce-65 Link encap:Ethernet  HWaddr 26:0e:04:69:99:1a
          inet addr:10.5.5.1  Bcast:10.5.5.255  Mask:255.255.255.0
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

tap45ffdc5f-da Link encap:Ethernet  HWaddr 0e:bd:f3:93:4b:b0
          inet addr:10.5.5.2  Bcast:10.5.5.255  Mask:255.255.255.0
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@os-network:~# ovs-vsctl show
e232f8c8-1cb8-4cf5-9de5-49f41e59fd38
    Bridge br-int
        Port "qr-9f9041ce-65"
            tag: 1
            Interface "qr-9f9041ce-65"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
        Port "tap45ffdc5f-da"
            tag: 1
            Interface "tap45ffdc5f-da"
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-28108125-11"
            Interface "qg-28108125-11"
                type: internal
        Port "eth2"
            Interface "eth2"
    Bridge br-tun
        Port "gre-2"
            Interface "gre-2"
                type: gre
                options: {in_key=flow, out_key=flow, remote_ip="10.10.10.2"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.4.0+build0"
root@os-network:~# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x1): ver:0x1, dpid:00005e5ac3074442
n_tables:255, n_buffers:256
features: capabilities:0xc7, actions:0xfff
 1(qr-9f9041ce-65): addr:26:0e:04:69:99:1a
     config:     PORT_DOWN
     state:      LINK_DOWN
 2(tap45ffdc5f-da): addr:0e:bd:f3:93:4b:b0
     config:     PORT_DOWN
     state:      LINK_DOWN
 3(patch-tun): addr:6a:b6:66:41:31:54
     config:     0
     state:      0
 LOCAL(br-int): addr:5e:5a:c3:07:44:42
     config:     PORT_DOWN
     state:      LINK_DOWN
OFPT_GET_CONFIG_REPLY (xid=0x3): frags=normal miss_send_len=0
root@os-network:~# ovs-ofctl show br-ex
OFPT_FEATURES_REPLY (xid=0x1): ver:0x1, dpid:00000050568166d8
n_tables:255, n_buffers:256
features: capabilities:0xc7, actions:0xfff
 1(eth2): addr:00:50:56:81:66:d8
     config:     0
     state:      0
     current:    10GB-FD COPPER
     advertised: COPPER
     supported:  1GB-FD 10GB-FD COPPER
 2(qg-28108125-11): addr:6a:5e:96:0e:59:a0
     config:     PORT_DOWN
     state:      LINK_DOWN
 LOCAL(br-ex): addr:00:50:56:81:66:d8
     config:     PORT_DOWN
     state:      LINK_DOWN
OFPT_GET_CONFIG_REPLY (xid=0x3): frags=normal miss_send_len=0
root@os-network:~# ovs-ofctl show br-tun
OFPT_FEATURES_REPLY (xid=0x1): ver:0x1, dpid:0000562d9f6cac4f
n_tables:255, n_buffers:256
features: capabilities:0xc7, actions:0xfff
 1(patch-int): addr:8e:c3:2b:55:e5:73
     config:     0
     state:      0
 2(gre-2): addr:5a:73:ff:6b:16:37
     config:     0
     state:      0
 LOCAL(br-tun): addr:56:2d:9f:6c:ac:4f
     config:     PORT_DOWN
     state:      LINK_DOWN
OFPT_GET_CONFIG_REPLY (xid=0x3): frags=normal miss_send_len=0
root@os-network:~#
Follow ups

Re: Rebooted, now can't ping my guest
From: Aaron Rosen, 2013-02-27