yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #07773
[Bug 1263338] Re: neutron (devstack) network connections to launched instances fail
** Changed in: neutron
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1263338
Title:
neutron (devstack) network connections to launched instances fail
Status in OpenStack Neutron (virtual network service):
Invalid
Bug description:
When running neutron on devstack, I run into the issue where I can
successfully launch an instance, but cannot connect to it, not even
from the host running devstack. Commands like 'ping' and 'ssh' appear
to get no response at all.
Steps to reproduce:
./stack.sh
export OS_USERNAME=admin
export OS_PASSWORD=password
export OS_TENANT_NAME=demo
export OS_AUTH_URL=http://192.168.126.142:5000/v2.0/
source /usr/local/src/devstack/openrc admin
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
wget
https://launchpadlibrarian.net/83303699/cirros-0.3.0-i386-disk.img
glance image-create --name=cirros-0.3.0-i386 --is-public=true
--container-format=bare --disk-format=qcow2 <
cirros-0.3.0-i386-disk.img
nova boot --flavor m1.nano --image cirros-0.3.0-i386 myvm
nova show myvm
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | localhost.localdomain |
| OS-EXT-SRV-ATTR:hypervisor_hostname | localhost.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2013-12-21T14:19:29.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2013-12-21T14:19:22Z |
| flavor | m1.nano (42) |
| hostId | d751bc233fcbbf6622883cbe4ccfee1b1b41107c693836dc0b3666b1 |
| id | 5ba5e118-47ba-4791-be59-309eb6405dff |
| image | cirros-0.3.0-i386 (2c9602d1-b3b4-436e-bd92-532b3b03e541) |
| key_name | None |
| metadata | {} |
| name | myvm |
| os-extended-volumes:volumes_attached | [] |
| private network | 10.0.0.3 |
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | 2c79cf76ab5f488388de486a586aa23f |
| updated | 2013-12-21T14:19:29Z |
| user_id | 4f3c34c88167426baa94e20b26ccac8b |
+--------------------------------------+----------------------------------------------------------+
ping 10.0.0.3
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
^C
--- 10.0.0.3 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7008ms
ssh 10.0.0.3
^C
nova secgroup-list
+--------------------------------------+---------+-------------+
| Id | Name | Description |
+--------------------------------------+---------+-------------+
| 9e35010a-33b5-4437-8580-384975ce75c5 | default | default |
+--------------------------------------+---------+-------------+
nova secgroup-list-rules default
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp | -1 | -1 | 0.0.0.0/0 | |
| | | | | default |
| tcp | 22 | 22 | 0.0.0.0/0 | |
| | | | | default |
+-------------+-----------+---------+-----------+--------------+
But this works :
sudo ip netns exec qdhcp-fc342f8f-0211-4e86-9a2f-ec64c719ba67 ping 10.0.0.3
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_seq=1 ttl=64 time=1.26 ms
64 bytes from 10.0.0.3: icmp_seq=2 ttl=64 time=0.408 ms
^C
--- 10.0.0.3 ping statistics ---
3 packets transmitted, 2 received, 33% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.408/0.836/1.264/0.428 ms
sudo ip netns exec qrouter-4d26556c-9b29-4162-ba91-ce704b771fa6 ping 10.0.0.3
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_seq=1 ttl=64 time=1.98 ms
64 bytes from 10.0.0.3: icmp_seq=2 ttl=64 time=0.441 ms
^C
--- 10.0.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.441/1.213/1.986/0.773 ms
Here is the rest of the configuration.
ifconfig -a
br-ex: flags=67<UP,BROADCAST,RUNNING> mtu 1500
inet 172.24.4.225 netmask 255.255.255.128 broadcast 0.0.0.0
ether a2:8f:9a:28:63:4f txqueuelen 0 (Ethernet)
RX packets 12 bytes 976 (976.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3 bytes 270 (270.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
br-int: flags=67<UP,BROADCAST,RUNNING> mtu 1500
inet6 fe80::80bb:edff:fe2d:adc6 prefixlen 64 scopeid 0x20<link>
ether be:88:dc:b0:e1:44 txqueuelen 0 (Ethernet)
RX packets 35 bytes 3258 (3.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 83757 bytes 73811498 (70.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 83757 bytes 73811498 (70.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ovs-system: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 0a:ea:20:c3:4a:d5 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
p3p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.126.142 netmask 255.255.255.0 broadcast 192.168.126.255
inet6 fe80::20c:29ff:fe6e:32be prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:6e:32:be txqueuelen 1000 (Ethernet)
RX packets 78553 bytes 71897578 (68.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 79025 bytes 10351867 (9.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 19 base 0x2000
tap256fc450-51: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::fc16:3eff:fefa:4513 prefixlen 64 scopeid 0x20<link>
ether fe:16:3e:fa:45:13 txqueuelen 500 (Ethernet)
RX packets 44 bytes 4285 (4.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 46 bytes 4887 (4.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.126.2 0.0.0.0 UG 0 0 0 p3p1
172.24.4.128 * 255.255.255.128 U 0 0 0 br-ex
192.168.126.0 * 255.255.255.0 U 1 0 0 p3p1
ip netns list
qrouter-3eeb65d2-307a-496b-aa90-035bdaa77c5f
qdhcp-1c0789cd-0148-44a5-9297-c3e5a16e1514
ovs-vsctl show
ace3db50-ed2e-44ec-81aa-f427cc26a394
Bridge br-ex
Port "qg-7a28824e-75"
Interface "qg-7a28824e-75"
type: internal
Port br-ex
Interface br-ex
type: internal
Bridge br-int
Port "tap256fc450-51"
tag: 1
Interface "tap256fc450-51"
Port "tap201dde18-1e"
tag: 1
Interface "tap201dde18-1e"
type: internal
Port br-int
Interface br-int
type: internal
Port "qr-74a237e6-76"
tag: 1
Interface "qr-74a237e6-76"
type: internal
ovs_version: "1.11.0"
sudo brctl show
bridge name bridge id STP enabled interfaces
arp -a
? (192.168.126.1) at 00:50:56:c0:00:08 [ether] on p3p1
? (192.168.126.2) at 00:50:56:e6:61:ac [ether] on p3p1
? (192.168.126.254) at 00:50:56:fc:bc:9a [ether] on p3p1
When I ping the instance (10.0.0.3) from the host I run openstack on
(localhost.localdomain, 192.168.126.142) I dont see any arp
request/reply packets on any of the interfaces:
sudo tcpdump -vv -i br-int arp
tcpdump: WARNING: br-int: no IPv4 address assigned
tcpdump: listening on br-int, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
sudo tcpdump -vv -i br-ex arp
tcpdump: listening on br-ex, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
sudo tcpdump -vv -i tap256fc450-51 arp
tcpdump: WARNING: tap256fc450-51: no IPv4 address assigned
tcpdump: listening on tap256fc450-51, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
And this is my local.conf
cat local.conf
[[local|localrc]]
ADMIN_PASSWORD=password
MYSQL_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=tokentoken
RECLONE=yes
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen
LOGDAYS=1
#VERBOSE=True
HOST_IP_IFACE=p3p1
PUBLIC_INTERFACE=p3p1
VLAN_INTERFACE=p3p1
FLAT_INTERFACE=p3p1
HOST_IP=192.168.126.142
FIXED_RANGE=10.0.0.0/24
FIXED_NETWORK_SIZE=254
FLOATING_RANGE=192.168.42.128/25
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service neutron
enable_service q-lbaas
[[post-config|$NOVA_CONF]]
[DEFAULT]
debug = False
And the firewall rules
# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
neutron-openvswi-INPUT all -- anywhere anywhere
nova-api-INPUT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
INPUT_direct all -- anywhere anywhere
INPUT_ZONES_SOURCE all -- anywhere anywhere
INPUT_ZONES all -- anywhere anywhere
ACCEPT icmp -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-openvswi-FORWARD all -- anywhere anywhere
nova-filter-top all -- anywhere anywhere
nova-api-FORWARD all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
FORWARD_direct all -- anywhere anywhere
FORWARD_IN_ZONES_SOURCE all -- anywhere anywhere
FORWARD_IN_ZONES all -- anywhere anywhere
FORWARD_OUT_ZONES_SOURCE all -- anywhere anywhere
FORWARD_OUT_ZONES all -- anywhere anywhere
ACCEPT icmp -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- anywhere anywhere
neutron-openvswi-OUTPUT all -- anywhere anywhere
nova-filter-top all -- anywhere anywhere
nova-api-OUTPUT all -- anywhere anywhere
OUTPUT_direct all -- anywhere anywhere
Chain FORWARD_IN_ZONES (1 references)
target prot opt source destination
FWDI_public all -- anywhere anywhere
FWDI_public all -- anywhere anywhere
Chain FORWARD_IN_ZONES_SOURCE (1 references)
target prot opt source destination
Chain FORWARD_OUT_ZONES (1 references)
target prot opt source destination
FWDO_public all -- anywhere anywhere
FWDO_public all -- anywhere anywhere
Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target prot opt source destination
Chain FORWARD_direct (1 references)
target prot opt source destination
Chain FWDI_public (2 references)
target prot opt source destination
FWDI_public_log all -- anywhere anywhere
FWDI_public_deny all -- anywhere anywhere
FWDI_public_allow all -- anywhere anywhere
Chain FWDI_public_allow (1 references)
target prot opt source destination
Chain FWDI_public_deny (1 references)
target prot opt source destination
Chain FWDI_public_log (1 references)
target prot opt source destination
Chain FWDO_external (0 references)
target prot opt source destination
FWDO_external_log all -- anywhere anywhere
FWDO_external_deny all -- anywhere anywhere
FWDO_external_allow all -- anywhere anywhere
Chain FWDO_external_allow (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
Chain FWDO_external_deny (1 references)
target prot opt source destination
Chain FWDO_external_log (1 references)
target prot opt source destination
Chain FWDO_public (2 references)
target prot opt source destination
FWDO_public_log all -- anywhere anywhere
FWDO_public_deny all -- anywhere anywhere
FWDO_public_allow all -- anywhere anywhere
Chain FWDO_public_allow (1 references)
target prot opt source destination
Chain FWDO_public_deny (1 references)
target prot opt source destination
Chain FWDO_public_log (1 references)
target prot opt source destination
Chain INPUT_ZONES (1 references)
target prot opt source destination
IN_public all -- anywhere anywhere
IN_public all -- anywhere anywhere
Chain INPUT_ZONES_SOURCE (1 references)
target prot opt source destination
Chain INPUT_direct (1 references)
target prot opt source destination
Chain IN_dmz (0 references)
target prot opt source destination
IN_dmz_log all -- anywhere anywhere
IN_dmz_deny all -- anywhere anywhere
IN_dmz_allow all -- anywhere anywhere
Chain IN_dmz_allow (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp ctstate NEW
ACCEPT udp -- anywhere anywhere udp ctstate NEW
Chain IN_dmz_deny (1 references)
target prot opt source destination
Chain IN_dmz_log (1 references)
target prot opt source destination
Chain IN_external (0 references)
target prot opt source destination
IN_external_log all -- anywhere anywhere
IN_external_deny all -- anywhere anywhere
IN_external_allow all -- anywhere anywhere
Chain IN_external_allow (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp ctstate NEW
ACCEPT udp -- anywhere anywhere udp ctstate NEW
Chain IN_external_deny (1 references)
target prot opt source destination
Chain IN_external_log (1 references)
target prot opt source destination
Chain IN_home (0 references)
target prot opt source destination
IN_home_log all -- anywhere anywhere
IN_home_deny all -- anywhere anywhere
IN_home_allow all -- anywhere anywhere
Chain IN_home_allow (1 references)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:ipp ctstate NEW
ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns ctstate NEW
ACCEPT udp -- anywhere anywhere udp dpt:netbios-ns ctstate NEW
ACCEPT udp -- anywhere anywhere udp dpt:netbios-dgm ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp ctstate NEW
ACCEPT udp -- anywhere anywhere udp ctstate NEW
Chain IN_home_deny (1 references)
target prot opt source destination
Chain IN_home_log (1 references)
target prot opt source destination
Chain IN_internal (0 references)
target prot opt source destination
IN_internal_log all -- anywhere anywhere
IN_internal_deny all -- anywhere anywhere
IN_internal_allow all -- anywhere anywhere
Chain IN_internal_allow (1 references)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:ipp ctstate NEW
ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns ctstate NEW
ACCEPT udp -- anywhere anywhere udp dpt:netbios-ns ctstate NEW
ACCEPT udp -- anywhere anywhere udp dpt:netbios-dgm ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp ctstate NEW
ACCEPT udp -- anywhere anywhere udp ctstate NEW
Chain IN_internal_deny (1 references)
target prot opt source destination
Chain IN_internal_log (1 references)
target prot opt source destination
Chain IN_public (2 references)
target prot opt source destination
IN_public_log all -- anywhere anywhere
IN_public_deny all -- anywhere anywhere
IN_public_allow all -- anywhere anywhere
Chain IN_public_allow (1 references)
target prot opt source destination
ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp ctstate NEW
ACCEPT udp -- anywhere anywhere udp ctstate NEW
Chain IN_public_deny (1 references)
target prot opt source destination
Chain IN_public_log (1 references)
target prot opt source destination
Chain IN_work (0 references)
target prot opt source destination
IN_work_log all -- anywhere anywhere
IN_work_deny all -- anywhere anywhere
IN_work_allow all -- anywhere anywhere
Chain IN_work_allow (1 references)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:ipp ctstate NEW
ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate NEW
ACCEPT tcp -- anywhere anywhere tcp ctstate NEW
ACCEPT udp -- anywhere anywhere udp ctstate NEW
Chain IN_work_deny (1 references)
target prot opt source destination
Chain IN_work_log (1 references)
target prot opt source destination
Chain OUTPUT_direct (1 references)
target prot opt source destination
Chain neutron-filter-top (2 references)
target prot opt source destination
neutron-openvswi-local all -- anywhere anywhere
Chain neutron-openvswi-FORWARD (1 references)
target prot opt source destination
neutron-openvswi-sg-chain all -- anywhere anywhere PHYSDEV match --physdev-out tap256fc450-51 --physdev-is-bridged
neutron-openvswi-sg-chain all -- anywhere anywhere PHYSDEV match --physdev-in tap256fc450-51 --physdev-is-bridged
Chain neutron-openvswi-INPUT (1 references)
target prot opt source destination
neutron-openvswi-o256fc450-5 all -- anywhere anywhere PHYSDEV match --physdev-in tap256fc450-51 --physdev-is-bridged
Chain neutron-openvswi-OUTPUT (1 references)
target prot opt source destination
Chain neutron-openvswi-i256fc450-5 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere state INVALID
RETURN all -- anywhere anywhere state RELATED,ESTABLISHED
RETURN tcp -- anywhere anywhere tcp dpt:ssh
RETURN icmp -- anywhere anywhere
RETURN udp -- 10.0.0.2 anywhere udp spt:bootps dpt:bootpc
neutron-openvswi-sg-fallback all -- anywhere anywhere
Chain neutron-openvswi-local (1 references)
target prot opt source destination
Chain neutron-openvswi-o256fc450-5 (2 references)
target prot opt source destination
RETURN udp -- anywhere anywhere udp spt:bootpc dpt:bootps
neutron-openvswi-s256fc450-5 all -- anywhere anywhere
DROP udp -- anywhere anywhere udp spt:bootps dpt:bootpc
DROP all -- anywhere anywhere state INVALID
RETURN all -- anywhere anywhere state RELATED,ESTABLISHED
RETURN all -- anywhere anywhere
neutron-openvswi-sg-fallback all -- anywhere anywhere
Chain neutron-openvswi-s256fc450-5 (1 references)
target prot opt source destination
RETURN all -- 10.0.0.3 anywhere MAC FA:16:3E:FA:45:13
DROP all -- anywhere anywhere
Chain neutron-openvswi-sg-chain (2 references)
target prot opt source destination
neutron-openvswi-i256fc450-5 all -- anywhere anywhere PHYSDEV match --physdev-out tap256fc450-51 --physdev-is-bridged
neutron-openvswi-o256fc450-5 all -- anywhere anywhere PHYSDEV match --physdev-in tap256fc450-51 --physdev-is-bridged
ACCEPT all -- anywhere anywhere
Chain neutron-openvswi-sg-fallback (2 references)
target prot opt source destination
DROP all -- anywhere anywhere
Chain nova-api-FORWARD (1 references)
target prot opt source destination
Chain nova-api-INPUT (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere localhost.localdomain tcp dpt:8775
Chain nova-api-OUTPUT (1 references)
target prot opt source destination
Chain nova-api-local (1 references)
target prot opt source destination
Chain nova-filter-top (2 references)
target prot opt source destination
nova-api-local all -- anywhere anywhere
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1263338/+subscriptions
References
