yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #07632
[Bug 1262785] [NEW] devstack-exercises floating_ips broken
Public bug reported:
Enabling the Q_USE_DEBUG_COMMAND=True in devstack localrc uses the set_neutron_debug using namespace fails. This usually shows as :
[Call Trace]
/opt/stack/new/devstack/exercises/volumes.sh:147:ping_check
/opt/stack/new/devstack/functions:1700:_ping_check_neutron
/opt/stack/new/devstack/lib/neutron:886:die
[ERROR] /opt/stack/new/devstack/lib/neutron:886 [Fail] Couldn't ping server
=====================================================================
SKIP boot_from_volume
SKIP client-env
SKIP marconi
SKIP savanna
SKIP trove
PASS aggregates
PASS bundle
PASS client-args
PASS euca
PASS horizon
PASS sec_groups
PASS swift
FAILED floating_ips
FAILED neutron-adv-test
FAILED volumes
=====================================================================
The env is running devstack in a local environment. With the followinjg localrc:
ubuntu@gate-t1:~/reddwarf/gate-t$ cat /opt/stack/new/devstack/localrc
Q_USE_DEBUG_COMMAND=True
NETWORK_GATEWAY=10.1.0.1
Q_USE_DEBUG_COMMAND=True
Q_PLUGIN=ml2
Q_AGENT=openvswitch
DEST=/opt/stack/new
ACTIVE_TIMEOUT=90
BOOT_TIMEOUT=90
ASSOCIATE_TIMEOUT=60
TERMINATE_TIMEOUT=60
MYSQL_PASSWORD=secret
RABBIT_PASSWORD=secret
ADMIN_PASSWORD=secret
SERVICE_PASSWORD=secret
SERVICE_TOKEN=111222333444
SWIFT_HASH=1234123412341234
ROOTSLEEP=0
ERROR_ON_CLONE=False
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-sch,horizon,mysql,rabbit,swift,cinder,c-api,c-vol,c-sch,n-cond,neutron,q-svc,q-agt,q-dhcp,q-l3,q-meta
SKIP_EXERCISES=boot_from_volume,client-env
SERVICE_HOST=127.0.0.1
SYSLOG=True
SCREEN_LOGDIR=/opt/stack/new/screen-logs
LOGFILE=/opt/stack/new/devstacklog.txt
VERBOSE=True
FIXED_RANGE=10.1.0.0/24
FIXED_NETWORK_SIZE=256
NETWORK_GATEWAY=10.1.0.1
VIRT_DRIVER=libvirt
SWIFT_REPLICAS=1
export OS_NO_CACHE=True
CINDER_SECURE_DELETE=False
API_RATE_LIMIT=False
VOLUME_BACKING_FILE_SIZE=5G
CINDER_SECURE_DELETE=False
This situation also cause the neutron Grenade(check-grenade-dsvm-neutron) testing to fail as well, see http://logs.openstack.org/63/61663/2/check/check-grenade-dsvm-neutron/5f49167/.
the failing command in floating_ips.sh :
check_command='while ! sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec qprobe-24ae41f0-4135-4c67-a16f-2eb5f4c313ec ping -w 1 -c 1 10.1.0.4; do sleep 1; done'
+ timeout 90 sh -c 'while ! sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec qprobe-24ae41f0-4135-4c67-a16f-2eb5f4c313ec ping -w 1 -c 1 10.1.0.4; do sleep 1; done'
PING 10.1.0.4 (10.1.0.4) 56(84) bytes of data.
--- 10.1.0.4 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING 10.1.0.4 (10.1.0.4) 56(84) bytes of data.
>From 10.1.0.2 icmp_seq=1 Destination Host Unreachable
--- 10.1.0.4 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
After the setup_neutron_debug (https://github.com/openstack-dev/devstack/blob/master/stack.sh#L1110) is called ( ttps://github.com/openstack-dev/devstack/blob/master/lib/neutron#L847), the ovs-vsctl show looks like:
ubuntu@gate-t1:~$ sudo ovs-vsctl show
c6a9fce5-7834-47cd-b92a-8d5a22ba5c87
Bridge br-ex
Port "qg-3ac19751-f0"
Interface "qg-3ac19751-f0"
type: internal
Port "tap2d117674-ab"
Interface "tap2d117674-ab"
type: internal
Port br-ex
Interface br-ex
type: internal
Bridge br-int
Port br-int
Interface br-int
type: internal
Port "tap4fe7e74b-0a"
tag: 1
Interface "tap4fe7e74b-0a"
Port "tap24ae41f0-41"
tag: 4095
Interface "tap24ae41f0-41"
type: internal
Port "qr-b835f1ef-38"
tag: 1
Interface "qr-b835f1ef-38"
type: internal
Port "tap23056976-35"
tag: 1
Interface "tap23056976-35"
type: internal
ovs_version: "1.4.3"
====== which the Port "tap24ae41f0-41" has a tag of 4095 which should
be 1, and the 10.1.0.4 ip address which should be private is pingable
from the host.
ubuntu@gate-t1:~$ neutron port-list
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| 23056976-352c-4563-9c21-5fd4d5226371 | | fa:16:3e:63:ca:eb | {"subnet_id": "0b9eda0b-fbec-482e-834a-2f2f5cbbf800", "ip_address": "10.1.0.3"} |
| 24ae41f0-4135-4c67-a16f-2eb5f4c313ec | | fa:16:3e:aa:b8:a6 | {"subnet_id": "0b9eda0b-fbec-482e-834a-2f2f5cbbf800", "ip_address": "10.1.0.2"} |
| 2d117674-abe5-4f74-a9e9-5ebc6206a510 | | fa:16:3e:78:49:ab | {"subnet_id": "94e8425c-b6f6-4dbe-970f-dcafd50a1324", "ip_address": "172.24.4.3"} |
| 3ac19751-f063-4bb3-88a0-61374c0b3548 | | fa:16:3e:1b:18:0b | {"subnet_id": "94e8425c-b6f6-4dbe-970f-dcafd50a1324", "ip_address": "172.24.4.2"} |
| 4fe7e74b-0a88-4550-a2ec-f499efda8cb3 | | fa:16:3e:b9:4c:bb | {"subnet_id": "0b9eda0b-fbec-482e-834a-2f2f5cbbf800", "ip_address": "10.1.0.4"} |
| 6d102fae-36b3-4421-b721-b6d231b790ee | | fa:16:3e:dc:48:75 | {"subnet_id": "e7642e63-72c9-44f9-8fc9-34c2f3f9b7b9", "ip_address": "10.10.0.2"} |
| 9af77d01-37fe-4fc2-80eb-1609577176f9 | | fa:16:3e:eb:24:0f | {"subnet_id": "e7642e63-72c9-44f9-8fc9-34c2f3f9b7b9", "ip_address": "10.10.0.4"} |
| 9cbb95d4-c36b-476c-a0ad-fcb21345c230 | | fa:16:3e:ee:93:8a | {"subnet_id": "e7642e63-72c9-44f9-8fc9-34c2f3f9b7b9", "ip_address": "10.10.0.3"} |
| b7189d72-07e1-4497-85e7-cb472fc0c429 | | fa:16:3e:8e:71:2d | {"subnet_id": "0b9eda0b-fbec-482e-834a-2f2f5cbbf800", "ip_address": "10.1.0.5"} |
| b835f1ef-387e-4f0e-aa47-725325e55033 | | fa:16:3e:65:fe:57 | {"subnet_id": "0b9eda0b-fbec-482e-834a-2f2f5cbbf800", "ip_address": "10.1.0.1"} |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
ubuntu@gate-t1:~$ neutron port-list -c id -c device_owner
+--------------------------------------+--------------------------+
| id | device_owner |
+--------------------------------------+--------------------------+
| 23056976-352c-4563-9c21-5fd4d5226371 | network:dhcp |
| 24ae41f0-4135-4c67-a16f-2eb5f4c313ec | compute:probe |
| 2d117674-abe5-4f74-a9e9-5ebc6206a510 | compute:probe |
| 3ac19751-f063-4bb3-88a0-61374c0b3548 | network:router_gateway |
| 4fe7e74b-0a88-4550-a2ec-f499efda8cb3 | compute:None |
| 6d102fae-36b3-4421-b721-b6d231b790ee | compute:probe |
| 9af77d01-37fe-4fc2-80eb-1609577176f9 | compute:None |
| 9cbb95d4-c36b-476c-a0ad-fcb21345c230 | network:dhcp |
| b7189d72-07e1-4497-85e7-cb472fc0c429 | compute:None |
| b835f1ef-387e-4f0e-aa47-725325e55033 | network:router_interface |
+--------------------------------------+--------------------------+
ubuntu@gate-t1:~$ ip netns list
qdhcp-a52e7df0-a4e8-4fa2-944c-bc4ad9a9da74
qprobe-6d102fae-36b3-4421-b721-b6d231b790ee
qrouter-920be1f4-87ce-4aba-b4d5-3948ed98fc78
qdhcp-1014c6c7-e061-4a8f-bc0a-b13e9c630dfe
qprobe-24ae41f0-4135-4c67-a16f-2eb5f4c313ec
qprobe-2d117674-abe5-4f74-a9e9-5ebc6206a510
ubuntu@gate-t1:~$
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1262785
Title:
devstack-exercises floating_ips broken
Status in OpenStack Neutron (virtual network service):
New
Bug description:
Enabling the Q_USE_DEBUG_COMMAND=True in devstack localrc uses the set_neutron_debug using namespace fails. This usually shows as :
[Call Trace]
/opt/stack/new/devstack/exercises/volumes.sh:147:ping_check
/opt/stack/new/devstack/functions:1700:_ping_check_neutron
/opt/stack/new/devstack/lib/neutron:886:die
[ERROR] /opt/stack/new/devstack/lib/neutron:886 [Fail] Couldn't ping server
=====================================================================
SKIP boot_from_volume
SKIP client-env
SKIP marconi
SKIP savanna
SKIP trove
PASS aggregates
PASS bundle
PASS client-args
PASS euca
PASS horizon
PASS sec_groups
PASS swift
FAILED floating_ips
FAILED neutron-adv-test
FAILED volumes
=====================================================================
The env is running devstack in a local environment. With the followinjg localrc:
ubuntu@gate-t1:~/reddwarf/gate-t$ cat /opt/stack/new/devstack/localrc
Q_USE_DEBUG_COMMAND=True
NETWORK_GATEWAY=10.1.0.1
Q_USE_DEBUG_COMMAND=True
Q_PLUGIN=ml2
Q_AGENT=openvswitch
DEST=/opt/stack/new
ACTIVE_TIMEOUT=90
BOOT_TIMEOUT=90
ASSOCIATE_TIMEOUT=60
TERMINATE_TIMEOUT=60
MYSQL_PASSWORD=secret
RABBIT_PASSWORD=secret
ADMIN_PASSWORD=secret
SERVICE_PASSWORD=secret
SERVICE_TOKEN=111222333444
SWIFT_HASH=1234123412341234
ROOTSLEEP=0
ERROR_ON_CLONE=False
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-sch,horizon,mysql,rabbit,swift,cinder,c-api,c-vol,c-sch,n-cond,neutron,q-svc,q-agt,q-dhcp,q-l3,q-meta
SKIP_EXERCISES=boot_from_volume,client-env
SERVICE_HOST=127.0.0.1
SYSLOG=True
SCREEN_LOGDIR=/opt/stack/new/screen-logs
LOGFILE=/opt/stack/new/devstacklog.txt
VERBOSE=True
FIXED_RANGE=10.1.0.0/24
FIXED_NETWORK_SIZE=256
NETWORK_GATEWAY=10.1.0.1
VIRT_DRIVER=libvirt
SWIFT_REPLICAS=1
export OS_NO_CACHE=True
CINDER_SECURE_DELETE=False
API_RATE_LIMIT=False
VOLUME_BACKING_FILE_SIZE=5G
CINDER_SECURE_DELETE=False
This situation also cause the neutron Grenade(check-grenade-dsvm-neutron) testing to fail as well, see http://logs.openstack.org/63/61663/2/check/check-grenade-dsvm-neutron/5f49167/.
the failing command in floating_ips.sh :
check_command='while ! sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec qprobe-24ae41f0-4135-4c67-a16f-2eb5f4c313ec ping -w 1 -c 1 10.1.0.4; do sleep 1; done'
+ timeout 90 sh -c 'while ! sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec qprobe-24ae41f0-4135-4c67-a16f-2eb5f4c313ec ping -w 1 -c 1 10.1.0.4; do sleep 1; done'
PING 10.1.0.4 (10.1.0.4) 56(84) bytes of data.
--- 10.1.0.4 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING 10.1.0.4 (10.1.0.4) 56(84) bytes of data.
From 10.1.0.2 icmp_seq=1 Destination Host Unreachable
--- 10.1.0.4 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
After the setup_neutron_debug (https://github.com/openstack-dev/devstack/blob/master/stack.sh#L1110) is called ( ttps://github.com/openstack-dev/devstack/blob/master/lib/neutron#L847), the ovs-vsctl show looks like:
ubuntu@gate-t1:~$ sudo ovs-vsctl show
c6a9fce5-7834-47cd-b92a-8d5a22ba5c87
Bridge br-ex
Port "qg-3ac19751-f0"
Interface "qg-3ac19751-f0"
type: internal
Port "tap2d117674-ab"
Interface "tap2d117674-ab"
type: internal
Port br-ex
Interface br-ex
type: internal
Bridge br-int
Port br-int
Interface br-int
type: internal
Port "tap4fe7e74b-0a"
tag: 1
Interface "tap4fe7e74b-0a"
Port "tap24ae41f0-41"
tag: 4095
Interface "tap24ae41f0-41"
type: internal
Port "qr-b835f1ef-38"
tag: 1
Interface "qr-b835f1ef-38"
type: internal
Port "tap23056976-35"
tag: 1
Interface "tap23056976-35"
type: internal
ovs_version: "1.4.3"
====== which the Port "tap24ae41f0-41" has a tag of 4095 which
should be 1, and the 10.1.0.4 ip address which should be private is
pingable from the host.
ubuntu@gate-t1:~$ neutron port-list
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| 23056976-352c-4563-9c21-5fd4d5226371 | | fa:16:3e:63:ca:eb | {"subnet_id": "0b9eda0b-fbec-482e-834a-2f2f5cbbf800", "ip_address": "10.1.0.3"} |
| 24ae41f0-4135-4c67-a16f-2eb5f4c313ec | | fa:16:3e:aa:b8:a6 | {"subnet_id": "0b9eda0b-fbec-482e-834a-2f2f5cbbf800", "ip_address": "10.1.0.2"} |
| 2d117674-abe5-4f74-a9e9-5ebc6206a510 | | fa:16:3e:78:49:ab | {"subnet_id": "94e8425c-b6f6-4dbe-970f-dcafd50a1324", "ip_address": "172.24.4.3"} |
| 3ac19751-f063-4bb3-88a0-61374c0b3548 | | fa:16:3e:1b:18:0b | {"subnet_id": "94e8425c-b6f6-4dbe-970f-dcafd50a1324", "ip_address": "172.24.4.2"} |
| 4fe7e74b-0a88-4550-a2ec-f499efda8cb3 | | fa:16:3e:b9:4c:bb | {"subnet_id": "0b9eda0b-fbec-482e-834a-2f2f5cbbf800", "ip_address": "10.1.0.4"} |
| 6d102fae-36b3-4421-b721-b6d231b790ee | | fa:16:3e:dc:48:75 | {"subnet_id": "e7642e63-72c9-44f9-8fc9-34c2f3f9b7b9", "ip_address": "10.10.0.2"} |
| 9af77d01-37fe-4fc2-80eb-1609577176f9 | | fa:16:3e:eb:24:0f | {"subnet_id": "e7642e63-72c9-44f9-8fc9-34c2f3f9b7b9", "ip_address": "10.10.0.4"} |
| 9cbb95d4-c36b-476c-a0ad-fcb21345c230 | | fa:16:3e:ee:93:8a | {"subnet_id": "e7642e63-72c9-44f9-8fc9-34c2f3f9b7b9", "ip_address": "10.10.0.3"} |
| b7189d72-07e1-4497-85e7-cb472fc0c429 | | fa:16:3e:8e:71:2d | {"subnet_id": "0b9eda0b-fbec-482e-834a-2f2f5cbbf800", "ip_address": "10.1.0.5"} |
| b835f1ef-387e-4f0e-aa47-725325e55033 | | fa:16:3e:65:fe:57 | {"subnet_id": "0b9eda0b-fbec-482e-834a-2f2f5cbbf800", "ip_address": "10.1.0.1"} |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
ubuntu@gate-t1:~$ neutron port-list -c id -c device_owner
+--------------------------------------+--------------------------+
| id | device_owner |
+--------------------------------------+--------------------------+
| 23056976-352c-4563-9c21-5fd4d5226371 | network:dhcp |
| 24ae41f0-4135-4c67-a16f-2eb5f4c313ec | compute:probe |
| 2d117674-abe5-4f74-a9e9-5ebc6206a510 | compute:probe |
| 3ac19751-f063-4bb3-88a0-61374c0b3548 | network:router_gateway |
| 4fe7e74b-0a88-4550-a2ec-f499efda8cb3 | compute:None |
| 6d102fae-36b3-4421-b721-b6d231b790ee | compute:probe |
| 9af77d01-37fe-4fc2-80eb-1609577176f9 | compute:None |
| 9cbb95d4-c36b-476c-a0ad-fcb21345c230 | network:dhcp |
| b7189d72-07e1-4497-85e7-cb472fc0c429 | compute:None |
| b835f1ef-387e-4f0e-aa47-725325e55033 | network:router_interface |
+--------------------------------------+--------------------------+
ubuntu@gate-t1:~$ ip netns list
qdhcp-a52e7df0-a4e8-4fa2-944c-bc4ad9a9da74
qprobe-6d102fae-36b3-4421-b721-b6d231b790ee
qrouter-920be1f4-87ce-4aba-b4d5-3948ed98fc78
qdhcp-1014c6c7-e061-4a8f-bc0a-b13e9c630dfe
qprobe-24ae41f0-4135-4c67-a16f-2eb5f4c313ec
qprobe-2d117674-abe5-4f74-a9e9-5ebc6206a510
ubuntu@gate-t1:~$
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1262785/+subscriptions
Follow ups
References
