yahoo-eng-team team mailing list archive

[IBM-Cloud-SH <IBM-Cloud-SH@xxxxxxxxxxxxxxx>]

Public bug reported:

[Summary]
the container can not be pinged via name space, after 860 tenants/networks/container created

[Topo]
1 controller, 2 network nodes, 6 compute nodes, all in ubuntu 14.04
(openstack version is 2015.1.2, linux kernel version is 3.19.0-31)
root@ah:~# uname -a
Linux ah.container13 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@ah:~# 
root@ah:~# dpkg -l | grep neutron
ii  neutron-common                      1:2015.1.2-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - common
ii  neutron-plugin-ml2                  1:2015.1.2-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - ML2 plugin
ii  neutron-server                      1:2015.1.2-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - server
ii  python-neutron                      1:2015.1.2-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - Python 

library
ii  python-neutron-fwaas                2015.1.2-0ubuntu2~cloud0              all          Firewall-as-a-Service driver for OpenStack Neutron
ii  python-neutronclient                1:2.3.11-0ubuntu1.2~cloud0            all          client - Neutron is a virtual network service for Openstack
root@ah:~#

[Description and expect result]
the container should be pinged via name space

[Reproduceable or not]
reproducible intermittently when large number of tenant/network/instance configed

[Recreate Steps]
1)use script to create: 860 tenants, 1 network/router in each tenant, 1 cirros container in each network, all containers are associate to FIP

2)create one more tenant, 1 network/contaier in the tenant, the
container can be in Active state, but can not be pinged via name space
>>>>ISSUE


[Configration]
config files on controller/network/compute are attached

[logs]
instance can be in Active state:
oot@ah:~# nova --os-tenant-id 73731bbaf2db48f89a067604e3556e05 list
+--------------------------------------+-----------------------------+--------+------------+-------------+----------------------------------------------------+
| ID                                   | Name                        | Status | Task State | Power State | Networks                                           |
+--------------------------------------+-----------------------------+--------+------------+-------------+----------------------------------------------------+
| d5ba18d5-aaf9-4ed6-9a2b-71d2b2f10bae | mexico_test_new_2_1_net1_vm | ACTIVE | -          | Running     | mexico_test_new_2_1_net1=10.10.32.3, 172.168.6.211 |
+--------------------------------------+-----------------------------+--------+------------+-------------+----------------------------------------------------+
root@ah:~# keystone tenant-list | grep test_new_2_1
| 73731bbaf2db48f89a067604e3556e05 | mexico_test_new_2_1 |   True  |
root@ah:~# neutron net-list | grep exico_test_new_2_1_net1
| a935642d-b56c-4a87-83c5-755f01bf0814 | mexico_test_new_2_1_net1 | bed0330f-e0ea-4bcc-bc75-96766dad32a7 10.10.32.0/24  |
root@ah:~#

on network node:
root@ah:~# ip netns | grep a935642d-b56c-4a87-83c5-755f01bf0814
qdhcp-a935642d-b56c-4a87-83c5-755f01bf0814
root@ah:~# ip netns exec qdhcp-a935642d-b56c-4a87-83c5-755f01bf0814 ping 10.10.32.3
PING 10.10.32.3 (10.10.32.3) 56(84) bytes of data.
>From 10.10.32.2 icmp_seq=1 Destination Host Unreachable
>From 10.10.32.2 icmp_seq=2 Destination Host Unreachable
>From 10.10.32.2 icmp_seq=3 Destination Host Unreachable
>From 10.10.32.2 icmp_seq=4 Destination Host Unreachable    >>>ISSUE

[Root cause anlyze or debug inf]
high load on controller and network node

[Attachment]
log files on controller/network/compute are attached

** Affects: neutron
     Importance: Undecided
         Status: New

** Attachment added: "log files and config files on controller/network/compute are attached"
   https://bugs.launchpad.net/bugs/1513048/+attachment/4512713/+files/log_config.rar

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1513048

Title:
  the container can not be pinged via name space, after 860
  tenants/networks/container created

Status in neutron:
  New

Bug description:
  [Summary]
  the container can not be pinged via name space, after 860 tenants/networks/container created

  [Topo]
  1 controller, 2 network nodes, 6 compute nodes, all in ubuntu 14.04
  (openstack version is 2015.1.2, linux kernel version is 3.19.0-31)
  root@ah:~# uname -a
  Linux ah.container13 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
  root@ah:~# 
  root@ah:~# dpkg -l | grep neutron
  ii  neutron-common                      1:2015.1.2-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - common
  ii  neutron-plugin-ml2                  1:2015.1.2-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - ML2 plugin
  ii  neutron-server                      1:2015.1.2-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - server
  ii  python-neutron                      1:2015.1.2-0ubuntu1~cloud0            all          Neutron is a virtual network service for Openstack - Python 

  library
  ii  python-neutron-fwaas                2015.1.2-0ubuntu2~cloud0              all          Firewall-as-a-Service driver for OpenStack Neutron
  ii  python-neutronclient                1:2.3.11-0ubuntu1.2~cloud0            all          client - Neutron is a virtual network service for Openstack
  root@ah:~#

  [Description and expect result]
  the container should be pinged via name space

  [Reproduceable or not]
  reproducible intermittently when large number of tenant/network/instance configed

  [Recreate Steps]
  1)use script to create: 860 tenants, 1 network/router in each tenant, 1 cirros container in each network, all containers are associate to FIP

  2)create one more tenant, 1 network/contaier in the tenant, the
  container can be in Active state, but can not be pinged via name space
  >>>>ISSUE

  
  [Configration]
  config files on controller/network/compute are attached

  [logs]
  instance can be in Active state:
  oot@ah:~# nova --os-tenant-id 73731bbaf2db48f89a067604e3556e05 list
  +--------------------------------------+-----------------------------+--------+------------+-------------+----------------------------------------------------+
  | ID                                   | Name                        | Status | Task State | Power State | Networks                                           |
  +--------------------------------------+-----------------------------+--------+------------+-------------+----------------------------------------------------+
  | d5ba18d5-aaf9-4ed6-9a2b-71d2b2f10bae | mexico_test_new_2_1_net1_vm | ACTIVE | -          | Running     | mexico_test_new_2_1_net1=10.10.32.3, 172.168.6.211 |
  +--------------------------------------+-----------------------------+--------+------------+-------------+----------------------------------------------------+
  root@ah:~# keystone tenant-list | grep test_new_2_1
  | 73731bbaf2db48f89a067604e3556e05 | mexico_test_new_2_1 |   True  |
  root@ah:~# neutron net-list | grep exico_test_new_2_1_net1
  | a935642d-b56c-4a87-83c5-755f01bf0814 | mexico_test_new_2_1_net1 | bed0330f-e0ea-4bcc-bc75-96766dad32a7 10.10.32.0/24  |
  root@ah:~#

  on network node:
  root@ah:~# ip netns | grep a935642d-b56c-4a87-83c5-755f01bf0814
  qdhcp-a935642d-b56c-4a87-83c5-755f01bf0814
  root@ah:~# ip netns exec qdhcp-a935642d-b56c-4a87-83c5-755f01bf0814 ping 10.10.32.3
  PING 10.10.32.3 (10.10.32.3) 56(84) bytes of data.
  From 10.10.32.2 icmp_seq=1 Destination Host Unreachable
  From 10.10.32.2 icmp_seq=2 Destination Host Unreachable
  From 10.10.32.2 icmp_seq=3 Destination Host Unreachable
  From 10.10.32.2 icmp_seq=4 Destination Host Unreachable    >>>ISSUE

  [Root cause anlyze or debug inf]
  high load on controller and network node

  [Attachment]
  log files on controller/network/compute are attached

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1513048/+subscriptions