yahoo-eng-team team mailing list archive

[Yuli <stremovsky@xxxxxxxxx>]

Public bug reported:

Hello

In our stress test, Neutron has performance degradation when
multiple subnets are added to the same router.

I am attaching script to reproduce this bug.

This script does the following:
1. creates a ROUTER-1
2. Create 10 networks titled: NETWORK-X
   2.1 Create 100 Subnets for each NETWORK-X and attach subnets to ROUTER-1
          using the neutron.add_interface_router()

Here is the result for the first 500 subnets:

Time spend to create 100 subnets: 281 seconds
Time spend to create additional 100 subnets: 594 seconds
Time spend to create additional 100 subnets: 912 seconds
Time spend to create additional 100 subnets: 1191 seconds
Time spend to create additional 100 subnets: 1464 seconds

-----------------
Total time to create 500 subnets is 74 minutes.
When it took only 10 minutes to create first 100 subnets and link to router.


To be able to run the test I had to disable quota and restart q-svc.
>From my neutron config file: /etc/neutron/neutron.conf

[quotas]
default_quota = -1
quota_network = -1
quota_subnet = -1
track_quota_usage = false
quota_router = -1
quota_floatingip = -1
quota_security_group = -1
quota_security_group_rule = -1

I am using regular all in one devstack installation using the l2/l3
agent.

Linux is Ubuntu 14.04.3 LTS

This is my local.config file:
-----------------

[[local|localrc]]
NETWORK_GATEWAY=10.0.0.1
PUBLIC_NETWORK_GATEWAY=10.100.100.8
Q_FLOATING_ALLOCATION_POOL=start=10.100.201.200,end=10.100.201.230
RABBIT_PASSWORD=pwd123
SERVICE_PASSWORD=pwd123
SERVICE_TOKEN=pwd123
ADMIN_PASSWORD=pwd123
DATABASE_PASSWORD=pwd123

disable_service n-net
disable_service tempest
enable_service heat
enable_service h-eng h-api h-api-cfn h-api-cw
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service q-svc

FIXED_NETWORK_SIZE=256
FIXED_RANGE=10.0.0.0/24
FLAT_INTERFACE=eno1
FLOATING_RANGE=10.100.0.0/16
HOST_IP=10.100.100.8

---------

Thanks and best regards,
Yuli

** Affects: neutron
     Importance: Undecided
         Status: New

** Attachment added: "neutron-stress-test.py"
   https://bugs.launchpad.net/bugs/1558101/+attachment/4601165/+files/neutron-stress-test.py

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1558101

Title:
  Neutron performance degradation when multiple subnets attached to same
  router

Status in neutron:
  New

Bug description:
  Hello

  In our stress test, Neutron has performance degradation when
  multiple subnets are added to the same router.

  I am attaching script to reproduce this bug.

  This script does the following:
  1. creates a ROUTER-1
  2. Create 10 networks titled: NETWORK-X
     2.1 Create 100 Subnets for each NETWORK-X and attach subnets to ROUTER-1
            using the neutron.add_interface_router()

  Here is the result for the first 500 subnets:

  Time spend to create 100 subnets: 281 seconds
  Time spend to create additional 100 subnets: 594 seconds
  Time spend to create additional 100 subnets: 912 seconds
  Time spend to create additional 100 subnets: 1191 seconds
  Time spend to create additional 100 subnets: 1464 seconds

  -----------------
  Total time to create 500 subnets is 74 minutes.
  When it took only 10 minutes to create first 100 subnets and link to router.

  
  To be able to run the test I had to disable quota and restart q-svc.
  From my neutron config file: /etc/neutron/neutron.conf

  [quotas]
  default_quota = -1
  quota_network = -1
  quota_subnet = -1
  track_quota_usage = false
  quota_router = -1
  quota_floatingip = -1
  quota_security_group = -1
  quota_security_group_rule = -1

  I am using regular all in one devstack installation using the l2/l3
  agent.

  Linux is Ubuntu 14.04.3 LTS

  This is my local.config file:
  -----------------

  [[local|localrc]]
  NETWORK_GATEWAY=10.0.0.1
  PUBLIC_NETWORK_GATEWAY=10.100.100.8
  Q_FLOATING_ALLOCATION_POOL=start=10.100.201.200,end=10.100.201.230
  RABBIT_PASSWORD=pwd123
  SERVICE_PASSWORD=pwd123
  SERVICE_TOKEN=pwd123
  ADMIN_PASSWORD=pwd123
  DATABASE_PASSWORD=pwd123

  disable_service n-net
  disable_service tempest
  enable_service heat
  enable_service h-eng h-api h-api-cfn h-api-cw
  enable_service q-agt
  enable_service q-dhcp
  enable_service q-l3
  enable_service q-meta
  enable_service q-svc

  FIXED_NETWORK_SIZE=256
  FIXED_RANGE=10.0.0.0/24
  FLAT_INTERFACE=eno1
  FLOATING_RANGE=10.100.0.0/16
  HOST_IP=10.100.100.8

  ---------

  Thanks and best regards,
  Yuli

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1558101/+subscriptions