yahoo-eng-team team mailing list archive

[OpenStack Infra <2081087@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/929941
Committed: https://opendev.org/openstack/neutron/commit/c1b05e29adf9d0d68c1ac636013a8a363a92eb85
Submitter: "Zuul (22348)"
Branch:    master

commit c1b05e29adf9d0d68c1ac636013a8a363a92eb85
Author: Rodolfo Alonso Hernandez <ralonsoh@xxxxxxxxxx>
Date:   Thu Sep 19 14:00:57 2024 +0000

    Change the load method of SG rule "default_security_group"
    
    Since [1], the SG rule SQL view also retrieves the table
    "default_security_group", using a complex relationship [2].
    When the number of SG rules of a SG is high (above 50 it
    is clearly noticeable the performance degradation), the
    API call can take several seconds. For example, for 100
    SG rules it can take up to one minute.
    
    This patch changes the load method of the SG rule
    "default_security_group" relationship to "selectin".
    Benchmarks with a single default SG and 100 rules,
    doing "openstack security group show $sg":
    * 2023.2 (without this feature): around 0.05 seconds
    * master: between 45-50 seconds (1000x time increase)
    * loading method "selectin" or "dynamic": around 0.5 seconds.
    
    NOTE: this feature [1] was implemented in 2024.1. At this
    time, SQLAlchemy version was <2.0 and "selectin" method was
    not available. For this version, "dynamic" can be used instead.
    
    [1]https://review.opendev.org/q/topic:%22bug/2019960%22
    [2]https://github.com/openstack/neutron/blob/08fff4087dc342be40db179fca0cd9bbded91053/neutron/db/models/securitygroup.py#L120-L121
    
    Closes-Bug: #2081087
    Change-Id: I46af1179f6905307c0d60b5c0fdee264a40a4eac


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2081087

Title:
  Performance regression in neutron-server from 2023.1 to 2024.1 when
  fetching a Security Group

Status in neutron:
  Fix Released

Bug description:
  With upgrade from 2023.1 to 2024.1 with driver ML2/OVS we've spotted a
  significant (10 times) performance regression on some operations.

  As best example - we can take security groups operations.

  Neutron is running in eventlet, since uWSGI is not yet fully
  functional for 2024.1 (see
  https://review.opendev.org/c/openstack/neutron/+/926922).

  So neutron-server is just being launched with exactly same database
  and config, just from different venvs.

  ```
  # cat /etc/systemd/system/neutron-server.service 
  [Unit]
  Description = neutron-server service
  After = network-online.target
  After = syslog.target

  [Service]
  Type = simple
  User = neutron
  Group = neutron
  ExecStart = /openstack/venvs/neutron-29.0.2/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
  ExecReload = /bin/kill -HUP $MAINPID
  # Give a reasonable amount of time for the server to start up/shut down
  TimeoutSec = 120
  Restart = on-failure
  RestartSec = 2
  # This creates a specific slice which all services will operate from
  #  The accounting options give us the ability to see resource usage through
  #  the `systemd-cgtop` command.
  Slice = neutron.slice
  # Set Accounting
  CPUAccounting = True
  BlockIOAccounting = True
  MemoryAccounting = True
  TasksAccounting = True
  # Set Sandboxing
  PrivateTmp = False
  PrivateDevices = False
  PrivateNetwork = False
  PrivateUsers = False

  [Install]
  WantedBy = multi-user.target

  # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups?project_id=${OS_PROJECT_ID} -H "X-Auth-Token: ${TOKEN}"
  ...
  real    0m24.450s
  user    0m0.008s
  sys     0m0.010s
  # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups/${security_group_uuid} -H "X-Auth-Token: ${TOKEN}"
  ...
  real    0m54.841s
  user    0m0.010s
  sys     0m0.012s
  # sed -i 's/29.0.2/27.4.0/g' /etc/systemd/system/neutron-server.service
  # systemctl daemon-reload
  # systemctl restart neutron-server
  # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups?project_id=${OS_PROJECT_ID} -H "X-Auth-Token: ${TOKEN}"
  ...
  real    0m1.040s
  user    0m0.011s
  sys     0m0.007s
  # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups/${security_group_uuid} -H "X-Auth-Token: ${TOKEN}"
  ...
  real    0m0.589s
  user    0m0.012s
  sys     0m0.007s
  ```

  So as you might see, difference in response time is very significant,
  while the only change I've made is to use previous codebase for the
  service.

  I am also providing pip freeze for both venvs for comparison, though both of them were using upper-constraints:
  # /openstack/venvs/neutron-27.4.0/bin/pip freeze
  alembic==1.8.1
  amqp==5.1.1
  appdirs==1.4.4
  attrs==22.1.0
  autopage==0.5.1
  bcrypt==4.0.0
  cachetools==5.2.0
  certifi==2023.11.17
  cffi==1.15.1
  charset-normalizer==2.1.1
  cliff==4.2.0
  cmd2==2.4.2
  cryptography==38.0.2
  debtcollector==2.5.0
  decorator==5.1.1
  dnspython==2.2.1
  dogpile.cache==1.1.8
  eventlet==0.33.1
  extras==1.0.0
  fasteners==0.18
  fixtures==4.0.1
  futurist==2.4.1
  greenlet==1.1.3
  httplib2==0.20.4
  idna==3.4
  importlib-metadata==5.0.0
  iso8601==1.1.0
  Jinja2==3.1.2
  jmespath==1.0.1
  jsonpatch==1.32
  jsonpointer==2.3
  jsonschema==4.16.0
  keystoneauth1==5.1.2
  keystonemiddleware==10.2.0
  kombu==5.2.4
  logutils==0.3.5
  lxml==4.9.1
  Mako==1.2.3
  MarkupSafe==2.1.1
  msgpack==1.0.4
  ncclient==0.6.13
  netaddr==0.8.0
  netifaces==0.11.0
  neutron==22.1.1.dev52
  neutron-dynamic-routing==22.0.0
  neutron-lib==3.4.1
  neutron-vpnaas==22.0.1.dev2
  openstacksdk==1.0.1
  os-client-config==2.1.0
  os-ken==2.6.0
  os-resource-classes==1.1.0
  os-service-types==1.7.0
  os-traits==2.10.0
  os-vif==3.1.1
  osc-lib==2.7.0
  oslo.cache==3.3.1
  oslo.concurrency==5.1.1
  oslo.config==9.1.1
  oslo.context==5.6.0
  oslo.db==12.3.1
  oslo.i18n==6.0.0
  oslo.log==5.2.0
  oslo.messaging==14.7.1
  oslo.metrics==0.6.0
  oslo.middleware==5.1.1
  oslo.policy==4.1.1
  oslo.privsep==3.1.0
  oslo.reports==3.0.0
  oslo.rootwrap==7.0.1
  oslo.serialization==5.1.1
  oslo.service==3.1.1
  oslo.upgradecheck==2.1.1
  oslo.utils==6.1.0
  oslo.versionedobjects==3.1.0
  osprofiler==3.4.3
  ovs==2.17.1.post1
  ovsdbapp==2.2.1
  packaging==21.3
  paramiko==2.11.0
  Paste==3.5.2
  PasteDeploy==2.1.1
  pbr==5.11.1
  pecan==1.4.2
  prettytable==3.4.1
  prometheus-client==0.14.1
  psutil==5.9.2
  pycadf==3.1.1
  pycparser==2.21
  pyinotify==0.9.6
  pymemcache==3.5.2
  PyMySQL==1.0.2
  PyNaCl==1.5.0
  pyOpenSSL==22.1.0
  pyparsing==3.0.9
  pyperclip==1.8.2
  pyroute2==0.7.3
  pyrsistent==0.18.1
  python-binary-memcached==0.31.1
  python-dateutil==2.8.2
  python-designateclient==5.2.0
  python-keystoneclient==5.1.0
  python-memcached==1.59
  python-neutronclient==9.0.0
  python-novaclient==18.3.0
  pytz==2022.4
  PyYAML==6.0
  repoze.lru==0.7
  requests==2.28.1
  requestsexceptions==1.4.0
  rfc3986==1.5.0
  Routes==2.5.1
  setproctitle==1.3.2
  simplejson==3.17.6
  six==1.16.0
  sortedcontainers==2.4.0
  SQLAlchemy==1.4.41
  sqlalchemy-migrate==0.13.0
  sqlparse==0.4.3
  statsd==3.3.0
  stevedore==5.0.0
  systemd-python==234
  Tempita==0.5.2
  tenacity==6.3.1
  testresources==2.0.1
  testscenarios==0.5.0
  testtools==2.5.0
  tooz==3.2.0
  uhashring==2.1
  urllib3==1.26.12
  vine==5.0.0
  voluptuous==0.13.1
  wcwidth==0.2.5
  WebOb==1.8.7
  wrapt==1.14.1
  yappi==1.4.0
  zipp==3.8.1

  # /openstack/venvs/neutron-29.0.2/bin/pip freeze
  alembic==1.9.4
  amqp==5.2.0
  async-timeout==4.0.3
  attrs==23.2.0
  autopage==0.5.2
  bcrypt==4.1.2
  cachetools==5.3.2
  certifi==2024.8.30
  cffi==1.16.0
  charset-normalizer==3.3.2
  cliff==4.6.0
  cmd2==2.4.3
  cryptography==42.0.4
  debtcollector==3.0.0
  decorator==5.1.1
  dnspython==2.6.1
  dogpile.cache==1.3.1
  eventlet==0.36.1
  fasteners==0.19
  fixtures==4.1.0
  futurist==3.0.0
  greenlet==3.0.3
  httplib2==0.22.0
  idna==3.6
  iso8601==2.1.0
  Jinja2==3.1.3
  jmespath==1.0.1
  jsonpatch==1.33
  jsonpointer==2.4
  jsonschema==4.19.2
  jsonschema-specifications==2023.12.1
  keystoneauth1==5.6.0
  keystonemiddleware==10.6.0
  kombu==5.3.5
  logutils==0.3.5
  lxml==5.1.0
  Mako==1.3.2
  MarkupSafe==2.1.5
  msgpack==1.0.7
  ncclient==0.6.15
  netaddr==0.10.1
  netifaces==0.11.0
  neutron==24.0.2.dev15
  neutron-dynamic-routing==24.0.1.dev2
  neutron-lib==3.11.0
  neutron-vpnaas==24.0.2.dev1
  openstacksdk==3.0.0
  os-client-config==2.1.0
  os-ken==2.8.1
  os-resource-classes==1.1.0
  os-service-types==1.7.0
  os-traits==3.0.0
  os-vif==3.5.0
  osc-lib==3.0.1
  oslo.cache==3.7.0
  oslo.concurrency==6.0.0
  oslo.config==9.4.0
  oslo.context==5.5.0
  oslo.db==15.0.0
  oslo.i18n==6.3.0
  oslo.log==5.5.1
  oslo.messaging==14.7.1
  oslo.metrics==0.8.0
  oslo.middleware==6.1.0
  oslo.policy==4.3.0
  oslo.privsep==3.3.0
  oslo.reports==3.3.0
  oslo.rootwrap==7.2.0
  oslo.serialization==5.4.0
  oslo.service==3.4.1
  oslo.upgradecheck==2.3.0
  oslo.utils==7.1.0
  oslo.versionedobjects==3.3.0
  osprofiler==4.1.0
  ovs==3.1.2
  ovsdbapp==2.6.1
  packaging==23.2
  paramiko==3.4.0
  Paste==3.7.1
  PasteDeploy==3.1.0
  pbr==6.0.0
  pecan==1.5.1
  platformdirs==4.2.0
  prettytable==3.9.0
  prometheus_client==0.20.0
  psutil==5.9.8
  pycadf==3.1.1
  pycparser==2.21
  pyinotify==0.9.6
  PyJWT==2.8.0
  pymemcache==4.0.0
  PyMySQL==1.1.0
  PyNaCl==1.5.0
  pyOpenSSL==24.0.0
  pyparsing==3.1.1
  pyperclip==1.8.2
  pyroute2==0.7.12
  python-binary-memcached==0.31.2
  python-dateutil==2.8.2
  python-designateclient==6.0.1
  python-keystoneclient==5.4.0
  python-memcached==1.62
  python-neutronclient==11.2.0
  python-novaclient==18.6.0
  pytz==2024.1
  PyYAML==6.0.1
  redis==5.0.1
  referencing==0.33.0
  repoze.lru==0.7
  requests==2.31.0
  requestsexceptions==1.4.0
  rfc3986==2.0.0
  Routes==2.5.1
  rpds-py==0.18.0
  setproctitle==1.3.3
  simplejson==3.19.2
  six==1.16.0
  sortedcontainers==2.4.0
  SQLAlchemy==1.4.51
  statsd==4.0.1
  stevedore==5.2.0
  systemd-python==235
  tenacity==8.2.3
  testresources==2.0.1
  testscenarios==0.5.0
  testtools==2.7.1
  tooz==6.2.0
  typing_extensions==4.9.0
  tzdata==2024.1
  uhashring==2.3
  urllib3==1.26.18
  vine==5.1.0
  voluptuous==0.14.2
  wcwidth==0.2.13
  WebOb==1.8.7
  wrapt==1.16.0
  yappi==1.6.0
  #

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2081087/+subscriptions