yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2081087] [NEW] Performance regression in neutron-server from 2023.1 to 2024.1

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dmitriy Rabotyagov <2081087@xxxxxxxxxxxxxxxxxx>
Date: Wed, 18 Sep 2024 16:41:12 -0000
Reply-to: Bug 2081087 <2081087@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

With upgrade from 2023.1 to 2024.1 with driver ML2/OVS we've spotted a
significant (10 times) performance regression on some operations.

As best example - we can take security groups operations.

Neutron is running in eventlet, since uWSGI is not yet fully functional
for 2024.1 (see
https://review.opendev.org/c/openstack/neutron/+/926922).

So neutron-server is just being launched with exactly same database and
config, just from different venvs.

```
# cat /etc/systemd/system/neutron-server.service 
[Unit]
Description = neutron-server service
After = network-online.target
After = syslog.target

[Service]
Type = simple
User = neutron
Group = neutron
ExecStart = /openstack/venvs/neutron-29.0.2/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
ExecReload = /bin/kill -HUP $MAINPID
# Give a reasonable amount of time for the server to start up/shut down
TimeoutSec = 120
Restart = on-failure
RestartSec = 2
# This creates a specific slice which all services will operate from
#  The accounting options give us the ability to see resource usage through
#  the `systemd-cgtop` command.
Slice = neutron.slice
# Set Accounting
CPUAccounting = True
BlockIOAccounting = True
MemoryAccounting = True
TasksAccounting = True
# Set Sandboxing
PrivateTmp = False
PrivateDevices = False
PrivateNetwork = False
PrivateUsers = False

[Install]
WantedBy = multi-user.target

# time curl -X GET http://127.0.0.1:9696/v2.0/security-groups?project_id=${OS_PROJECT_ID} -H "X-Auth-Token: ${TOKEN}"
...
real    0m24.450s
user    0m0.008s
sys     0m0.010s
# time curl -X GET http://127.0.0.1:9696/v2.0/security-groups/${security_group_uuid} -H "X-Auth-Token: ${TOKEN}"
...
real    0m54.841s
user    0m0.010s
sys     0m0.012s
# sed -i 's/29.0.2/27.4.0/g' /etc/systemd/system/neutron-server.service
# systemctl daemon-reload
# systemctl restart neutron-server
# time curl -X GET http://127.0.0.1:9696/v2.0/security-groups?project_id=${OS_PROJECT_ID} -H "X-Auth-Token: ${TOKEN}"
...
real    0m1.040s
user    0m0.011s
sys     0m0.007s
# time curl -X GET http://127.0.0.1:9696/v2.0/security-groups/${security_group_uuid} -H "X-Auth-Token: ${TOKEN}"
...
real    0m0.589s
user    0m0.012s
sys     0m0.007s
```

So as you might see, difference in response time is very significant,
while the only change I've made is to use previous codebase for the
service.

I am also providing pip freeze for both venvs for comparison, though both of them were using upper-constraints:
# /openstack/venvs/neutron-27.4.0/bin/pip freeze
alembic==1.8.1
amqp==5.1.1
appdirs==1.4.4
attrs==22.1.0
autopage==0.5.1
bcrypt==4.0.0
cachetools==5.2.0
certifi==2023.11.17
cffi==1.15.1
charset-normalizer==2.1.1
cliff==4.2.0
cmd2==2.4.2
cryptography==38.0.2
debtcollector==2.5.0
decorator==5.1.1
dnspython==2.2.1
dogpile.cache==1.1.8
eventlet==0.33.1
extras==1.0.0
fasteners==0.18
fixtures==4.0.1
futurist==2.4.1
greenlet==1.1.3
httplib2==0.20.4
idna==3.4
importlib-metadata==5.0.0
iso8601==1.1.0
Jinja2==3.1.2
jmespath==1.0.1
jsonpatch==1.32
jsonpointer==2.3
jsonschema==4.16.0
keystoneauth1==5.1.2
keystonemiddleware==10.2.0
kombu==5.2.4
logutils==0.3.5
lxml==4.9.1
Mako==1.2.3
MarkupSafe==2.1.1
msgpack==1.0.4
ncclient==0.6.13
netaddr==0.8.0
netifaces==0.11.0
neutron==22.1.1.dev52
neutron-dynamic-routing==22.0.0
neutron-lib==3.4.1
neutron-vpnaas==22.0.1.dev2
openstacksdk==1.0.1
os-client-config==2.1.0
os-ken==2.6.0
os-resource-classes==1.1.0
os-service-types==1.7.0
os-traits==2.10.0
os-vif==3.1.1
osc-lib==2.7.0
oslo.cache==3.3.1
oslo.concurrency==5.1.1
oslo.config==9.1.1
oslo.context==5.6.0
oslo.db==12.3.1
oslo.i18n==6.0.0
oslo.log==5.2.0
oslo.messaging==14.7.1
oslo.metrics==0.6.0
oslo.middleware==5.1.1
oslo.policy==4.1.1
oslo.privsep==3.1.0
oslo.reports==3.0.0
oslo.rootwrap==7.0.1
oslo.serialization==5.1.1
oslo.service==3.1.1
oslo.upgradecheck==2.1.1
oslo.utils==6.1.0
oslo.versionedobjects==3.1.0
osprofiler==3.4.3
ovs==2.17.1.post1
ovsdbapp==2.2.1
packaging==21.3
paramiko==2.11.0
Paste==3.5.2
PasteDeploy==2.1.1
pbr==5.11.1
pecan==1.4.2
prettytable==3.4.1
prometheus-client==0.14.1
psutil==5.9.2
pycadf==3.1.1
pycparser==2.21
pyinotify==0.9.6
pymemcache==3.5.2
PyMySQL==1.0.2
PyNaCl==1.5.0
pyOpenSSL==22.1.0
pyparsing==3.0.9
pyperclip==1.8.2
pyroute2==0.7.3
pyrsistent==0.18.1
python-binary-memcached==0.31.1
python-dateutil==2.8.2
python-designateclient==5.2.0
python-keystoneclient==5.1.0
python-memcached==1.59
python-neutronclient==9.0.0
python-novaclient==18.3.0
pytz==2022.4
PyYAML==6.0
repoze.lru==0.7
requests==2.28.1
requestsexceptions==1.4.0
rfc3986==1.5.0
Routes==2.5.1
setproctitle==1.3.2
simplejson==3.17.6
six==1.16.0
sortedcontainers==2.4.0
SQLAlchemy==1.4.41
sqlalchemy-migrate==0.13.0
sqlparse==0.4.3
statsd==3.3.0
stevedore==5.0.0
systemd-python==234
Tempita==0.5.2
tenacity==6.3.1
testresources==2.0.1
testscenarios==0.5.0
testtools==2.5.0
tooz==3.2.0
uhashring==2.1
urllib3==1.26.12
vine==5.0.0
voluptuous==0.13.1
wcwidth==0.2.5
WebOb==1.8.7
wrapt==1.14.1
yappi==1.4.0
zipp==3.8.1

# /openstack/venvs/neutron-29.0.2/bin/pip freeze
alembic==1.9.4
amqp==5.2.0
async-timeout==4.0.3
attrs==23.2.0
autopage==0.5.2
bcrypt==4.1.2
cachetools==5.3.2
certifi==2024.8.30
cffi==1.16.0
charset-normalizer==3.3.2
cliff==4.6.0
cmd2==2.4.3
cryptography==42.0.4
debtcollector==3.0.0
decorator==5.1.1
dnspython==2.6.1
dogpile.cache==1.3.1
eventlet==0.36.1
fasteners==0.19
fixtures==4.1.0
futurist==3.0.0
greenlet==3.0.3
httplib2==0.22.0
idna==3.6
iso8601==2.1.0
Jinja2==3.1.3
jmespath==1.0.1
jsonpatch==1.33
jsonpointer==2.4
jsonschema==4.19.2
jsonschema-specifications==2023.12.1
keystoneauth1==5.6.0
keystonemiddleware==10.6.0
kombu==5.3.5
logutils==0.3.5
lxml==5.1.0
Mako==1.3.2
MarkupSafe==2.1.5
msgpack==1.0.7
ncclient==0.6.15
netaddr==0.10.1
netifaces==0.11.0
neutron==24.0.2.dev15
neutron-dynamic-routing==24.0.1.dev2
neutron-lib==3.11.0
neutron-vpnaas==24.0.2.dev1
openstacksdk==3.0.0
os-client-config==2.1.0
os-ken==2.8.1
os-resource-classes==1.1.0
os-service-types==1.7.0
os-traits==3.0.0
os-vif==3.5.0
osc-lib==3.0.1
oslo.cache==3.7.0
oslo.concurrency==6.0.0
oslo.config==9.4.0
oslo.context==5.5.0
oslo.db==15.0.0
oslo.i18n==6.3.0
oslo.log==5.5.1
oslo.messaging==14.7.1
oslo.metrics==0.8.0
oslo.middleware==6.1.0
oslo.policy==4.3.0
oslo.privsep==3.3.0
oslo.reports==3.3.0
oslo.rootwrap==7.2.0
oslo.serialization==5.4.0
oslo.service==3.4.1
oslo.upgradecheck==2.3.0
oslo.utils==7.1.0
oslo.versionedobjects==3.3.0
osprofiler==4.1.0
ovs==3.1.2
ovsdbapp==2.6.1
packaging==23.2
paramiko==3.4.0
Paste==3.7.1
PasteDeploy==3.1.0
pbr==6.0.0
pecan==1.5.1
platformdirs==4.2.0
prettytable==3.9.0
prometheus_client==0.20.0
psutil==5.9.8
pycadf==3.1.1
pycparser==2.21
pyinotify==0.9.6
PyJWT==2.8.0
pymemcache==4.0.0
PyMySQL==1.1.0
PyNaCl==1.5.0
pyOpenSSL==24.0.0
pyparsing==3.1.1
pyperclip==1.8.2
pyroute2==0.7.12
python-binary-memcached==0.31.2
python-dateutil==2.8.2
python-designateclient==6.0.1
python-keystoneclient==5.4.0
python-memcached==1.62
python-neutronclient==11.2.0
python-novaclient==18.6.0
pytz==2024.1
PyYAML==6.0.1
redis==5.0.1
referencing==0.33.0
repoze.lru==0.7
requests==2.31.0
requestsexceptions==1.4.0
rfc3986==2.0.0
Routes==2.5.1
rpds-py==0.18.0
setproctitle==1.3.3
simplejson==3.19.2
six==1.16.0
sortedcontainers==2.4.0
SQLAlchemy==1.4.51
statsd==4.0.1
stevedore==5.2.0
systemd-python==235
tenacity==8.2.3
testresources==2.0.1
testscenarios==0.5.0
testtools==2.7.1
tooz==6.2.0
typing_extensions==4.9.0
tzdata==2024.1
uhashring==2.3
urllib3==1.26.18
vine==5.1.0
voluptuous==0.14.2
wcwidth==0.2.13
WebOb==1.8.7
wrapt==1.16.0
yappi==1.6.0
#

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2081087

Title:
  Performance regression in neutron-server from 2023.1 to 2024.1

Status in neutron:
  New

Bug description:
  With upgrade from 2023.1 to 2024.1 with driver ML2/OVS we've spotted a
  significant (10 times) performance regression on some operations.

  As best example - we can take security groups operations.

  Neutron is running in eventlet, since uWSGI is not yet fully
  functional for 2024.1 (see
  https://review.opendev.org/c/openstack/neutron/+/926922).

  So neutron-server is just being launched with exactly same database
  and config, just from different venvs.

  ```
  # cat /etc/systemd/system/neutron-server.service 
  [Unit]
  Description = neutron-server service
  After = network-online.target
  After = syslog.target

  [Service]
  Type = simple
  User = neutron
  Group = neutron
  ExecStart = /openstack/venvs/neutron-29.0.2/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
  ExecReload = /bin/kill -HUP $MAINPID
  # Give a reasonable amount of time for the server to start up/shut down
  TimeoutSec = 120
  Restart = on-failure
  RestartSec = 2
  # This creates a specific slice which all services will operate from
  #  The accounting options give us the ability to see resource usage through
  #  the `systemd-cgtop` command.
  Slice = neutron.slice
  # Set Accounting
  CPUAccounting = True
  BlockIOAccounting = True
  MemoryAccounting = True
  TasksAccounting = True
  # Set Sandboxing
  PrivateTmp = False
  PrivateDevices = False
  PrivateNetwork = False
  PrivateUsers = False

  [Install]
  WantedBy = multi-user.target

  # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups?project_id=${OS_PROJECT_ID} -H "X-Auth-Token: ${TOKEN}"
  ...
  real    0m24.450s
  user    0m0.008s
  sys     0m0.010s
  # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups/${security_group_uuid} -H "X-Auth-Token: ${TOKEN}"
  ...
  real    0m54.841s
  user    0m0.010s
  sys     0m0.012s
  # sed -i 's/29.0.2/27.4.0/g' /etc/systemd/system/neutron-server.service
  # systemctl daemon-reload
  # systemctl restart neutron-server
  # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups?project_id=${OS_PROJECT_ID} -H "X-Auth-Token: ${TOKEN}"
  ...
  real    0m1.040s
  user    0m0.011s
  sys     0m0.007s
  # time curl -X GET http://127.0.0.1:9696/v2.0/security-groups/${security_group_uuid} -H "X-Auth-Token: ${TOKEN}"
  ...
  real    0m0.589s
  user    0m0.012s
  sys     0m0.007s
  ```

  So as you might see, difference in response time is very significant,
  while the only change I've made is to use previous codebase for the
  service.

  I am also providing pip freeze for both venvs for comparison, though both of them were using upper-constraints:
  # /openstack/venvs/neutron-27.4.0/bin/pip freeze
  alembic==1.8.1
  amqp==5.1.1
  appdirs==1.4.4
  attrs==22.1.0
  autopage==0.5.1
  bcrypt==4.0.0
  cachetools==5.2.0
  certifi==2023.11.17
  cffi==1.15.1
  charset-normalizer==2.1.1
  cliff==4.2.0
  cmd2==2.4.2
  cryptography==38.0.2
  debtcollector==2.5.0
  decorator==5.1.1
  dnspython==2.2.1
  dogpile.cache==1.1.8
  eventlet==0.33.1
  extras==1.0.0
  fasteners==0.18
  fixtures==4.0.1
  futurist==2.4.1
  greenlet==1.1.3
  httplib2==0.20.4
  idna==3.4
  importlib-metadata==5.0.0
  iso8601==1.1.0
  Jinja2==3.1.2
  jmespath==1.0.1
  jsonpatch==1.32
  jsonpointer==2.3
  jsonschema==4.16.0
  keystoneauth1==5.1.2
  keystonemiddleware==10.2.0
  kombu==5.2.4
  logutils==0.3.5
  lxml==4.9.1
  Mako==1.2.3
  MarkupSafe==2.1.1
  msgpack==1.0.4
  ncclient==0.6.13
  netaddr==0.8.0
  netifaces==0.11.0
  neutron==22.1.1.dev52
  neutron-dynamic-routing==22.0.0
  neutron-lib==3.4.1
  neutron-vpnaas==22.0.1.dev2
  openstacksdk==1.0.1
  os-client-config==2.1.0
  os-ken==2.6.0
  os-resource-classes==1.1.0
  os-service-types==1.7.0
  os-traits==2.10.0
  os-vif==3.1.1
  osc-lib==2.7.0
  oslo.cache==3.3.1
  oslo.concurrency==5.1.1
  oslo.config==9.1.1
  oslo.context==5.6.0
  oslo.db==12.3.1
  oslo.i18n==6.0.0
  oslo.log==5.2.0
  oslo.messaging==14.7.1
  oslo.metrics==0.6.0
  oslo.middleware==5.1.1
  oslo.policy==4.1.1
  oslo.privsep==3.1.0
  oslo.reports==3.0.0
  oslo.rootwrap==7.0.1
  oslo.serialization==5.1.1
  oslo.service==3.1.1
  oslo.upgradecheck==2.1.1
  oslo.utils==6.1.0
  oslo.versionedobjects==3.1.0
  osprofiler==3.4.3
  ovs==2.17.1.post1
  ovsdbapp==2.2.1
  packaging==21.3
  paramiko==2.11.0
  Paste==3.5.2
  PasteDeploy==2.1.1
  pbr==5.11.1
  pecan==1.4.2
  prettytable==3.4.1
  prometheus-client==0.14.1
  psutil==5.9.2
  pycadf==3.1.1
  pycparser==2.21
  pyinotify==0.9.6
  pymemcache==3.5.2
  PyMySQL==1.0.2
  PyNaCl==1.5.0
  pyOpenSSL==22.1.0
  pyparsing==3.0.9
  pyperclip==1.8.2
  pyroute2==0.7.3
  pyrsistent==0.18.1
  python-binary-memcached==0.31.1
  python-dateutil==2.8.2
  python-designateclient==5.2.0
  python-keystoneclient==5.1.0
  python-memcached==1.59
  python-neutronclient==9.0.0
  python-novaclient==18.3.0
  pytz==2022.4
  PyYAML==6.0
  repoze.lru==0.7
  requests==2.28.1
  requestsexceptions==1.4.0
  rfc3986==1.5.0
  Routes==2.5.1
  setproctitle==1.3.2
  simplejson==3.17.6
  six==1.16.0
  sortedcontainers==2.4.0
  SQLAlchemy==1.4.41
  sqlalchemy-migrate==0.13.0
  sqlparse==0.4.3
  statsd==3.3.0
  stevedore==5.0.0
  systemd-python==234
  Tempita==0.5.2
  tenacity==6.3.1
  testresources==2.0.1
  testscenarios==0.5.0
  testtools==2.5.0
  tooz==3.2.0
  uhashring==2.1
  urllib3==1.26.12
  vine==5.0.0
  voluptuous==0.13.1
  wcwidth==0.2.5
  WebOb==1.8.7
  wrapt==1.14.1
  yappi==1.4.0
  zipp==3.8.1

  # /openstack/venvs/neutron-29.0.2/bin/pip freeze
  alembic==1.9.4
  amqp==5.2.0
  async-timeout==4.0.3
  attrs==23.2.0
  autopage==0.5.2
  bcrypt==4.1.2
  cachetools==5.3.2
  certifi==2024.8.30
  cffi==1.16.0
  charset-normalizer==3.3.2
  cliff==4.6.0
  cmd2==2.4.3
  cryptography==42.0.4
  debtcollector==3.0.0
  decorator==5.1.1
  dnspython==2.6.1
  dogpile.cache==1.3.1
  eventlet==0.36.1
  fasteners==0.19
  fixtures==4.1.0
  futurist==3.0.0
  greenlet==3.0.3
  httplib2==0.22.0
  idna==3.6
  iso8601==2.1.0
  Jinja2==3.1.3
  jmespath==1.0.1
  jsonpatch==1.33
  jsonpointer==2.4
  jsonschema==4.19.2
  jsonschema-specifications==2023.12.1
  keystoneauth1==5.6.0
  keystonemiddleware==10.6.0
  kombu==5.3.5
  logutils==0.3.5
  lxml==5.1.0
  Mako==1.3.2
  MarkupSafe==2.1.5
  msgpack==1.0.7
  ncclient==0.6.15
  netaddr==0.10.1
  netifaces==0.11.0
  neutron==24.0.2.dev15
  neutron-dynamic-routing==24.0.1.dev2
  neutron-lib==3.11.0
  neutron-vpnaas==24.0.2.dev1
  openstacksdk==3.0.0
  os-client-config==2.1.0
  os-ken==2.8.1
  os-resource-classes==1.1.0
  os-service-types==1.7.0
  os-traits==3.0.0
  os-vif==3.5.0
  osc-lib==3.0.1
  oslo.cache==3.7.0
  oslo.concurrency==6.0.0
  oslo.config==9.4.0
  oslo.context==5.5.0
  oslo.db==15.0.0
  oslo.i18n==6.3.0
  oslo.log==5.5.1
  oslo.messaging==14.7.1
  oslo.metrics==0.8.0
  oslo.middleware==6.1.0
  oslo.policy==4.3.0
  oslo.privsep==3.3.0
  oslo.reports==3.3.0
  oslo.rootwrap==7.2.0
  oslo.serialization==5.4.0
  oslo.service==3.4.1
  oslo.upgradecheck==2.3.0
  oslo.utils==7.1.0
  oslo.versionedobjects==3.3.0
  osprofiler==4.1.0
  ovs==3.1.2
  ovsdbapp==2.6.1
  packaging==23.2
  paramiko==3.4.0
  Paste==3.7.1
  PasteDeploy==3.1.0
  pbr==6.0.0
  pecan==1.5.1
  platformdirs==4.2.0
  prettytable==3.9.0
  prometheus_client==0.20.0
  psutil==5.9.8
  pycadf==3.1.1
  pycparser==2.21
  pyinotify==0.9.6
  PyJWT==2.8.0
  pymemcache==4.0.0
  PyMySQL==1.1.0
  PyNaCl==1.5.0
  pyOpenSSL==24.0.0
  pyparsing==3.1.1
  pyperclip==1.8.2
  pyroute2==0.7.12
  python-binary-memcached==0.31.2
  python-dateutil==2.8.2
  python-designateclient==6.0.1
  python-keystoneclient==5.4.0
  python-memcached==1.62
  python-neutronclient==11.2.0
  python-novaclient==18.6.0
  pytz==2024.1
  PyYAML==6.0.1
  redis==5.0.1
  referencing==0.33.0
  repoze.lru==0.7
  requests==2.31.0
  requestsexceptions==1.4.0
  rfc3986==2.0.0
  Routes==2.5.1
  rpds-py==0.18.0
  setproctitle==1.3.3
  simplejson==3.19.2
  six==1.16.0
  sortedcontainers==2.4.0
  SQLAlchemy==1.4.51
  statsd==4.0.1
  stevedore==5.2.0
  systemd-python==235
  tenacity==8.2.3
  testresources==2.0.1
  testscenarios==0.5.0
  testtools==2.7.1
  tooz==6.2.0
  typing_extensions==4.9.0
  tzdata==2024.1
  uhashring==2.3
  urllib3==1.26.18
  vine==5.1.0
  voluptuous==0.14.2
  wcwidth==0.2.13
  WebOb==1.8.7
  wrapt==1.16.0
  yappi==1.6.0
  #

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2081087/+subscriptions
Follow ups

[Bug 2081087] Re: Performance regression in neutron-server from 2023.1 to 2024.1 when fetching a Security Group
From: OpenStack Infra, 2024-09-20