yahoo-eng-team team mailing list archive

[David Moreau Simard <dms@xxxxxxxxxx>]

Public bug reported:

Relevant information TL;DR:
RDO stable Mitaka deployed with OpenStack puppet modules.
Keystone with fernet tokens, Nova with keystone_authtoken memcached caching set (also reproduced with sql tokens and nova without memcached caching for authtoken). nova-api is handled behind apache with mod wsgi.

Problem TL;DR:
In an empty new deployment on bare metal (no images, no virtual machines, etc.), "openstack server list" will intermittently take >5 seconds (up to ~10) to return. The only noticeable thing that happens when "openstack server list" is slow is that the logs will be plastered with a bunch of "_create_extension_schema", "_rebuild_extension_schema", "_register_resources" and so on.
All other installed projects (glance, keystone, neutron) return command output consistently under 2 seconds.

Version info:
# rpm -qa |grep nova
openstack-nova-api-13.0.0-1.el7.noarch
python-novaclient-3.3.0-1.el7.noarch
openstack-nova-scheduler-13.0.0-1.el7.noarch
python-nova-13.0.0-1.el7.noarch
openstack-nova-conductor-13.0.0-1.el7.noarch
openstack-nova-common-13.0.0-1.el7.noarch

/etc/nova/nova.conf: http://paste.fedoraproject.org/367251/34295211/raw/

Example log of when things are fast: http://paste.fedoraproject.org/367272/32783146/raw/
Example log of when things are slow: http://paste.fedoraproject.org/367271/34327341/raw/

Happy to help provider further details if required.

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  Relevant information TL;DR:
  RDO stable Mitaka deployed with OpenStack puppet modules.
  Keystone with fernet tokens, Nova with keystone_authtoken memcached caching set (also reproduced with sql tokens and nova without memcached caching for authtoken)
  
  Problem TL;DR:
  In an empty new deployment on bare metal (no images, no virtual machines, etc.), "openstack server list" will intermittently take >5 seconds (up to ~10) to return. The only noticeable thing that happens when "openstack server list" is slow is that the logs will be plastered with a bunch of "_create_extension_schema", "_rebuild_extension_schema", "_register_resources" and so on.
- All other installer projects (glance, keystone, neutron) return command output consistently under 2 seconds.
+ All other installed projects (glance, keystone, neutron) return command output consistently under 2 seconds.
  
  Version info:
  # rpm -qa |grep nova
  openstack-nova-api-13.0.0-1.el7.noarch
  python-novaclient-3.3.0-1.el7.noarch
  openstack-nova-scheduler-13.0.0-1.el7.noarch
  python-nova-13.0.0-1.el7.noarch
  openstack-nova-conductor-13.0.0-1.el7.noarch
  openstack-nova-common-13.0.0-1.el7.noarch
  
  /etc/nova/nova.conf: http://paste.fedoraproject.org/367251/34295211/raw/
  
  Example log of when things are fast: http://paste.fedoraproject.org/367272/32783146/raw/
  Example log of when things are slow: http://paste.fedoraproject.org/367271/34327341/raw/
  
  Happy to help provider further details if required.

** Description changed:

  Relevant information TL;DR:
  RDO stable Mitaka deployed with OpenStack puppet modules.
- Keystone with fernet tokens, Nova with keystone_authtoken memcached caching set (also reproduced with sql tokens and nova without memcached caching for authtoken)
+ Keystone with fernet tokens, Nova with keystone_authtoken memcached caching set (also reproduced with sql tokens and nova without memcached caching for authtoken). nova-api is handled behind apache with mod wsgi.
  
  Problem TL;DR:
  In an empty new deployment on bare metal (no images, no virtual machines, etc.), "openstack server list" will intermittently take >5 seconds (up to ~10) to return. The only noticeable thing that happens when "openstack server list" is slow is that the logs will be plastered with a bunch of "_create_extension_schema", "_rebuild_extension_schema", "_register_resources" and so on.
  All other installed projects (glance, keystone, neutron) return command output consistently under 2 seconds.
  
  Version info:
  # rpm -qa |grep nova
  openstack-nova-api-13.0.0-1.el7.noarch
  python-novaclient-3.3.0-1.el7.noarch
  openstack-nova-scheduler-13.0.0-1.el7.noarch
  python-nova-13.0.0-1.el7.noarch
  openstack-nova-conductor-13.0.0-1.el7.noarch
  openstack-nova-common-13.0.0-1.el7.noarch
  
  /etc/nova/nova.conf: http://paste.fedoraproject.org/367251/34295211/raw/
  
  Example log of when things are fast: http://paste.fedoraproject.org/367272/32783146/raw/
  Example log of when things are slow: http://paste.fedoraproject.org/367271/34327341/raw/
  
  Happy to help provider further details if required.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1582409

Title:
  nova-api intermittent performance issues

Status in OpenStack Compute (nova):
  New

Bug description:
  Relevant information TL;DR:
  RDO stable Mitaka deployed with OpenStack puppet modules.
  Keystone with fernet tokens, Nova with keystone_authtoken memcached caching set (also reproduced with sql tokens and nova without memcached caching for authtoken). nova-api is handled behind apache with mod wsgi.

  Problem TL;DR:
  In an empty new deployment on bare metal (no images, no virtual machines, etc.), "openstack server list" will intermittently take >5 seconds (up to ~10) to return. The only noticeable thing that happens when "openstack server list" is slow is that the logs will be plastered with a bunch of "_create_extension_schema", "_rebuild_extension_schema", "_register_resources" and so on.
  All other installed projects (glance, keystone, neutron) return command output consistently under 2 seconds.

  Version info:
  # rpm -qa |grep nova
  openstack-nova-api-13.0.0-1.el7.noarch
  python-novaclient-3.3.0-1.el7.noarch
  openstack-nova-scheduler-13.0.0-1.el7.noarch
  python-nova-13.0.0-1.el7.noarch
  openstack-nova-conductor-13.0.0-1.el7.noarch
  openstack-nova-common-13.0.0-1.el7.noarch

  /etc/nova/nova.conf:
  http://paste.fedoraproject.org/367251/34295211/raw/

  Example log of when things are fast: http://paste.fedoraproject.org/367272/32783146/raw/
  Example log of when things are slow: http://paste.fedoraproject.org/367271/34327341/raw/

  Happy to help provider further details if required.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1582409/+subscriptions