yahoo-eng-team team mailing list archive

[Antonio Messina <antonio.s.messina@xxxxxxxxx>]

Public bug reported:

Issue found on  Kilo 2015.1.0 on Ubuntu Trusty
(1:2015.1.0-0ubuntu1.1~cloud0) from http://ubuntu-
cloud.archive.canonical.com/ubuntu

When running "nova usage" on a tenant that started many instances
O(100k) during the current month, the following happens:

* nova-api is stuck at 100% for a long time
* as a consequence, nova CLI returns "ERROR (ConnectionRefused):
Unable to establish connection to ..."
* on MySQL slow query log I see there is a query like:

SELECT instance_system_metadata.created_at AS
instance_system_metadata_created_at,
instance_system_metadata.updated_at AS
instance_system_metadata_updated_at,
instance_system_metadata.deleted_at AS
instance_system_metadata_deleted_at, instance_system_metadata.deleted
AS instance_system_metadata_deleted, instance_system_metadata.id AS
instance_system_metadata_id, instance_system_metadata.`key` AS
instance_system_metadata_key, instance_system_metadata.value AS
instance_system_metadata_value, instance_system_metadata.instance_uuid
AS instance_system_metadata_instance_uuid
FROM instance_system_metadata
WHERE instance_system_metadata.deleted = 0 AND
 instance_system_metadata.instance_uuid IN (<list of ~100k UUID>)

which took 1.8 seconds.

Also, when logging in from Horizon, login is very slow, and I get an error
"Error: Unable to retrieve usage information.".

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1481262

Title:
  "nova usage" taking too much time with many VMs in database

Status in OpenStack Compute (nova):
  New

Bug description:
  Issue found on  Kilo 2015.1.0 on Ubuntu Trusty
  (1:2015.1.0-0ubuntu1.1~cloud0) from http://ubuntu-
  cloud.archive.canonical.com/ubuntu

  When running "nova usage" on a tenant that started many instances
  O(100k) during the current month, the following happens:

  * nova-api is stuck at 100% for a long time
  * as a consequence, nova CLI returns "ERROR (ConnectionRefused):
  Unable to establish connection to ..."
  * on MySQL slow query log I see there is a query like:

  SELECT instance_system_metadata.created_at AS
  instance_system_metadata_created_at,
  instance_system_metadata.updated_at AS
  instance_system_metadata_updated_at,
  instance_system_metadata.deleted_at AS
  instance_system_metadata_deleted_at, instance_system_metadata.deleted
  AS instance_system_metadata_deleted, instance_system_metadata.id AS
  instance_system_metadata_id, instance_system_metadata.`key` AS
  instance_system_metadata_key, instance_system_metadata.value AS
  instance_system_metadata_value, instance_system_metadata.instance_uuid
  AS instance_system_metadata_instance_uuid
  FROM instance_system_metadata
  WHERE instance_system_metadata.deleted = 0 AND
   instance_system_metadata.instance_uuid IN (<list of ~100k UUID>)

  which took 1.8 seconds.

  Also, when logging in from Horizon, login is very slow, and I get an error
  "Error: Unable to retrieve usage information.".

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1481262/+subscriptions