yahoo-eng-team team mailing list archive

[Masanori Kuroha <2106380@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description:
The accumulation of instance_group_member records in the nova-api database causes the server list to slow down.

Details:
Information about instance group members is currently stored in the instance_group_member table of the nova-api database.
Even if an instance belonging to a particular instance group is deleted, it is not removed from the instance_group_member table. There is a mechanism to delete records of deleted instances from instance_group_member using the nova-manage db archive_deleted_rows command. [1]

In os-compute-api-version 2.96, it was updated to obtain and display the AZ information from request_specs when listing (showing) servers. [2]
At this time, instance_group information is also retrieved. [3]

If the records in instance_group_member have grown significantly, the
response time for the server list becomes extremely slow. In our
environment, there were nearly 20,000 instance_group_member records
linked to a single instance group (the cause of which was a bug in the
Kubernetes managed service on our OpenStack).

References:
[1] https://github.com/openstack/nova/blob/9d910ec4bf2a12baf3b5f0ec3bc41686413538fb/nova/cmd/manage.py#L493-L495
[2] https://review.opendev.org/c/openstack/nova/+/904568/17
[3] https://github.com/openstack/nova/blob/9d910ec4bf2a12baf3b5f0ec3bc41686413538fb/nova/objects/request_spec.py#L747-L763

Environment:
By os-compute-api-version 2.96

Our Reproduction Environment:
Dalmatial release (2024.2)

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2106380

Title:
  Slow server list response with API version 2.96

Status in OpenStack Compute (nova):
  New

Bug description:
  Description:
  The accumulation of instance_group_member records in the nova-api database causes the server list to slow down.

  Details:
  Information about instance group members is currently stored in the instance_group_member table of the nova-api database.
  Even if an instance belonging to a particular instance group is deleted, it is not removed from the instance_group_member table. There is a mechanism to delete records of deleted instances from instance_group_member using the nova-manage db archive_deleted_rows command. [1]

  In os-compute-api-version 2.96, it was updated to obtain and display the AZ information from request_specs when listing (showing) servers. [2]
  At this time, instance_group information is also retrieved. [3]

  If the records in instance_group_member have grown significantly, the
  response time for the server list becomes extremely slow. In our
  environment, there were nearly 20,000 instance_group_member records
  linked to a single instance group (the cause of which was a bug in the
  Kubernetes managed service on our OpenStack).

  References:
  [1] https://github.com/openstack/nova/blob/9d910ec4bf2a12baf3b5f0ec3bc41686413538fb/nova/cmd/manage.py#L493-L495
  [2] https://review.opendev.org/c/openstack/nova/+/904568/17
  [3] https://github.com/openstack/nova/blob/9d910ec4bf2a12baf3b5f0ec3bc41686413538fb/nova/objects/request_spec.py#L747-L763

  Environment:
  By os-compute-api-version 2.96

  Our Reproduction Environment:
  Dalmatial release (2024.2)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2106380/+subscriptions