yahoo-eng-team team mailing list archive

[Andrew Bonney <2086675@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Hi,
For the past few releases (Caracal, Bobcat, maybe earlier) we have noticed that listing images is particularly slow. I've finally had some time to dig into this, and I believe I've tracked it to https://review.opendev.org/c/openstack/glance/+/886811

We're running the RBD back end primarily, but also have HTTP and Cinder
listed in 'enabled_backends', meaning that sort_image_locations executes
in full. It appears that when this function executes for the RBD back
end, it causes a connection to be opened to the back end. When doing a
full image list operation, this happens once for every image in the list
(the connection is not re-used). This appears to carry a 20-30ms time
penalty per image. As such, for any reasonable set of images the
response ends up taking several seconds.

In our case, images are unlikely to be held in more than one back end at
a time, and I noted that adding a length check to the locations list in
https://github.com/openstack/glance/blob/master/glance/common/utils.py#L718
so that the sorting doesn't occur when the list has just one element
resolves the performance issue entirely.

Whilst a length check is a workaround, does the sorting operation
actually require connections to be opened to the RBD back end? If they
are required, could the connections at least be re-used to avoid this
time penalty growing linearly with the number of images held by Glance?

Thanks

** Affects: glance
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/2086675

Title:
  Suspected performance regression for RBD back end linked to location
  sorting

Status in Glance:
  New

Bug description:
  Hi,
  For the past few releases (Caracal, Bobcat, maybe earlier) we have noticed that listing images is particularly slow. I've finally had some time to dig into this, and I believe I've tracked it to https://review.opendev.org/c/openstack/glance/+/886811

  We're running the RBD back end primarily, but also have HTTP and
  Cinder listed in 'enabled_backends', meaning that sort_image_locations
  executes in full. It appears that when this function executes for the
  RBD back end, it causes a connection to be opened to the back end.
  When doing a full image list operation, this happens once for every
  image in the list (the connection is not re-used). This appears to
  carry a 20-30ms time penalty per image. As such, for any reasonable
  set of images the response ends up taking several seconds.

  In our case, images are unlikely to be held in more than one back end
  at a time, and I noted that adding a length check to the locations
  list in
  https://github.com/openstack/glance/blob/master/glance/common/utils.py#L718
  so that the sorting doesn't occur when the list has just one element
  resolves the performance issue entirely.

  Whilst a length check is a workaround, does the sorting operation
  actually require connections to be opened to the RBD back end? If they
  are required, could the connections at least be re-used to avoid this
  time penalty growing linearly with the number of images held by
  Glance?

  Thanks

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/2086675/+subscriptions