yahoo-eng-team team mailing list archive

[Eimad Zamrik <2114934@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

We found that Glance's image import flow, specifically during the image
conversion process where it executes 'qemu-img info', will randomly fail
with "RuntimeError: Unexpected error while running command."

Eventually we learned it was due to 'qemu-img info' sometimes taking
more than 2 seconds to process the image, causing the prlimits to kill
the process

Glance's image_conversion plugin currently hardcodes the cpu and as
prlimit parameters with values cpu=2 and as=(1 * units.Gi), which cannot
be overridden or configured via glance-api.conf

Changing the cpu prlimit to a higher value let's us image imports
successfully, and the error no longer appears

We found a similarly reported bug related to prlimits, in which they later added the ability to configure those prlimits in cinder
 - bug:    https://bugs.launchpad.net/nova/+bug/1705340
 - change: https://review.opendev.org/c/openstack/cinder/+/703682/5/cinder/image/image_utils.py

We think Glance should also allow users to override the prlimit
parameters via glance-api.conf

** Affects: glance
     Importance: Undecided
         Status: New

** Description changed:

  We found that Glance's image import flow, specifically during the image
  conversion process where it executes 'qemu-img info', will randomly fail
  with "RuntimeError: Unexpected error while running command."
  
  Eventually we learned it was due to 'qemu-img info' sometimes taking
  more than 2 seconds to process the image, causing the prlimits to kill
  the process
  
  Glance's image_conversion plugin currently hardcodes the cpu and as
  prlimit parameters with values cpu=2 and as=(1 * units.Gi), which cannot
  be overridden or configured via glance-api.conf
  
  Changing the cpu prlimit to a higher value let's us image imports
  successfully, and the error no longer appears
  
  we found a similarly reported bug related to prlimits, in which they later added the ability to configure those prlimits in cinder
-  - bug:    https://bugs.launchpad.net/nova/+bug/1705340
-  - change: https://review.opendev.org/c/openstack/cinder/+/703682/5/cinder/image/image_utils.py
+  - bug:    https://bugs.launchpad.net/nova/+bug/1705340
+  - change: https://review.opendev.org/c/openstack/cinder/+/703682/5/cinder/image/image_utils.py
  
  We think Glance should also allow users to override the prlimit
  parameters via glance-api.conf

** Description changed:

  We found that Glance's image import flow, specifically during the image
  conversion process where it executes 'qemu-img info', will randomly fail
  with "RuntimeError: Unexpected error while running command."
  
  Eventually we learned it was due to 'qemu-img info' sometimes taking
  more than 2 seconds to process the image, causing the prlimits to kill
  the process
  
  Glance's image_conversion plugin currently hardcodes the cpu and as
  prlimit parameters with values cpu=2 and as=(1 * units.Gi), which cannot
  be overridden or configured via glance-api.conf
  
  Changing the cpu prlimit to a higher value let's us image imports
  successfully, and the error no longer appears
  
- we found a similarly reported bug related to prlimits, in which they later added the ability to configure those prlimits in cinder
+ We found a similarly reported bug related to prlimits, in which they later added the ability to configure those prlimits in cinder
   - bug:    https://bugs.launchpad.net/nova/+bug/1705340
   - change: https://review.opendev.org/c/openstack/cinder/+/703682/5/cinder/image/image_utils.py
  
  We think Glance should also allow users to override the prlimit
  parameters via glance-api.conf

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/2114934

Title:
  Glance image_conversion plugin does not allow for configuration of
  prlimit parameters

Status in Glance:
  New

Bug description:
  We found that Glance's image import flow, specifically during the
  image conversion process where it executes 'qemu-img info', will
  randomly fail with "RuntimeError: Unexpected error while running
  command."

  Eventually we learned it was due to 'qemu-img info' sometimes taking
  more than 2 seconds to process the image, causing the prlimits to kill
  the process

  Glance's image_conversion plugin currently hardcodes the cpu and as
  prlimit parameters with values cpu=2 and as=(1 * units.Gi), which
  cannot be overridden or configured via glance-api.conf

  Changing the cpu prlimit to a higher value let's us image imports
  successfully, and the error no longer appears

  We found a similarly reported bug related to prlimits, in which they later added the ability to configure those prlimits in cinder
   - bug:    https://bugs.launchpad.net/nova/+bug/1705340
   - change: https://review.opendev.org/c/openstack/cinder/+/703682/5/cinder/image/image_utils.py

  We think Glance should also allow users to override the prlimit
  parameters via glance-api.conf

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/2114934/+subscriptions