yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2116852] Re: nova-ceph-multistore job fails constantly with memory allocation error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <2116852@xxxxxxxxxxxxxxxxxx>
Date: Wed, 16 Jul 2025 11:27:48 -0000
Reply-to: Bug 2116852 <2116852@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
** Project changed: nova => ceph

** Also affects: nova
   Importance: Undecided
       Status: New

** Changed in: nova
   Importance: Undecided => High

** Changed in: nova
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2116852

Title:
  nova-ceph-multistore job fails constantly with memory allocation error

Status in Ceph:
  New
Status in OpenStack Compute (nova):
  Triaged

Bug description:
  https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-
  multistore&skip=0

  First failure was at 2025-07-11 08:33:54
  https://zuul.opendev.org/t/openstack/build/3c87e32281df4101b698b01ae6b1f3d5

  Since then each run fails as the test case
  test_extend_attached_encrypted_volume_luksv1[id-e93243ec-7c37-4b5b-a099-ebf052c13216]
  fails with

  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/api/volume/admin/test_encrypted_volumes_extend.py", line 41, in test_extend_attached_encrypted_volume_luksv1
      self._test_extend_attached_volume(volume)
    File "/opt/stack/tempest/tempest/api/volume/test_volumes_extend.py", line 173, in _test_extend_attached_volume
      self.assertTrue(
    File "/usr/lib/python3.12/unittest/case.py", line 727, in assertTrue
      raise self.failureException(msg)
  AssertionError: False is not true : Unexpected compute_extend_volume result 'Error' for request req-9ec57a2c-91f8-46f6-8718-6171d653020a.

  
  While the following log is visible in the nova-compute log:

  
  Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [req-ed795c55-2c45-4828-98ce-40c79568e8b6 req-9ec57a2c-91f8-46f6-8718-6171d653020a service nova] [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6] Unknown error when attempting to find the payload_offset for LUKSv1 encrypted disk rbd:volumes/volume-514bd4d2-0a87-427d-a69e-c03efe96ba05:id=cinder.: nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on rbd:volumes/volume-514bd4d2-0a87-427d-a69e-c03efe96ba05:id=cinder : Unexpected error while running command.
  76292	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: Command: /opt/stack/data/venv/bin/python3.12 -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info rbd:volumes/volume-514bd4d2-0a87-427d-a69e-c03efe96ba05:id=cinder --force-share --output=json
  76293	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: Exit code: -6
  76294	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: Stdout: ''
  76295	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: Stderr: 'failed to allocate memory for stack: Cannot allocate memory\n'
  76296	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6] Traceback (most recent call last):
  76297	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2969, in _resize_attached_encrypted_volume
  76298	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]     info = images.privileged_qemu_img_info(path)
  76299	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  76300	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]   File "/opt/stack/nova/nova/virt/images.py", line 57, in privileged_qemu_img_info
  76301	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]     info = nova.privsep.qemu.privileged_qemu_img_info(path, format=format)
  76302	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  76303	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_privsep/priv_context.py", line 267, in _wrap
  76304	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]     return self.channel.remote_call(name, args, kwargs,
  76305	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  76306	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_privsep/daemon.py", line 227, in remote_call
  76307	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6]     raise exc_type(*result[2])
  76308	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6] nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on rbd:volumes/volume-514bd4d2-0a87-427d-a69e-c03efe96ba05:id=cinder : Unexpected error while running command.
  76309	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6] Command: /opt/stack/data/venv/bin/python3.12 -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info rbd:volumes/volume-514bd4d2-0a87-427d-a69e-c03efe96ba05:id=cinder --force-share --output=json
  76310	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6] Exit code: -6
  76311	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6] Stdout: ''
  76312	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6] Stderr: 'failed to allocate memory for stack: Cannot allocate memory\n'
  76313	Jul 13 13:52:20.363897 np42aaac35a2c74 nova-compute[93712]: ERROR nova.virt.libvirt.driver [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6] 
  76314	Jul 13 13:52:20.367937 np42aaac35a2c74 nova-compute[93712]: WARNING nova.compute.manager [req-ed795c55-2c45-4828-98ce-40c79568e8b6 req-9ec57a2c-91f8-46f6-8718-6171d653020a service nova] [instance: 259e9dfb-5882-4de0-9672-c619cb9ea4e6] Extend volume failed, volume_id=514bd4d2-0a87-427d-a69e-c03efe96ba05, reason: Disk info file is invalid: qemu-img failed to execute on rbd:volumes/volume-514bd4d2-0a87-427d-a69e-c03efe96ba05:id=cinder : Unexpected error while running command.

  https://zuul.opendev.org/t/openstack/build/46d5b5c8de4f4cd38bd882693c9e1b06/log/controller/logs/screen-
  n-cpu.txt#76291

  
  logsearch log --job nova-ceph-multistore --result FAILURE --file controller/logs/screen-n-cpu.txt --days 7 'Unknown error when attempting to find the payload_offset for LUKSv1 encrypted disk rbd:volumes.*: nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on rbd:volumes.*: Unexpected error while running command.'

  Builds with matching logs 35/40:
  +----------------------------------+---------------------+--------------------+----------+-----------------------------------+--------+
  | uuid                             | finished            | project            | pipeline | review                            | branch |
  +----------------------------------+---------------------+--------------------+----------+-----------------------------------+--------+
  | 46d5b5c8de4f4cd38bd882693c9e1b06 | 2025-07-13T14:14:55 | openstack/nova     | check    | https://review.opendev.org/948186 | master |
  | 6eb3deb7440e4048830040f10f3cd4cb | 2025-07-13T04:54:29 | openstack/devstack | check    | https://review.opendev.org/954750 | master |
  | fe9ffb6424c74ef7b894aa6cd0b3fd6b | 2025-07-12T09:28:10 | openstack/nova     | check    | https://review.opendev.org/940508 | master |
  | 397e40dfc1b6488c8272bf425aa1041c | 2025-07-12T05:40:13 | openstack/tempest  | check    | https://review.opendev.org/954404 | master |
  | 6f087929b98844dfb1432b3623bba8a1 | 2025-07-11T22:59:56 | openstack/tempest  | check    | https://review.opendev.org/954404 | master |
  | a3c7f03f5c8145ff96f46dda94c2505a | 2025-07-11T18:53:59 | openstack/nova     | check    | https://review.opendev.org/940508 | master |
  | 25509218b73d44909f5aa13b17549a3e | 2025-07-11T17:52:28 | openstack/devstack | check    | https://review.opendev.org/954751 | master |
  | aeb854c93e684e22b82d53f698468447 | 2025-07-11T17:08:06 | openstack/devstack | check    | https://review.opendev.org/954750 | master |
  | caf13dce4ea345f18791e3355f7826ba | 2025-07-11T17:08:31 | openstack/nova     | check    | https://review.opendev.org/940508 | master |
  | 6409d84e7dce4beea35a08ef252a30e9 | 2025-07-11T16:55:14 | openstack/devstack | check    | https://review.opendev.org/915980 | master |
  | f992cc90cc5145e29c6742f33166571d | 2025-07-11T16:56:37 | openstack/devstack | check    | https://review.opendev.org/936089 | master |
  | c5ea790752424e739638dccf8c71cdd9 | 2025-07-11T15:58:40 | openstack/nova     | check    | https://review.opendev.org/952666 | master |
  | 7bbd100af7204665bce4152853c34e55 | 2025-07-11T15:24:39 | openstack/nova     | check    | https://review.opendev.org/953436 | master |
  | e3c0e66d95d94acdb1d3203d30496272 | 2025-07-11T16:21:40 | openstack/nova     | check    | https://review.opendev.org/948311 | master |
  | 6cb5dade8b074853a5c08523216916ba | 2025-07-11T16:00:42 | openstack/nova     | check    | https://review.opendev.org/953815 | master |
  | 533c09d432e14917aaf74aac4d720786 | 2025-07-11T15:39:09 | openstack/nova     | check    | https://review.opendev.org/951957 | master |
  | 6074e8fc93324f328d75c4f887d060b8 | 2025-07-11T16:00:11 | openstack/nova     | check    | https://review.opendev.org/948087 | master |
  | 6aeea098e1814ef28e19f7f0f520d3bb | 2025-07-11T15:55:59 | openstack/nova     | check    | https://review.opendev.org/948187 | master |
  | bfc1f8c9711c475e9a87885a902a4671 | 2025-07-11T15:52:37 | openstack/nova     | check    | https://review.opendev.org/953475 | master |
  | 00680c804d44472d861687b710c0a05b | 2025-07-11T16:04:51 | openstack/nova     | check    | https://review.opendev.org/948450 | master |
  | a1722357540a4216852e21179b9c9bef | 2025-07-11T15:57:02 | openstack/nova     | check    | https://review.opendev.org/948340 | master |
  | 31deee8d4e1b4ec2a21606c4092ad419 | 2025-07-11T15:58:44 | openstack/nova     | check    | https://review.opendev.org/948086 | master |
  | 03d3dd1627b64776b40f0592956de636 | 2025-07-11T15:35:00 | openstack/nova     | check    | https://review.opendev.org/950994 | master |
  | 75bb8c2efb814394ab02fd51fae70473 | 2025-07-11T15:40:12 | openstack/nova     | check    | https://review.opendev.org/948188 | master |
  | c87e4c615ceb40b99a3f355f0cc04758 | 2025-07-11T15:07:46 | openstack/nova     | check    | https://review.opendev.org/948186 | master |
  | ee8214075c4e4c90ba7623a5650cf199 | 2025-07-11T14:44:43 | openstack/nova     | check    | https://review.opendev.org/953338 | master |
  | 3f985e2d1c564154a3d697c010000ad1 | 2025-07-11T14:47:47 | openstack/nova     | check    | https://review.opendev.org/940508 | master |
  | 6b933d4362a548368ceee5ecfd5115dd | 2025-07-11T12:35:30 | openstack/nova     | check    | https://review.opendev.org/953704 | master |
  | c4335675bc3c41d09f4b7dc27d6bc052 | 2025-07-11T12:09:26 | openstack/nova     | check    | https://review.opendev.org/902688 | master |
  | bf6bf0af10bf40b3b9ed1310e99c7e60 | 2025-07-11T12:24:00 | openstack/nova     | check    | https://review.opendev.org/953705 | master |
  | 46b2b1a3d69c420b94f81c1be35e3da9 | 2025-07-11T12:12:51 | openstack/nova     | check    | https://review.opendev.org/953703 | master |
  | d5e8e47f4e574d0d8e4e2981a5f66f25 | 2025-07-11T12:15:29 | openstack/nova     | check    | https://review.opendev.org/940508 | master |
  | bcacd1b14bb44af4836a124ebfba230e | 2025-07-11T11:06:51 | openstack/nova     | check    | https://review.opendev.org/954210 | master |
  | 3269938364c143dfa4d7f40a08a7e3a6 | 2025-07-11T09:44:50 | openstack/nova     | check    | https://review.opendev.org/954613 | master |
  | 3c87e32281df4101b698b01ae6b1f3d5 | 2025-07-11T09:31:38 | openstack/nova     | check    | https://review.opendev.org/954149 | master |
  +----------------------------------+---------------------+--------------------+----------+-----------------------------------+--------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/ceph/+bug/2116852/+subscriptions
References

[Bug 2116852] [NEW] nova-ceph-multistore job fails constantly with memory allocation error
From: Balazs Gibizer, 2025-07-14