← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #72941

[Bug 1771700] [NEW] nova-lvm tempest job failing with InvalidDiskInfo

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

There has been a recent regression in the nova-lvm tempest job. The most
recent passing run was on 2018-05-11 [1][2], so something regressed it
between then and yesterday 2018-05-15.

The build fails and the following trace is seen in the n-cpu log:

May 15 23:01:40.174233 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager Traceback (most recent call last):
May 15 23:01:40.174457 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/manager.py", line 7343, in update_available_resource_for_node
May 15 23:01:40.174699 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
May 15 23:01:40.174922 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 664, in update_available_resource
May 15 23:01:40.175170 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     resources = self.driver.get_available_resource(nodename)
May 15 23:01:40.175414 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 6391, in get_available_resource
May 15 23:01:40.175641 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
May 15 23:01:40.175868 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7935, in _get_disk_over_committed_size_total
May 15 23:01:40.176091 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     config, block_device_info)
May 15 23:01:40.176333 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7852, in _get_instance_disk_info_from_config
May 15 23:01:40.176555 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     virt_size = disk_api.get_disk_size(path)
May 15 23:01:40.176773 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/disk/api.py", line 99, in get_disk_size
May 15 23:01:40.176994 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     return images.qemu_img_info(path).virtual_size
May 15 23:01:40.177215 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/images.py", line 87, in qemu_img_info
May 15 23:01:40.177452 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     raise exception.InvalidDiskInfo(reason=msg)
May 15 23:01:40.177674 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on /dev/stack-volumes-default/8a1d5912-13e1-4583-876e-a04396b6b712_disk : Unexpected error while running command.
May 15 23:01:40.177902 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager Command: /usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /dev/stack-volumes-default/8a1d5912-13e1-4583-876e-a04396b6b712_disk --force-share
May 15 23:01:40.178118 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager Exit code: 1
May 15 23:01:40.178344 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager Stdout: u''
May 15 23:01:40.178989 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager Stderr: u"qemu-img: Could not open '/dev/stack-volumes-default/8a1d5912-13e1-4583-876e-a04396b6b712_disk': Could not open '/dev/stack-volumes-default/8a1d5912-13e1-4583-876e-a04396b6b712_disk': Permission denied\n"

I think the failure is related to this change that merged on 2018-05-15:

https://review.openstack.org/567899

which moved the call of disk_api.get_disk_size(path) to be called for
all disk types instead of only for qcow2 and ploop. Based on the
surrounding code, only lvm.get_volume_size(path) should be called in the
case of lvm.

[1] http://zuul.openstack.org/builds.html?job_name=nova-lvm
[2] https://review.openstack.org/567916

** Affects: nova
     Importance: High
         Status: Confirmed


** Tags: libvirt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1771700

Title:
  nova-lvm tempest job failing with InvalidDiskInfo

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  There has been a recent regression in the nova-lvm tempest job. The
  most recent passing run was on 2018-05-11 [1][2], so something
  regressed it between then and yesterday 2018-05-15.

  The build fails and the following trace is seen in the n-cpu log:

  May 15 23:01:40.174233 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager Traceback (most recent call last):
  May 15 23:01:40.174457 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/manager.py", line 7343, in update_available_resource_for_node
  May 15 23:01:40.174699 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     rt.update_available_resource(context, nodename)
  May 15 23:01:40.174922 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/compute/resource_tracker.py", line 664, in update_available_resource
  May 15 23:01:40.175170 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     resources = self.driver.get_available_resource(nodename)
  May 15 23:01:40.175414 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 6391, in get_available_resource
  May 15 23:01:40.175641 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
  May 15 23:01:40.175868 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7935, in _get_disk_over_committed_size_total
  May 15 23:01:40.176091 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     config, block_device_info)
  May 15 23:01:40.176333 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7852, in _get_instance_disk_info_from_config
  May 15 23:01:40.176555 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     virt_size = disk_api.get_disk_size(path)
  May 15 23:01:40.176773 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/disk/api.py", line 99, in get_disk_size
  May 15 23:01:40.176994 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     return images.qemu_img_info(path).virtual_size
  May 15 23:01:40.177215 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager   File "/opt/stack/new/nova/nova/virt/images.py", line 87, in qemu_img_info
  May 15 23:01:40.177452 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager     raise exception.InvalidDiskInfo(reason=msg)
  May 15 23:01:40.177674 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on /dev/stack-volumes-default/8a1d5912-13e1-4583-876e-a04396b6b712_disk : Unexpected error while running command.
  May 15 23:01:40.177902 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager Command: /usr/bin/python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /dev/stack-volumes-default/8a1d5912-13e1-4583-876e-a04396b6b712_disk --force-share
  May 15 23:01:40.178118 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager Exit code: 1
  May 15 23:01:40.178344 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager Stdout: u''
  May 15 23:01:40.178989 ubuntu-xenial-rax-dfw-0004040560 nova-compute[28718]: ERROR nova.compute.manager Stderr: u"qemu-img: Could not open '/dev/stack-volumes-default/8a1d5912-13e1-4583-876e-a04396b6b712_disk': Could not open '/dev/stack-volumes-default/8a1d5912-13e1-4583-876e-a04396b6b712_disk': Permission denied\n"

  I think the failure is related to this change that merged on
  2018-05-15:

  https://review.openstack.org/567899

  which moved the call of disk_api.get_disk_size(path) to be called for
  all disk types instead of only for qcow2 and ploop. Based on the
  surrounding code, only lvm.get_volume_size(path) should be called in
  the case of lvm.

  [1] http://zuul.openstack.org/builds.html?job_name=nova-lvm
  [2] https://review.openstack.org/567916

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1771700/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next