yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1832248] [NEW] tempest.api.volume.test_volumes_extend.VolumesExtendAttachedTest.test_extend_attached_volume failing when usinug the Q35 machine type

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lee Yarwood <lyarwood@xxxxxxxxxx>
Date: Mon, 10 Jun 2019 17:42:09 -0000
Reply-to: Bug 1832248 <1832248@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========

tempest.api.volume.test_volumes_extend.VolumesExtendAttachedTest.test_extend_attached_volume
is failing when using the Q35 machine type as configured as part of the
following DNM test change:

DNM: Run tempest-full-py3 with q35 machine type
https://review.opendev.org/#/c/662887/

libvirtd appears to be receiving a DEVICE_DELETED event from QEMU just
after the SCSI rescan and well before we attempt to block resize the
disk within the domain:

Instance UUID: b3b8394a-866c-441f-b792-14d3b7da464c
Domain name:   instance-00000055

http://logs.openstack.org/87/662887/19/check/tempest-full-
py3/e0caad1/controller/logs/screen-n-cpu.txt.gz?#_Jun_09_14_51_08_780983

http://logs.openstack.org/87/662887/19/check/tempest-full-
py3/e0caad1/controller/logs/libvirt/libvirtd_log.txt.gz#_2019-06-09_14_51_08_215

http://logs.openstack.org/87/662887/19/check/tempest-full-
py3/e0caad1/controller/logs/screen-n-cpu.txt.gz?#_Jun_09_14_51_20_840546

We currently end up waiting for 12 seconds here as os-brick is
attempting to find a mpath device, even when use_multipath=False. I've
created the following bug for this issue and proposed a change:

find_multipath_device_path being called needlessly by linuxscsi.extend_volume
https://bugs.launchpad.net/os-brick/+bug/1832247

FWIW this works around the issue locally for me by calling for a block
resize before QEMU has a chance to raise the DELETED_DEVICE notification
to libvirtd.


Steps to reproduce
==================
* Use the Q35 machine type

  [libvirt]\hw_machine_type = x86_64=q35

* Run the test_extend_attached_volume test

  $ tempest run --regex
tempest.api.volume.test_volumes_extend.VolumesExtendAttachedTest.test_extend_attached_volume

Expected result
===============
Test passes.

Actual result
=============
Test fails.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   1316c1c2850d2f966f335b628f7f5fe88cef611c

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libvirt + KVM
   qemu-system-x86                       1:2.11+dfsg-1ubuntu7.14 
   libvirt0:amd64                        4.0.0-1ubuntu8.10     

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   LVM/iSCSI

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

Logs & Configs
==============


2019-06-09 14:51:08.215+0000: 22679: debug : qemuMonitorJSONIOProcessLine:193 : Line [{"timestamp": {"seconds": 1560091868, "microseconds": 215572}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/virtio-disk1/virtio-backend"}}]

Jun 09 14:51:20.840546 ubuntu-bionic-rax-dfw-0007351839 nova-
compute[18218]: ERROR nova.virt.libvirt.driver [req-81eae1ea-9bab-
470b-8436-0c66701368b4 req-3be591ea-bcb2-44a6-bb9d-85adae6ca3c0 service
nova] [instance: b3b8394a-866c-441f-b792-14d3b7da464c] resizing block
device failed.: libvirt.libvirtError: invalid argument: invalid path:
/dev/sda

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1832248

Title:
  tempest.api.volume.test_volumes_extend.VolumesExtendAttachedTest.test_extend_attached_volume
  failing when usinug the Q35 machine type

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  tempest.api.volume.test_volumes_extend.VolumesExtendAttachedTest.test_extend_attached_volume
  is failing when using the Q35 machine type as configured as part of
  the following DNM test change:

  DNM: Run tempest-full-py3 with q35 machine type
  https://review.opendev.org/#/c/662887/

  libvirtd appears to be receiving a DEVICE_DELETED event from QEMU just
  after the SCSI rescan and well before we attempt to block resize the
  disk within the domain:

  Instance UUID: b3b8394a-866c-441f-b792-14d3b7da464c
  Domain name:   instance-00000055

  http://logs.openstack.org/87/662887/19/check/tempest-full-
  py3/e0caad1/controller/logs/screen-n-cpu.txt.gz?#_Jun_09_14_51_08_780983

  http://logs.openstack.org/87/662887/19/check/tempest-full-
  py3/e0caad1/controller/logs/libvirt/libvirtd_log.txt.gz#_2019-06-09_14_51_08_215

  http://logs.openstack.org/87/662887/19/check/tempest-full-
  py3/e0caad1/controller/logs/screen-n-cpu.txt.gz?#_Jun_09_14_51_20_840546

  We currently end up waiting for 12 seconds here as os-brick is
  attempting to find a mpath device, even when use_multipath=False. I've
  created the following bug for this issue and proposed a change:

  find_multipath_device_path being called needlessly by linuxscsi.extend_volume
  https://bugs.launchpad.net/os-brick/+bug/1832247

  FWIW this works around the issue locally for me by calling for a block
  resize before QEMU has a chance to raise the DELETED_DEVICE
  notification to libvirtd.

  
  Steps to reproduce
  ==================
  * Use the Q35 machine type

    [libvirt]\hw_machine_type = x86_64=q35

  * Run the test_extend_attached_volume test

    $ tempest run --regex
  tempest.api.volume.test_volumes_extend.VolumesExtendAttachedTest.test_extend_attached_volume

  Expected result
  ===============
  Test passes.

  Actual result
  =============
  Test fails.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

     1316c1c2850d2f966f335b628f7f5fe88cef611c

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     Libvirt + KVM
     qemu-system-x86                       1:2.11+dfsg-1ubuntu7.14 
     libvirt0:amd64                        4.0.0-1ubuntu8.10     

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     LVM/iSCSI

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     N/A

  Logs & Configs
  ==============

  
  2019-06-09 14:51:08.215+0000: 22679: debug : qemuMonitorJSONIOProcessLine:193 : Line [{"timestamp": {"seconds": 1560091868, "microseconds": 215572}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/virtio-disk1/virtio-backend"}}]

  Jun 09 14:51:20.840546 ubuntu-bionic-rax-dfw-0007351839 nova-
  compute[18218]: ERROR nova.virt.libvirt.driver [req-81eae1ea-9bab-
  470b-8436-0c66701368b4 req-3be591ea-bcb2-44a6-bb9d-85adae6ca3c0
  service nova] [instance: b3b8394a-866c-441f-b792-14d3b7da464c]
  resizing block device failed.: libvirt.libvirtError: invalid argument:
  invalid path: /dev/sda

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1832248/+subscriptions
Follow ups

[Bug 1832248] Re: tempest.api.volume.test_volumes_extend.VolumesExtendAttachedTest.test_extend_attached_volume failing when using the Q35 machine type
From: Alexandre arents, 2020-11-04