yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

Public bug reported:

I've been trying to unskip the test_stamp_pattern scenario in tempest
and it's been randomly failing (but very recreatable) because when we
attach the 2nd volume (from volume snapshot) to the 2nd instance (from
snapshot), the attach API operation is successful but the device
partition never shows up on the guest.

https://review.openstack.org/#/c/218355/6,publish

There is dmesg debug output when the test fails and it clearly shows
there is no vdb attach:

http://logs.openstack.org/55/218355/6/check/gate-tempest-dsvm-
full/e0da291/console.html#_2015-09-02_17_48_53_884

Checking the libvirtd logs, test_stamp_pattern is running against domain
00000006 and I don't see any call to virDomainAttachDeviceFlags for that
domain.

I do see it for others, like:
http://logs.openstack.org/55/218355/6/check/gate-tempest-dsvm-full/e0da291/logs/libvirt/libvirtd.txt.gz#_2015-09-02_17_25_13_297

So it appears that libvirt just completely fails to do the attach device
call.

>From the n-cpu log, here is where we've come back from connecting the
iscsi volume via os-brick and we get the device xml for the attach all
to libvirt:

http://logs.openstack.org/55/218355/6/check/gate-tempest-dsvm-
full/e0da291/logs/screen-n-cpu.txt.gz#_2015-09-02_17_42_10_274

2015-09-02 17:42:10.274 DEBUG nova.virt.libvirt.config [req-
6b7dde2f-4466-4cbc-a0c6-de05880c3275 tempest-TestStampPattern-1375625978
tempest-TestStampPattern-227460230] Generated XML ('<disk type="block"
device="disk">\n <driver name="qemu" type="raw" cache="none"/>\n <source
dev="/dev/disk/by-path/ip-10.0.0.126:3260-iscsi-
iqn.2010-10.org.openstack:volume-
c0604f38-eb3d-4840-8ee3-6b0444faddb5-lun-1"/>\n <target bus="virtio"
dev="vdb"/>\n
<serial>c0604f38-eb3d-4840-8ee3-6b0444faddb5</serial>\n</disk>\n',)
to_xml /opt/stack/new/nova/nova/virt/libvirt/config.py:82

But there is no corresponding call for virDomainAttachDeviceFlags and
that domain in libvirtd logs - but it doesn't fail either, it's like
it's just silently ignored or something.

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: libvirt volumes

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1492026

Title:
  domain.attachDeviceFlags randomly doesn't do anything in libvirt

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  I've been trying to unskip the test_stamp_pattern scenario in tempest
  and it's been randomly failing (but very recreatable) because when we
  attach the 2nd volume (from volume snapshot) to the 2nd instance (from
  snapshot), the attach API operation is successful but the device
  partition never shows up on the guest.

  https://review.openstack.org/#/c/218355/6,publish

  There is dmesg debug output when the test fails and it clearly shows
  there is no vdb attach:

  http://logs.openstack.org/55/218355/6/check/gate-tempest-dsvm-
  full/e0da291/console.html#_2015-09-02_17_48_53_884

  Checking the libvirtd logs, test_stamp_pattern is running against
  domain 00000006 and I don't see any call to virDomainAttachDeviceFlags
  for that domain.

  I do see it for others, like:
  http://logs.openstack.org/55/218355/6/check/gate-tempest-dsvm-full/e0da291/logs/libvirt/libvirtd.txt.gz#_2015-09-02_17_25_13_297

  So it appears that libvirt just completely fails to do the attach
  device call.

  From the n-cpu log, here is where we've come back from connecting the
  iscsi volume via os-brick and we get the device xml for the attach all
  to libvirt:

  http://logs.openstack.org/55/218355/6/check/gate-tempest-dsvm-
  full/e0da291/logs/screen-n-cpu.txt.gz#_2015-09-02_17_42_10_274

  2015-09-02 17:42:10.274 DEBUG nova.virt.libvirt.config [req-
  6b7dde2f-4466-4cbc-a0c6-de05880c3275 tempest-
  TestStampPattern-1375625978 tempest-TestStampPattern-227460230]
  Generated XML ('<disk type="block" device="disk">\n <driver
  name="qemu" type="raw" cache="none"/>\n <source dev="/dev/disk/by-
  path/ip-10.0.0.126:3260-iscsi-iqn.2010-10.org.openstack:volume-
  c0604f38-eb3d-4840-8ee3-6b0444faddb5-lun-1"/>\n <target bus="virtio"
  dev="vdb"/>\n
  <serial>c0604f38-eb3d-4840-8ee3-6b0444faddb5</serial>\n</disk>\n',)
  to_xml /opt/stack/new/nova/nova/virt/libvirt/config.py:82

  But there is no corresponding call for virDomainAttachDeviceFlags and
  that domain in libvirtd logs - but it doesn't fail either, it's like
  it's just silently ignored or something.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1492026/+subscriptions