yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1522488] [NEW] 'TestBootVolumePattern' fails in 'gate-grenade-dsvm-multinode' job with "libvirtError: cannot read header '/dev/disk/by-path/ip-[...]-lun-1'"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kashyap Chamarthy <kchamart@xxxxxxxxxx>
Date: Thu, 03 Dec 2015 16:42:26 -0000
Reply-to: Bug 1522488 <1522488@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

There seem to be 17 occurrences of this error in the last 7 days
[starting 3DEC2015]:

    message: "libvirtError: cannot read header" AND tags:
"screen-n-cpu.txt"

And, 195 occurrences of  the below error message string when looked at
logstash, :

    message: "session 2 login rejected: Initiator failed authentication
with target" AND tags: "syslog"

From:
http://logs.openstack.org/80/251980/1/check/gate-grenade-dsvm-multinode/5716e36/logs/old/screen-n-cpu.txt.gz#_2015-12-02_10_57_35_812

[. . .]
2015-12-02 10:57:35.812 ERROR nova.compute.manager [req-b5b4691f-da88-4abd-bffd-a72d52242dae tempest-TestVolumeBootPattern-440462397 tempest-TestVolumeBootPattern-1264952841] [instance: 4d176a3a-2678-4207-99c6-f84894a23778] Instance failed to spawn
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] Traceback (most recent call last):
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/compute/manager.py", line 2155, in _build_resources
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     yield resources
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/compute/manager.py", line 2009, in _build_and_run_instance
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     block_device_info=block_device_info)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 2444, in spawn
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     block_device_info=block_device_info)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 4519, in _create_domain_and_network
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     xml, pause=pause, power_on=power_on)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 4449, in _create_domain
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     guest.launch(pause=pause)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 141, in launch
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     self._encoded_xml, errors='ignore')
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     six.reraise(self.type_, self.value, self.tb)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 136, in launch
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     return self._domain.createWithFlags(flags)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     rv = execute(f, *args, **kwargs)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     six.reraise(c, e, tb)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     rv = meth(*args, **kwargs)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 900, in createWithFlags
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] libvirtError: cannot read header '/dev/disk/by-path/ip-149.202.189.35:3260-iscsi-iqn.2010-10.org.openstack:volume-690bf429-192d-40f1-97bf-f558a8106208-lun-1': Input/output error
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] 
[. . .]


And, there are errors from 'iscsid' from syslog:

http://logs.openstack.org/80/251980/1/check/gate-grenade-dsvm-
multinode/5716e36/logs/syslog.txt.gz#_Dec__2_10_55_36

[. . .]
Dec  2 10:55:36 ubuntu iscsid: Kernel reported iSCSI connection 2:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Dec  2 10:55:36 ubuntu iscsid: Login failed to authenticate with target iqn.2010-10.org.openstack:volume-6fb73a19-26f9-4c76-95e0-ccfa57489593
Dec  2 10:55:36 ubuntu iscsid: session 1 login rejected: Initiator failed authentication with target
Dec  2 10:55:36 ubuntu object-auditor: Object audit (ALL) "forever" mode completed: 3.39s. Total quarantined: 0, Total errors: 0, Total files/sec: 7.66, Total bytes/sec: 9986657.72, Auditing time: 3.39, Rate: 1.00
Dec  2 10:55:36 ubuntu kernel: [ 2566.962683] type=1400 audit(1449053736.838:29): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-4ac13a03-c8c8-4c20-8cd1-2f2354131a67" pid=1807 comm="apparmor_parser"
Dec  2 10:55:38 ubuntu kernel: [ 2568.159143]  connection2:0: detected conn error (1020)
[. . .]

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1522488

Title:
  'TestBootVolumePattern' fails in 'gate-grenade-dsvm-multinode' job
  with "libvirtError: cannot read header '/dev/disk/by-
  path/ip-[...]-lun-1'"

Status in OpenStack Compute (nova):
  New

Bug description:
  There seem to be 17 occurrences of this error in the last 7 days
  [starting 3DEC2015]:

      message: "libvirtError: cannot read header" AND tags:
  "screen-n-cpu.txt"

  And, 195 occurrences of  the below error message string when looked at
  logstash, :

      message: "session 2 login rejected: Initiator failed
  authentication with target" AND tags: "syslog"

  From:
  http://logs.openstack.org/80/251980/1/check/gate-grenade-dsvm-multinode/5716e36/logs/old/screen-n-cpu.txt.gz#_2015-12-02_10_57_35_812

  [. . .]
  2015-12-02 10:57:35.812 ERROR nova.compute.manager [req-b5b4691f-da88-4abd-bffd-a72d52242dae tempest-TestVolumeBootPattern-440462397 tempest-TestVolumeBootPattern-1264952841] [instance: 4d176a3a-2678-4207-99c6-f84894a23778] Instance failed to spawn
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] Traceback (most recent call last):
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/compute/manager.py", line 2155, in _build_resources
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     yield resources
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/compute/manager.py", line 2009, in _build_and_run_instance
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     block_device_info=block_device_info)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 2444, in spawn
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     block_device_info=block_device_info)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 4519, in _create_domain_and_network
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     xml, pause=pause, power_on=power_on)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 4449, in _create_domain
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     guest.launch(pause=pause)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 141, in launch
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     self._encoded_xml, errors='ignore')
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     six.reraise(self.type_, self.value, self.tb)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 136, in launch
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     return self._domain.createWithFlags(flags)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     rv = execute(f, *args, **kwargs)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     six.reraise(c, e, tb)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     rv = meth(*args, **kwargs)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 900, in createWithFlags
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] libvirtError: cannot read header '/dev/disk/by-path/ip-149.202.189.35:3260-iscsi-iqn.2010-10.org.openstack:volume-690bf429-192d-40f1-97bf-f558a8106208-lun-1': Input/output error
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] 
  [. . .]

  
  And, there are errors from 'iscsid' from syslog:

  http://logs.openstack.org/80/251980/1/check/gate-grenade-dsvm-
  multinode/5716e36/logs/syslog.txt.gz#_Dec__2_10_55_36

  [. . .]
  Dec  2 10:55:36 ubuntu iscsid: Kernel reported iSCSI connection 2:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
  Dec  2 10:55:36 ubuntu iscsid: Login failed to authenticate with target iqn.2010-10.org.openstack:volume-6fb73a19-26f9-4c76-95e0-ccfa57489593
  Dec  2 10:55:36 ubuntu iscsid: session 1 login rejected: Initiator failed authentication with target
  Dec  2 10:55:36 ubuntu object-auditor: Object audit (ALL) "forever" mode completed: 3.39s. Total quarantined: 0, Total errors: 0, Total files/sec: 7.66, Total bytes/sec: 9986657.72, Auditing time: 3.39, Rate: 1.00
  Dec  2 10:55:36 ubuntu kernel: [ 2566.962683] type=1400 audit(1449053736.838:29): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-4ac13a03-c8c8-4c20-8cd1-2f2354131a67" pid=1807 comm="apparmor_parser"
  Dec  2 10:55:38 ubuntu kernel: [ 2568.159143]  connection2:0: detected conn error (1020)
  [. . .]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1522488/+subscriptions
Follow ups

[Bug 1522488] Re: 'TestBootVolumePattern' fails in 'gate-grenade-dsvm-multinode' job with "libvirtError: cannot read header '/dev/disk/by-path/ip-[...]-lun-1'"
From: Matt Riedemann, 2015-12-10