yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

I think this was fixed by an infra change, the two hosts had the same
hostname so n-cpu was getting tripped up.

https://review.openstack.org/#/c/254858/

** Changed in: nova
       Status: New => Confirmed

** Changed in: nova
   Importance: Undecided => High

** Changed in: nova
       Status: Confirmed => Fix Released

** Changed in: nova
     Assignee: (unassigned) => Monty Taylor (mordred)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1522488

Title:
  'TestBootVolumePattern' fails in 'gate-grenade-dsvm-multinode' job
  with "libvirtError: cannot read header '/dev/disk/by-
  path/ip-[...]-lun-1'"

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  There seem to be 17 occurrences of this error in the last 7 days
  [starting 3DEC2015]:

      message: "libvirtError: cannot read header" AND tags:
  "screen-n-cpu.txt"

  And, 195 occurrences of  the below error message string when looked at
  logstash, :

      message: "session 2 login rejected: Initiator failed
  authentication with target" AND tags: "syslog"

  From:
  http://logs.openstack.org/80/251980/1/check/gate-grenade-dsvm-multinode/5716e36/logs/old/screen-n-cpu.txt.gz#_2015-12-02_10_57_35_812

  [. . .]
  2015-12-02 10:57:35.812 ERROR nova.compute.manager [req-b5b4691f-da88-4abd-bffd-a72d52242dae tempest-TestVolumeBootPattern-440462397 tempest-TestVolumeBootPattern-1264952841] [instance: 4d176a3a-2678-4207-99c6-f84894a23778] Instance failed to spawn
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] Traceback (most recent call last):
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/compute/manager.py", line 2155, in _build_resources
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     yield resources
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/compute/manager.py", line 2009, in _build_and_run_instance
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     block_device_info=block_device_info)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 2444, in spawn
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     block_device_info=block_device_info)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 4519, in _create_domain_and_network
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     xml, pause=pause, power_on=power_on)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 4449, in _create_domain
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     guest.launch(pause=pause)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 141, in launch
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     self._encoded_xml, errors='ignore')
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     six.reraise(self.type_, self.value, self.tb)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 136, in launch
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     return self._domain.createWithFlags(flags)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     rv = execute(f, *args, **kwargs)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     six.reraise(c, e, tb)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     rv = meth(*args, **kwargs)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 900, in createWithFlags
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] libvirtError: cannot read header '/dev/disk/by-path/ip-149.202.189.35:3260-iscsi-iqn.2010-10.org.openstack:volume-690bf429-192d-40f1-97bf-f558a8106208-lun-1': Input/output error
  2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] 
  [. . .]

  
  And, there are errors from 'iscsid' from syslog:

  http://logs.openstack.org/80/251980/1/check/gate-grenade-dsvm-
  multinode/5716e36/logs/syslog.txt.gz#_Dec__2_10_55_36

  [. . .]
  Dec  2 10:55:36 ubuntu iscsid: Kernel reported iSCSI connection 2:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
  Dec  2 10:55:36 ubuntu iscsid: Login failed to authenticate with target iqn.2010-10.org.openstack:volume-6fb73a19-26f9-4c76-95e0-ccfa57489593
  Dec  2 10:55:36 ubuntu iscsid: session 1 login rejected: Initiator failed authentication with target
  Dec  2 10:55:36 ubuntu object-auditor: Object audit (ALL) "forever" mode completed: 3.39s. Total quarantined: 0, Total errors: 0, Total files/sec: 7.66, Total bytes/sec: 9986657.72, Auditing time: 3.39, Rate: 1.00
  Dec  2 10:55:36 ubuntu kernel: [ 2566.962683] type=1400 audit(1449053736.838:29): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-4ac13a03-c8c8-4c20-8cd1-2f2354131a67" pid=1807 comm="apparmor_parser"
  Dec  2 10:55:38 ubuntu kernel: [ 2568.159143]  connection2:0: detected conn error (1020)
  [. . .]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1522488/+subscriptions