← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #43841

[Bug 1530214] Re: Tempest failures due to iSCSI DB failure on compute node

  Thread Previous • Date Previous • • Thread Next 
*** This bug is a duplicate of bug 1324670 ***
    https://bugs.launchpad.net/bugs/1324670

Looks like this is an old one we thought was fixed on the brick side.  Removing Nova and marking as a duplicate of the original bug:
1324670


** No longer affects: nova

** This bug has been marked a duplicate of bug 1324670
   'iscsiadm ... -o delete' fails occasionally on bulk deployments

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1530214

Title:
  Tempest failures due to iSCSI DB failure on compute node

Status in os-brick:
  New

Bug description:
  Noticed a couple of these today in the SolidFire CI system.  These are initiator side errors in Nova.  Excerpt from log is below, but additional logs can also be viewed here:
  http://54.164.167.86/solidfire-ci-logs/refs-changes-67-244867-12/logs/screen-n-cpu.log.txt


  2015-12-30 20:53:50.829 ERROR nova.virt.block_device [req-0f1602ac-0604-4051-8d65-81ede0976dc5 tempest-DeleteServersTestJSON-1052804341 tempest-DeleteServersTestJSON-308638552] [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] Driver failed to attach volume 23049317-f6c2-40b2-9306-0b838848f911 at /dev/vdb
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] Traceback (most recent call last):
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]   File "/opt/stack/nova/nova/virt/block_device.py", line 288, in attach
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]     device_type=self['device_type'], encryption=encryption)
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1114, in attach_volume
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]     self._connect_volume(connection_info, disk_info)
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1064, in _connect_volume
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]     vol_driver.connect_volume(connection_info, disk_info)
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]   File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 84, in connect_volume
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]     device_info = self.connector.connect_volume(connection_info['data'])
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]     return f(*args, **kwargs)
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]   File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 763, in connect_volume
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]     connection_properties)
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]   File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 591, in _get_potential_volume_paths
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]     if self._connect_to_iscsi_portal(props):
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]   File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 1044, in _connect_to_iscsi_portal
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]     self._run_iscsiadm(connection_properties, ())
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]   File "/opt/stack/os-brick/os_brick/initiator/connector.py", line 948, in _run_iscsiadm
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]     delay_on_retry=delay_on_retry)
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 312, in execute
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]     cmd=sanitized_cmd)
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] ProcessExecutionError: Unexpected error while running command.
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-01.com.solidfire:l74j.uuid-23049317-f6c2-40b2-9306-0b838848f911.374375 -p 10.10.64.3:3260
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] Exit code: 6
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] Stdout: u''
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c] Stderr: u'iscsiadm: Could not execute operation on all records: encountered iSCSI database failure\n'
  2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance: d1526309-a49f-46b3-9a29-7d7ae285be9c]

To manage notifications about this bug go to:
https://bugs.launchpad.net/os-brick/+bug/1530214/+subscriptions

References

  Thread Previous • Date Previous • • Thread Next