yahoo-eng-team team mailing list archive

[Patrick East <patrick.east@xxxxxxxxxxxxxxx>]

Public bug reported:

When disconnecting a volume from an instance the ISCSI multipath
connection is not always cleaned up correctly. When running the temepest
tests we see test failures related to this as the connection is not
closed, but then it is requesting to disconnect through the cinder
driver which ends up breaking the iscsi connection. The end result being
that there are still entries in /dev/disk/by-path for the old ISCSI
connections, but they are in an error state and cannot be used.

In the syslog we get errors like:

Oct 25 17:23:21 localhost kernel: [ 2974.200680]  connection44:0: detected conn error (1020)
Oct 25 17:23:21 localhost kernel: [ 2974.200819]  connection43:0: detected conn error (1020)
Oct 25 17:23:21 localhost iscsid: Kernel reported iSCSI connection 44:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Oct 25 17:23:21 localhost iscsid: Kernel reported iSCSI connection 43:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)

After running the tests if I run "multipath -l" there are numerous
entries (which shouldn't exist anymore), and if I run "iscsiadm -m node"
it will show the connections to the backend, even though they are
supposed to have been disconnected (and have been on the backend via the
cinder driver).

The disconnect code in cinder/brick seems to not suffere from these
issues, from the looks of the source code it works a little bit
differently when disconnecting multipath volumes and will always clean
up the scsi connection first. We might need to do something more like
that in nova/virt/libvirt/volume.py too.

I'm seeing this on the latest master and Juno branches, haven't yet
tested on icehouse but looks like it will probably repro there too.

** Affects: nova
     Importance: Undecided
     Assignee: Patrick East (patrick-east)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Patrick East (patrick-east)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1385798

Title:
  Multipath ISCSI connections left open after disconnecting volume with
  libvirt

Status in OpenStack Compute (Nova):
  New

Bug description:
  When disconnecting a volume from an instance the ISCSI multipath
  connection is not always cleaned up correctly. When running the
  temepest tests we see test failures related to this as the connection
  is not closed, but then it is requesting to disconnect through the
  cinder driver which ends up breaking the iscsi connection. The end
  result being that there are still entries in /dev/disk/by-path for the
  old ISCSI connections, but they are in an error state and cannot be
  used.

  In the syslog we get errors like:

  Oct 25 17:23:21 localhost kernel: [ 2974.200680]  connection44:0: detected conn error (1020)
  Oct 25 17:23:21 localhost kernel: [ 2974.200819]  connection43:0: detected conn error (1020)
  Oct 25 17:23:21 localhost iscsid: Kernel reported iSCSI connection 44:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
  Oct 25 17:23:21 localhost iscsid: Kernel reported iSCSI connection 43:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)

  After running the tests if I run "multipath -l" there are numerous
  entries (which shouldn't exist anymore), and if I run "iscsiadm -m
  node" it will show the connections to the backend, even though they
  are supposed to have been disconnected (and have been on the backend
  via the cinder driver).

  The disconnect code in cinder/brick seems to not suffere from these
  issues, from the looks of the source code it works a little bit
  differently when disconnecting multipath volumes and will always clean
  up the scsi connection first. We might need to do something more like
  that in nova/virt/libvirt/volume.py too.

  I'm seeing this on the latest master and Juno branches, haven't yet
  tested on icehouse but looks like it will probably repro there too.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1385798/+subscriptions