yahoo-eng-team team mailing list archive

[James Page <james.page@xxxxxxxxxx>]

This bug was fixed in the package nova - 2:17.0.9-0ubuntu3~cloud0
---------------

 nova (2:17.0.9-0ubuntu3~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:17.0.9-0ubuntu3) bionic; urgency=medium
 .
   * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
     virsh disk attach does not fail silently (LP: #1825882).
   * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
     always disconnect volumes after libvirt exceptions (LP: #1826523).


** Changed in: cloud-archive/queens
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1825882

Title:
  [SRU] Virsh disk attach errors silently ignored

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive queens series:
  Fix Released
Status in Ubuntu Cloud Archive rocky series:
  Fix Committed
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) queens series:
  Fix Committed
Status in OpenStack Compute (nova) rocky series:
  Fix Committed
Status in OpenStack Compute (nova) stein series:
  Fix Committed
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Bionic:
  Fix Released
Status in nova source package in Cosmic:
  Fix Released
Status in nova source package in Disco:
  Fix Committed

Bug description:
  [Impact]

  The following commit (1) is causing volume attachments which fail due
  to libvirt device attach erros to be silently ignored and Nova report
  the attachment as successful.

  It seems that the original intention of the commit was to log a
  condition and re-raise the exeption, but if the exception is of type
  libvirt.libvirtError and does not contain the searched pattern, the
  exception is ignored. If you unindent the raise statement, errors are
  reported again.

  In our case we had ceph/apparmor configuration problems in compute
  nodes which prevented virsh attaching the device; volumes appeared as
  successfully attached but the corresponding block device missing in
  guests VMs. Other libvirt attach error conditions are ignored also, as
  when you have already occuppied device names (i.e. 'Target vdb already
  exists', device is busy, etc.)

  (1)
  https://github.com/openstack/nova/commit/78891c2305bff6e16706339a9c5eca99a84e409c

  [Test Case]

  * Deploy any OpenStack version up to Pike , which includes ceph backed cinder
  * Create a guest VM (openstack server ...)
  * Create a test cinder volume

  $ openstack volume create test --size 10

  * Force a drop on ceph traffic. Run the following command on the nova
  hypervisor on which the server runs.

  $ iptables -A OUTPUT -d ceph-mon-addr -p tcp --dport 6800 -j DROP

  * Attach the volume to a running instance.

  $ openstack server add volume 7151f507-a6b7-4f6d-a4cc-fd223d9feb5d
  742ff117-21ae-4d1b-a52b-5b37955716ff

  * This should cause the volume attachment to fail

  $ virsh domblklist instance-xxxxx
  Target Source
  ------------------------------------------------
  vda nova/7151f507-a6b7-4f6d-a4cc-fd223d9feb5d_disk

  No volume should attached after this step.

  * If the behavior is fixed:

     * Check that openstack server show , doesn't displays the displays the volume as attached.
     * Check that proper log entries states the libvirt exception and error.

  * If the behavior isn't fixed:

     * openstack server show <ID> , will display the volume in the
  volumes_attached property.

  [Expected result]

  * Volume attach fails and a proper exception is logged.

  [Actual result]

  * Volume attach fails but remains connected to the host and no further
  exception gets logged.

  [Regression Potential]

  * The patches have been cherry-picked from upstream which helps to
  reduce the regression potential of these fixes.

  [Other Info]

  * N/A

  Description

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1825882/+subscriptions