yahoo-eng-team team mailing list archive

[Andrey Pavlov <andrey.mp@xxxxxxxxx>]

same behavior in ec2-api. Some attached volumes have empty attached
list.

Request to cinder for attach volume -
http://logs.openstack.org/48/386348/1/check/gate-functional-nova-
network-dsvm-ec2api/226be9a/logs/screen-
ec2-api.txt.gz#_2016-10-17_15_02_00_560

and then this volume has status 'in-use' and empty attachments list.

** Also affects: ec2-api
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1633535

Title:
  Generic driver fails to attach Cinder volume to Nova VM

Status in Cinder:
  New
Status in ec2-api:
  New
Status in Manila:
  In Progress
Status in OpenStack Compute (nova):
  New

Bug description:
  Starting from 14th of October, Friday. Our CI jobs (Manila) with
  Generic driver (Cinder as backend) started failing with traces [1].

  Investigation showed that problem appears when we get volume from Cinder with "in-use" status but with empty attachments.
  And Nova considers it as attached, but Cinder does not. And it is not attached indeed.
  After all, it is impossible to detach "in-use" volume [4].
  It is not stable bug. It is concurrency-based bug. If we can do something in Manila then wait some time after each operation we do...

  Test gerrit change [2] proves that commit to Cinder [3] is THE CAUSE
  of a bug.

  Also, bug was reproduced even before merge of [3] with "gate-rally-
  dsvm-cinder" CI job [4], but, I assume, no one has paid attention to
  this. And this job fails from time to time exactly because of this [3]
  change.

  Local testing shows that IF bug appears then volume never gets
  attached and list of attachments stays empty. And waiting between
  'create' (wait until 'available' status) and 'attach' commands does
  not help at all.

  How to reproduce:
  1) Create VM
  2) Create Volume
  3) Attach volume (2) to the VM (1)
  4) Repeat steps 1-3 while bug does not appear.

  [1] http://logs.openstack.org/64/386364/1/check/gate-manila-tempest-
  dsvm-postgres-generic-singlebackend-ubuntu-xenial-
  nv/eef11b0/logs/screen-m-shr.txt.gz?level=TRACE#_2016-10-14_15_15_19_898

  [2] https://review.openstack.org/387915

  [3]
  https://github.com/openstack/cinder/commit/6f174b412696bfa6262a5bea3ac42f45efbbe2ce
  ( https://review.openstack.org/385122 )

  [4] http://logs.openstack.org/22/385122/1/check/gate-rally-dsvm-
  cinder/b0332e2/rally-
  plot/results.html.gz#/CinderVolumes.create_snapshot_and_attach_volume/failures

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1633535/+subscriptions