yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1530847] Re: Gate jobs are not detaching volumes and eventually fail with no devices left

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1530847@xxxxxxxxxxxxxxxxxx>
Date: Thu, 07 Jan 2016 05:04:22 -0000
Reply-to: Bug 1530847 <1530847@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/263664
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=f4a00c59655ec3b7d65cec79b6e88bee8a5d601d
Submitter: Jenkins
Branch:    master

commit f4a00c59655ec3b7d65cec79b6e88bee8a5d601d
Author: vponomaryov <vponomaryov@xxxxxxxxxxxx>
Date:   Tue Jan 5 12:29:31 2016 +0200

    Fix CI Tempest jobs
    
    After merge of change [1] to devstack project, preconfigured creds for
    Tempest cannot be used anymore. But Manila requires such.
    For the moment, latest Tempest interfaces cannot allow us to use
    preconfigured creds because of bug #1524717.
    So, until this bug fixed, we should workaround Devstack changes and
    set legacy opts explicitly.
    
    Also, redefine new Nutron option 'dnsmasq_local_resolv' to 'False' value,
    that is 'True' by default and was added in change [2].
    Default value causes Nova operations initiated by Manila be very slow.
    That leads to breakage of CI.
    
    [1] I65b56ff681d6c27094380693c953fbc3664eceb0
    [2] I17a884f467d307432a06f67a9dd93ed2fa6081a3
    
    Change-Id: I516a6c9ab4396b14e3984d5f810210e4fcf7ec85
    Related-Bug: #1531049
    Closes-Bug: #1530847


** Changed in: manila
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1530847

Title:
  Gate jobs are not detaching volumes and eventually fail with no
  devices left

Status in Manila:
  Fix Released
Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Manila project has been facing following error since "2016, january 3,
  ~16:00+" of ZUUL's timezone:

  Paste: http://paste.openstack.org/show/482932/

  Logs: http://logs.openstack.org/07/263207/1/check/gate-manila-tempest-
  dsvm-neutron-no-share-servers-
  multibackend/6717bb8/logs/screen-n-api.txt.gz?level=TRACE#_2016-01-04_11_47_11_087

  Raw:
  2016-01-04 11:47:11.087 ERROR nova.api.openstack.extensions [req-d3ea820b-5b7e-4174-aa92-60b5d6283ee9 nova service] Unexpected exception in API method
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 478, in wrapped
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/volumes.py", line 283, in create
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     volume_id, device)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 235, in wrapped
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     return func(self, context, target, *args, **kwargs)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 224, in inner
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 205, in inner
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 3082, in attach_volume
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     disk_bus, device_type)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 3055, in _attach_volume
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     device_type=device_type)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/rpcapi.py", line 813, in reserve_block_device_name
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     volume_bdm = cctxt.call(ctxt, 'reserve_block_device_name', **kw)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     retry=self.retry)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     retry=retry)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     raise result
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions TypeError: __init__() takes at most 2 arguments (3 given)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     executor_callback))
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     executor_callback)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     result = func(ctxt, **new_args)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/exception.py", line 110, in wrapped
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     payload)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/exception.py", line 89, in wrapped
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     return f(self, context, *args, **kw)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/manager.py", line 355, in decorated_function
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     LOG.warning(msg, e, instance=instance)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/manager.py", line 328, in decorated_function
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/manager.py", line 383, in decorated_function
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     kwargs['instance'], e, sys.exc_info())
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/manager.py", line 371, in decorated_function
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/manager.py", line 4649, in reserve_block_device_name
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     return do_reserve()
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/manager.py", line 4643, in do_reserve
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     instance, bdms, new_bdm)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/manager.py", line 1663, in _get_device_name_for_instance
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     instance, bdms, block_device_obj)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7249, in get_device_name_for_instance
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     block_device_obj, mapping=instance_info['mapping'])
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/virt/libvirt/blockinfo.py", line 397, in get_info_from_bdm
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     device_name = find_disk_dev_for_disk_bus(padded_mapping, bdm_bus)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/virt/libvirt/blockinfo.py", line 201, in find_disk_dev_for_disk_bus
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions     dev_prefix)
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions
  2016-01-04 11:47:11.087 26423 ERROR nova.api.openstack.extensions TypeError: __init__() takes at most 2 arguments (3 given)

  Currently observed things:

  1) Manila project has no recent changes that could cause it.
  2) Error appears on attempt to attach Cinder volume to Nova VM.
  3) Amount of attachments increased comparing to "ok" results of other runs before error appearance. Example: was 55, became 98 and then job timeout reached, so, it is not final count of attach attempts.
  4) Nova fails to dedicate device ID for volume starting from 26th device. This limit should not have been reached in normal situation.

  Suspect cause: reserved device IDs for Nova are not released as it was
  and expected. Nova has bug handling case when device ID cannot be
  reserved.

  https://github.com/openstack/nova/commit/9f9711578f6cca05b3b50b934d261f9b50a1e377
  ?

  How to reproduce:
  1) Create instance
  And then perform following in cycle 26 times:
  2) Create volume
  3) Attach volume to instance (1)
  4) Detach volume (2)

To manage notifications about this bug go to:
https://bugs.launchpad.net/manila/+bug/1530847/+subscriptions
References

[Bug 1530847] [NEW] TypeError attaching volume to instance
From: Valeriy Ponomaryov, 2016-01-04