yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1770527] Re: add volume fails over 26vols and returns 500 API error with libvirt driver

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1770527@xxxxxxxxxxxxxxxxxx>
Date: Wed, 30 Jan 2019 02:16:51 -0000
Reply-to: Bug 1770527 <1770527@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/632904
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6489f2d2b44827d133dad9a3bb52436ee304a934
Submitter: Zuul
Branch:    master

commit 6489f2d2b44827d133dad9a3bb52436ee304a934
Author: melanie witt <melwittt@xxxxxxxxx>
Date:   Fri Jan 18 16:30:40 2019 +0000

    Raise 403 instead of 500 error from attach volume API
    
    Currently, the libvirt driver has a limit on the maximum number of
    disk devices allowed to attach to a single instance of 26. If a user
    attempts to attach a volume which would make the total number of
    attached disk devices > 26 for the instance, the user receives a
    500 error from the API.
    
    This adds a new exception type TooManyDiskDevices and raises it for the
    "No free disk devices names" condition, instead of InternalError, and
    handles it in the attach volume API. We raise TooManyDiskDevices
    directly from the libvirt driver because InternalError is ambiguous and
    can be raised for different error reasons within the same method call.
    
    Closes-Bug: #1770527
    
    Change-Id: I1b08ed6826d7eb41ecdfc7102e5e8fcf3d1eb2e1


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1770527

Title:
  add volume fails over 26vols and returns 500 API error with libvirt
  driver

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========
  openstack server add volume fails over 26vols

  Steps to reproduce
  ==================
  * I did execute this openstack command.

  # create a instance with single volume named sles15rc
  # openstack server add volume sles15rc vol2
  # openstack server add volume sles15rc vol3
       :
  # openstack server add volume sles15rc vol26
  # openstack server add volume sles15rc vol27
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'NovaException_Remote'> (HTTP 500) (Request-ID: req-d95fea94-31fe-4063-9262-a84088cbaf29)
  #

  Expected result
  ===============
  This command will success.
  # openstack server add volume sles15rc vol27
  and instance will had '/dev/vdaa' device volumes.

  Actual result
  =============
  # openstack server add volume sles15rc vol26
   instance will get named /dev/vdz volume.
  Next,
  # openstack server add volume sles15rc vol27
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'NovaException_Remote'> (HTTP 500) (Request-ID: req-d95fea94-31fe-4063-9262-a84088cbaf29)

  I did kvm based command, Its OK.

  #virsh attach-device instance-0000001e ~/vol27.xml
  #virsh attach-device instance-0000001e ~/vol28.xml

   and instance will had a volumes, vdaa,vdab
  It's nova limitation problem.

  I made a concept blueprint.
  https://blueprints.launchpad.net/nova/+spec/nova-improvement-of-maximum-attach-volumes-more-than-26-vols

  Environment
  ===========

  SOC7(Suse Openstack Cloud version 7 openstack-newton)

  Logs & Configs
  ==============
  Nova API returned this result.
   NovaException_Remote: No free disk device names for prefix 'vd'

  2018-05-11 08:51:11.602 3667 INFO nova.osapi_compute.wsgi.server [req-6cf5fdbf-9681-445b-8117-a61acd4057f4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "POST /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/6a324dc8-d2ea-4eba-9870-e2ada1cb2bf4/os-volume_attachments HTT
  P/1.1" status: 200 len: 522 time: 0.2834640
  2018-05-11 08:51:11.716 3668 INFO nova.osapi_compute.wsgi.server [-] 127.0.0.1 "GET / HTTP/1.1" status: 200 len: 491 time: 0.0007210
  2018-05-11 08:51:13.880 3668 INFO nova.api.openstack.wsgi [req-c7fc0e7a-13ec-48e7-947c-cd22232aa31f 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d68048
  7f6f19 - default default] HTTP exception thrown: Instance sles15rc could not be found.
  2018-05-11 08:51:13.881 3668 INFO nova.osapi_compute.wsgi.server [req-c7fc0e7a-13ec-48e7-947c-cd22232aa31f 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/sles15rc HTTP/1.1" status: 404 len: 432 time: 0.0449822
  2018-05-11 08:51:13.887 3668 INFO nova.api.openstack.wsgi [req-7cbf4817-8810-4c30-8e09-8b106f66dee4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d68048
  7f6f19 - default default] HTTP exception thrown: Instance sles15rc could not be found.
  2018-05-11 08:51:13.889 3668 INFO nova.osapi_compute.wsgi.server [req-7cbf4817-8810-4c30-8e09-8b106f66dee4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/sles15rc HTTP/1.1" status: 404 len: 432 time: 0.0053861
  2018-05-11 08:51:13.929 3668 INFO nova.osapi_compute.wsgi.server [req-40067327-b2a6-4194-acc6-17696377d123 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers?name=sles15rc HTTP/1.1" status: 200 len: 687 time: 0.0372620
  2018-05-11 08:51:14.096 3668 INFO nova.osapi_compute.wsgi.server [req-4a17ff34-f457-4e4b-8975-ad9fafa2e635 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/6a324dc8-d2ea-4eba-9870-e2ada1cb2bf4 HTTP/1.1" status: 200 len:
   3110 time: 0.1642580
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions [req-d95fea94-31fe-4063-9262-a84088cbaf29 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] Unexpected exception in API method
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapp
  ed
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/volumes.py", line 325, in
  create
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     volume_id, device)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 166, in inner
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 147, in inner
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3518, in attach_volume
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     disk_bus, device_type)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3455, in _attach_volume
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     device_type=device_type)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3437, in _create_volume_bd
  m
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     device_type=device_type)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 926, in reserve_block_d
  evice_name
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return cctxt.call(ctxt, 'reserve_block_device_name', **kw)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     retry=self.retry)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     retry=retry)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     raise result
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions NovaException_Remote: No free disk device names for prefix 'vd'
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     res = self.dispatcher.dispatch(message)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return self._do_dispatch(endpoint, method, ctxt, args)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     result = func(ctxt, **new_args)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     function_name, call_dict, binary)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     self.force_reraise()
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(self, context, *args, **kw)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     LOG.warning(msg, e, instance=instance)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     self.force_reraise()
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     kwargs['instance'], e, sys.exc_info())
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     self.force_reraise()
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4683, in reserve_block_device_name
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return do_reserve()
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4677, in do_reserve
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     instance, bdms, new_bdm)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1496, in _get_device_name_for_instance
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     instance, bdms, block_device_obj)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7864, in get_device_name_for_instance
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     block_device_obj, mapping=instance_info['mapping'])
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 394, in get_info_from_bdm
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     device_name = find_disk_dev_for_disk_bus(padded_mapping, bdm_bus)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 197, in find_disk_dev_for_disk_bus
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     dev_prefix)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions NovaException: No free disk device names for prefix 'vd'
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.613 3668 INFO nova.api.openstack.wsgi [req-d95fea94-31fe-4063-9262-a84088cbaf29 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d680487f6f19 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'NovaException_Remote'>

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1770527/+subscriptions
References

[Bug 1770527] [NEW] openstack server add volume fails over 26vols
From: Tsuyoshi Nagata, 2018-05-11