yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1770527] [NEW] openstack server add volume fails over 26vols

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Tsuyoshi Nagata <yukari.papa@xxxxxxxxx>
Date: Fri, 11 May 2018 00:28:26 -0000
Reply-to: Bug 1770527 <1770527@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
openstack server add volume fails over 26vols

Steps to reproduce
==================
* I did execute this openstack command.

# create the instance like sles15rc
# openstack server add volume sles15rc vol2
# openstack server add volume sles15rc vol3
     :
# openstack server add volume sles15rc vol26
# openstack server add volume sles15rc vol27
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'NovaException_Remote'> (HTTP 500) (Request-ID: req-d95fea94-31fe-4063-9262-a84088cbaf29)
#

Expected result
===============
This command will success.
# openstack server add volume sles15rc vol27
and instance will had '/dev/vdaa' device volumes.

Actual result
=============
# openstack server add volume sles15rc vol26
 instance will get named /dev/vdz volume.
Next,
# openstack server add volume sles15rc vol27
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'NovaException_Remote'> (HTTP 500) (Request-ID: req-d95fea94-31fe-4063-9262-a84088cbaf29)

I did kvm based command, Its OK.

#virsh attach-device instance-0000001e ~/vol27.xml
#virsh attach-device instance-0000001e ~/vol28.xml

 and instance will had a volumes, vdaa,vdab
It's nova limitation problem.

I made a concept blueprint.
https://blueprints.launchpad.net/nova/+spec/nova-improvement-of-maximum-attach-volumes-more-than-26-vols

Environment
===========

SOC7(Suse Openstack Cloud version 7 openstack-newton)

Logs & Configs
==============
Nova API returned this result.
 NovaException_Remote: No free disk device names for prefix 'vd'

2018-05-11 08:51:11.602 3667 INFO nova.osapi_compute.wsgi.server [req-6cf5fdbf-9681-445b-8117-a61acd4057f4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
5d680487f6f19 - default default] 10.19.3.70 "POST /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/6a324dc8-d2ea-4eba-9870-e2ada1cb2bf4/os-volume_attachments HTT
P/1.1" status: 200 len: 522 time: 0.2834640
2018-05-11 08:51:11.716 3668 INFO nova.osapi_compute.wsgi.server [-] 127.0.0.1 "GET / HTTP/1.1" status: 200 len: 491 time: 0.0007210
2018-05-11 08:51:13.880 3668 INFO nova.api.openstack.wsgi [req-c7fc0e7a-13ec-48e7-947c-cd22232aa31f 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d68048
7f6f19 - default default] HTTP exception thrown: Instance sles15rc could not be found.
2018-05-11 08:51:13.881 3668 INFO nova.osapi_compute.wsgi.server [req-c7fc0e7a-13ec-48e7-947c-cd22232aa31f 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/sles15rc HTTP/1.1" status: 404 len: 432 time: 0.0449822
2018-05-11 08:51:13.887 3668 INFO nova.api.openstack.wsgi [req-7cbf4817-8810-4c30-8e09-8b106f66dee4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d68048
7f6f19 - default default] HTTP exception thrown: Instance sles15rc could not be found.
2018-05-11 08:51:13.889 3668 INFO nova.osapi_compute.wsgi.server [req-7cbf4817-8810-4c30-8e09-8b106f66dee4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/sles15rc HTTP/1.1" status: 404 len: 432 time: 0.0053861
2018-05-11 08:51:13.929 3668 INFO nova.osapi_compute.wsgi.server [req-40067327-b2a6-4194-acc6-17696377d123 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers?name=sles15rc HTTP/1.1" status: 200 len: 687 time: 0.0372620
2018-05-11 08:51:14.096 3668 INFO nova.osapi_compute.wsgi.server [req-4a17ff34-f457-4e4b-8975-ad9fafa2e635 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/6a324dc8-d2ea-4eba-9870-e2ada1cb2bf4 HTTP/1.1" status: 200 len:
 3110 time: 0.1642580
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions [req-d95fea94-31fe-4063-9262-a84088cbaf29 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
5d680487f6f19 - default default] Unexpected exception in API method
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapp
ed
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/volumes.py", line 325, in
create
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     volume_id, device)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 166, in inner
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 147, in inner
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3518, in attach_volume
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     disk_bus, device_type)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3455, in _attach_volume
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     device_type=device_type)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3437, in _create_volume_bd
m
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     device_type=device_type)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 926, in reserve_block_d
evice_name
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return cctxt.call(ctxt, 'reserve_block_device_name', **kw)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     retry=self.retry)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     retry=retry)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     raise result
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions NovaException_Remote: No free disk device names for prefix 'vd'
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     res = self.dispatcher.dispatch(message)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return self._do_dispatch(endpoint, method, ctxt, args)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     result = func(ctxt, **new_args)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     function_name, call_dict, binary)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     self.force_reraise()
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(self, context, *args, **kw)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     LOG.warning(msg, e, instance=instance)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     self.force_reraise()
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     kwargs['instance'], e, sys.exc_info())
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     self.force_reraise()
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4683, in reserve_block_device_name
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return do_reserve()
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4677, in do_reserve
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     instance, bdms, new_bdm)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1496, in _get_device_name_for_instance
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     instance, bdms, block_device_obj)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7864, in get_device_name_for_instance
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     block_device_obj, mapping=instance_info['mapping'])
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 394, in get_info_from_bdm
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     device_name = find_disk_dev_for_disk_bus(padded_mapping, bdm_bus)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 197, in find_disk_dev_for_disk_bus
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     dev_prefix)
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions NovaException: No free disk device names for prefix 'vd'
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
2018-05-11 08:51:14.613 3668 INFO nova.api.openstack.wsgi [req-d95fea94-31fe-4063-9262-a84088cbaf29 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d680487f6f19 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'NovaException_Remote'>

** Affects: nova
     Importance: Undecided
     Assignee: Tsuyoshi Nagata (yukari-papa)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Tsuyoshi Nagata (yukari-papa)

** Description changed:

  Description
  ===========
  openstack server add volume fails over 26vols
  
  Steps to reproduce
  ==================
  * I did execute this openstack command.
  
  # create the instance like sles15rc
  # openstack server add volume sles15rc vol2
  # openstack server add volume sles15rc vol3
-      :
+      :
  # openstack server add volume sles15rc vol26
  # openstack server add volume sles15rc vol27
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'NovaException_Remote'> (HTTP 500) (Request-ID: req-d95fea94-31fe-4063-9262-a84088cbaf29)
  #
  
  Expected result
  ===============
  This command will success.
  # openstack server add volume sles15rc vol27
  and instance will had '/dev/vdaa' device volumes.
  
  Actual result
  =============
+ # openstack server add volume sles15rc vol26
+  instance will get named /dev/vdz volume.
+ Next,
  # openstack server add volume sles15rc vol27
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'NovaException_Remote'> (HTTP 500) (Request-ID: req-d95fea94-31fe-4063-9262-a84088cbaf29)
  
  I did kvm based command, Its OK.
  
  #virsh attach-device instance-0000001e ~/vol27.xml
  #virsh attach-device instance-0000001e ~/vol28.xml
  
-  and instance will had a volumes, vdaa,vdab
+  and instance will had a volumes, vdaa,vdab
  It's nova limitation problem.
  
  I made a concept blueprint.
  https://blueprints.launchpad.net/nova/+spec/nova-improvement-of-maximum-attach-volumes-more-than-26-vols
  
  Environment
  ===========
  
  SOC7(Suse Openstack Cloud version 7 openstack-newton)
  
  Logs & Configs
  ==============
  Nova API returned this result.
-  NovaException_Remote: No free disk device names for prefix 'vd'
+  NovaException_Remote: No free disk device names for prefix 'vd'
  
  2018-05-11 08:51:11.602 3667 INFO nova.osapi_compute.wsgi.server [req-6cf5fdbf-9681-445b-8117-a61acd4057f4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "POST /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/6a324dc8-d2ea-4eba-9870-e2ada1cb2bf4/os-volume_attachments HTT
  P/1.1" status: 200 len: 522 time: 0.2834640
  2018-05-11 08:51:11.716 3668 INFO nova.osapi_compute.wsgi.server [-] 127.0.0.1 "GET / HTTP/1.1" status: 200 len: 491 time: 0.0007210
  2018-05-11 08:51:13.880 3668 INFO nova.api.openstack.wsgi [req-c7fc0e7a-13ec-48e7-947c-cd22232aa31f 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d68048
  7f6f19 - default default] HTTP exception thrown: Instance sles15rc could not be found.
  2018-05-11 08:51:13.881 3668 INFO nova.osapi_compute.wsgi.server [req-c7fc0e7a-13ec-48e7-947c-cd22232aa31f 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/sles15rc HTTP/1.1" status: 404 len: 432 time: 0.0449822
  2018-05-11 08:51:13.887 3668 INFO nova.api.openstack.wsgi [req-7cbf4817-8810-4c30-8e09-8b106f66dee4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d68048
  7f6f19 - default default] HTTP exception thrown: Instance sles15rc could not be found.
  2018-05-11 08:51:13.889 3668 INFO nova.osapi_compute.wsgi.server [req-7cbf4817-8810-4c30-8e09-8b106f66dee4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/sles15rc HTTP/1.1" status: 404 len: 432 time: 0.0053861
  2018-05-11 08:51:13.929 3668 INFO nova.osapi_compute.wsgi.server [req-40067327-b2a6-4194-acc6-17696377d123 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers?name=sles15rc HTTP/1.1" status: 200 len: 687 time: 0.0372620
  2018-05-11 08:51:14.096 3668 INFO nova.osapi_compute.wsgi.server [req-4a17ff34-f457-4e4b-8975-ad9fafa2e635 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/6a324dc8-d2ea-4eba-9870-e2ada1cb2bf4 HTTP/1.1" status: 200 len:
-  3110 time: 0.1642580
+  3110 time: 0.1642580
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions [req-d95fea94-31fe-4063-9262-a84088cbaf29 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] Unexpected exception in API method
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapp
  ed
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/volumes.py", line 325, in
  create
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     volume_id, device)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 166, in inner
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 147, in inner
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3518, in attach_volume
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     disk_bus, device_type)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3455, in _attach_volume
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     device_type=device_type)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3437, in _create_volume_bd
  m
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     device_type=device_type)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 926, in reserve_block_d
  evice_name
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return cctxt.call(ctxt, 'reserve_block_device_name', **kw)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     retry=self.retry)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     retry=retry)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     raise result
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions NovaException_Remote: No free disk device names for prefix 'vd'
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     res = self.dispatcher.dispatch(message)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return self._do_dispatch(endpoint, method, ctxt, args)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     result = func(ctxt, **new_args)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     function_name, call_dict, binary)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     self.force_reraise()
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(self, context, *args, **kw)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     LOG.warning(msg, e, instance=instance)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     self.force_reraise()
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     kwargs['instance'], e, sys.exc_info())
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     self.force_reraise()
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4683, in reserve_block_device_name
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return do_reserve()
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4677, in do_reserve
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     instance, bdms, new_bdm)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1496, in _get_device_name_for_instance
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     instance, bdms, block_device_obj)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7864, in get_device_name_for_instance
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     block_device_obj, mapping=instance_info['mapping'])
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 394, in get_info_from_bdm
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     device_name = find_disk_dev_for_disk_bus(padded_mapping, bdm_bus)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 197, in find_disk_dev_for_disk_bus
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     dev_prefix)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions NovaException: No free disk device names for prefix 'vd'
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.613 3668 INFO nova.api.openstack.wsgi [req-d95fea94-31fe-4063-9262-a84088cbaf29 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d680487f6f19 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'NovaException_Remote'>

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1770527

Title:
  openstack server add volume fails over 26vols

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  openstack server add volume fails over 26vols

  Steps to reproduce
  ==================
  * I did execute this openstack command.

  # create the instance like sles15rc
  # openstack server add volume sles15rc vol2
  # openstack server add volume sles15rc vol3
       :
  # openstack server add volume sles15rc vol26
  # openstack server add volume sles15rc vol27
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'NovaException_Remote'> (HTTP 500) (Request-ID: req-d95fea94-31fe-4063-9262-a84088cbaf29)
  #

  Expected result
  ===============
  This command will success.
  # openstack server add volume sles15rc vol27
  and instance will had '/dev/vdaa' device volumes.

  Actual result
  =============
  # openstack server add volume sles15rc vol26
   instance will get named /dev/vdz volume.
  Next,
  # openstack server add volume sles15rc vol27
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'NovaException_Remote'> (HTTP 500) (Request-ID: req-d95fea94-31fe-4063-9262-a84088cbaf29)

  I did kvm based command, Its OK.

  #virsh attach-device instance-0000001e ~/vol27.xml
  #virsh attach-device instance-0000001e ~/vol28.xml

   and instance will had a volumes, vdaa,vdab
  It's nova limitation problem.

  I made a concept blueprint.
  https://blueprints.launchpad.net/nova/+spec/nova-improvement-of-maximum-attach-volumes-more-than-26-vols

  Environment
  ===========

  SOC7(Suse Openstack Cloud version 7 openstack-newton)

  Logs & Configs
  ==============
  Nova API returned this result.
   NovaException_Remote: No free disk device names for prefix 'vd'

  2018-05-11 08:51:11.602 3667 INFO nova.osapi_compute.wsgi.server [req-6cf5fdbf-9681-445b-8117-a61acd4057f4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "POST /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/6a324dc8-d2ea-4eba-9870-e2ada1cb2bf4/os-volume_attachments HTT
  P/1.1" status: 200 len: 522 time: 0.2834640
  2018-05-11 08:51:11.716 3668 INFO nova.osapi_compute.wsgi.server [-] 127.0.0.1 "GET / HTTP/1.1" status: 200 len: 491 time: 0.0007210
  2018-05-11 08:51:13.880 3668 INFO nova.api.openstack.wsgi [req-c7fc0e7a-13ec-48e7-947c-cd22232aa31f 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d68048
  7f6f19 - default default] HTTP exception thrown: Instance sles15rc could not be found.
  2018-05-11 08:51:13.881 3668 INFO nova.osapi_compute.wsgi.server [req-c7fc0e7a-13ec-48e7-947c-cd22232aa31f 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/sles15rc HTTP/1.1" status: 404 len: 432 time: 0.0449822
  2018-05-11 08:51:13.887 3668 INFO nova.api.openstack.wsgi [req-7cbf4817-8810-4c30-8e09-8b106f66dee4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d68048
  7f6f19 - default default] HTTP exception thrown: Instance sles15rc could not be found.
  2018-05-11 08:51:13.889 3668 INFO nova.osapi_compute.wsgi.server [req-7cbf4817-8810-4c30-8e09-8b106f66dee4 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/sles15rc HTTP/1.1" status: 404 len: 432 time: 0.0053861
  2018-05-11 08:51:13.929 3668 INFO nova.osapi_compute.wsgi.server [req-40067327-b2a6-4194-acc6-17696377d123 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers?name=sles15rc HTTP/1.1" status: 200 len: 687 time: 0.0372620
  2018-05-11 08:51:14.096 3668 INFO nova.osapi_compute.wsgi.server [req-4a17ff34-f457-4e4b-8975-ad9fafa2e635 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] 10.19.3.70 "GET /v2.1/b516d75b77bf4dc2b705d680487f6f19/servers/6a324dc8-d2ea-4eba-9870-e2ada1cb2bf4 HTTP/1.1" status: 200 len:
   3110 time: 0.1642580
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions [req-d95fea94-31fe-4063-9262-a84088cbaf29 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b70
  5d680487f6f19 - default default] Unexpected exception in API method
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapp
  ed
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/volumes.py", line 325, in
  create
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     volume_id, device)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 166, in inner
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 147, in inner
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3518, in attach_volume
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     disk_bus, device_type)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3455, in _attach_volume
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     device_type=device_type)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3437, in _create_volume_bd
  m
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     device_type=device_type)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 926, in reserve_block_d
  evice_name
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return cctxt.call(ctxt, 'reserve_block_device_name', **kw)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     retry=self.retry)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     retry=retry)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     raise result
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions NovaException_Remote: No free disk device names for prefix 'vd'
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     res = self.dispatcher.dispatch(message)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return self._do_dispatch(endpoint, method, ctxt, args)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     result = func(ctxt, **new_args)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     function_name, call_dict, binary)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     self.force_reraise()
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(self, context, *args, **kw)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     LOG.warning(msg, e, instance=instance)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     self.force_reraise()
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     kwargs['instance'], e, sys.exc_info())
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     self.force_reraise()
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4683, in reserve_block_device_name
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return do_reserve()
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4677, in do_reserve
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     instance, bdms, new_bdm)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1496, in _get_device_name_for_instance
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     instance, bdms, block_device_obj)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7864, in get_device_name_for_instance
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     block_device_obj, mapping=instance_info['mapping'])
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 394, in get_info_from_bdm
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     device_name = find_disk_dev_for_disk_bus(padded_mapping, bdm_bus)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 197, in find_disk_dev_for_disk_bus
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions     dev_prefix)
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions NovaException: No free disk device names for prefix 'vd'
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.611 3668 ERROR nova.api.openstack.extensions
  2018-05-11 08:51:14.613 3668 INFO nova.api.openstack.wsgi [req-d95fea94-31fe-4063-9262-a84088cbaf29 2e2bd0e5b23c4665a5a78d710e2bbeb5 b516d75b77bf4dc2b705d680487f6f19 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'NovaException_Remote'>

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1770527/+subscriptions
Follow ups

[Bug 1770527] Re: add volume fails over 26vols and returns 500 API error with libvirt driver
From: Matt Riedemann, 2019-01-30
[Bug 1770527] Re: add volume fails over 26vols and returns 500 API error with libvirt driver
From: OpenStack Infra, 2019-01-30