yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1563141] [NEW] Nova boot Fails InvalidBDM from GLANCE IMAGE

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Daniel Smith <daniel.smith@xxxxxxxxxxxx>
Date: Tue, 29 Mar 2016 01:12:25 -0000
Reply-to: Bug 1563141 <1563141@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Running Liberty and on one of the image (CN-00) below - while all the
other nodes booted and created VOLUME from IMAGE, this one started to
FAIL.  No idea why.

As with the other instances, this nova boot command should result in the
GLANCE Image being used to create the CINDER volume (note - there is
5.5TB available so its not a space issue - and the error doesnt reflect
that the call ever makes it to CINDER).

EXPECTED RESULT:  nova boots the Instance, creating the CINDER volume
from the glance image-id as outlined in the command.

GLANCE IMAGES:
root@node-16:/var/log/nova# glance image-list
+--------------------------------------+-----------------+
| ID                                   | Name            |
+--------------------------------------+-----------------+
| 6be3e076-691e-48cf-b9ed-264de01965b8 | CN-00_R2C25     |
| f47da74d-15d0-42b7-8261-c6d3b8f0dc2a | DB-00-1_R2C25   |
| fd1d5774-5139-4644-81ac-1d5293d8410e | MS-00_R2C25     |
| 9b2dc827-b04a-4def-b36c-b1ee11449705 | NAS             |
| 3444ce43-99b1-482e-97ef-35d7bb15aecc | PGPOOL-00_R2C25 |
| ee11f00a-6ab4-42b3-b86d-f1ee1966ca5f | TestVM          |
| a1503971-ddcf-4740-bcae-9efc3afe2988 | UTILITY-R2C25   |
+--------------------------------------+-----------------+


NOVA BOOT COMMAND:
root@node-16:/var/log/nova# nova boot --flavor MDN-8G-8CPU --block-device source=image,id=6be3e076-691e-48cf-b9ed-264de01965b8,dest=volume,bus=scsi,bootindex=0  --nic port-id=5f9c10e9-09b9-4116-9d3a-348882d78817 --nic port-id=1c74b4bd-aaf0-4681-a566-92fdf0f69b0c --nic port-id=d4e59190-24ad-466c-8551-1e29ed3416fa --nic port-id=c08fa16e-11da-49d7-b12c-291dfe836d4c CN-00


ERROR:
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InvalidBDM'> (HTTP 500) (Request-ID: req-94b9fea7-1f45-486c-9292-47b2a3e6dc56)


NOVA-API.LOG
2016-03-29 01:05:05.173 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "auth_strategy" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-29 01:05:05.174 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_auth_url" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-29 01:05:05.175 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_username" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-29 01:05:05.176 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_password" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-29 01:05:05.176 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_tenant_name" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Unexpected exception in API method
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 611, in create
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     **create_kwargs)
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 149, in inner
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1581, in create
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1211, in _create_instance
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     instance_group, check_server_group_quota)
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1060, in _provision_instances
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     quotas.rollback()
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1023, in _provision_instances
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     num_instances, i, shutdown_terminate)
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1499, in create_db_entry_for_new_instance
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     instance.destroy()
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1495, in create_db_entry_for_new_instance
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     context, instance, instance_type, block_device_mapping)
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1331, in _validate_bdm
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     raise exception.InvalidBDM(message=_("Images with "
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions InvalidBDM: Images with destination_type 'volume' need to have a non-zero size specified
2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions
2016-03-29 01:05:06.480 41670 INFO nova.api.openstack.wsgi [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InvalidBDM'>
2016-03-29 01:05:06.481 41670 INFO nova.osapi_compute.wsgi.server [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] 192.168.0.2 "POST /v2/d37e0a300f154eb099d40ef9cf362123/os-volumes_boot HTTP/1.1" status: 500 len: 425 time: 2.1962600


GLANCE-IMAGE SHOW: (you can see that the IMAGE is NOT zero bytes):
root@node-16:/var/log/nova# glance image-show 6be3e076-691e-48cf-b9ed-264de01965b8
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| checksum         | e1177f3cb1625b36c55b7d171ba9d240     |
| container_format | bare                                 |
| created_at       | 2016-03-29T00:27:28Z                 |
| disk_format      | qcow2                                |
| id               | 6be3e076-691e-48cf-b9ed-264de01965b8 |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | CN-00_R2C25                          |
| owner            | d37e0a300f154eb099d40ef9cf362123     |
| protected        | False                                |
| size             | 3437887488                           |
| status           | active                               |
| tags             | []                                   |
| updated_at       | 2016-03-29T00:28:08Z                 |
| virtual_size     | None                                 |
| visibility       | public                               |
+------------------+------------


MORE INFO:

root@node-16:/var/log/nova# nova --version
2.30.2
root@node-16:/var/log/nova#


IMPOSSIBLE TO ADD SCRIPT - since it would mean pushing the image file to you.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1563141

Title:
  Nova boot Fails InvalidBDM from GLANCE IMAGE

Status in OpenStack Compute (nova):
  New

Bug description:
  Running Liberty and on one of the image (CN-00) below - while all the
  other nodes booted and created VOLUME from IMAGE, this one started to
  FAIL.  No idea why.

  As with the other instances, this nova boot command should result in
  the GLANCE Image being used to create the CINDER volume (note - there
  is 5.5TB available so its not a space issue - and the error doesnt
  reflect that the call ever makes it to CINDER).

  EXPECTED RESULT:  nova boots the Instance, creating the CINDER volume
  from the glance image-id as outlined in the command.

  GLANCE IMAGES:
  root@node-16:/var/log/nova# glance image-list
  +--------------------------------------+-----------------+
  | ID                                   | Name            |
  +--------------------------------------+-----------------+
  | 6be3e076-691e-48cf-b9ed-264de01965b8 | CN-00_R2C25     |
  | f47da74d-15d0-42b7-8261-c6d3b8f0dc2a | DB-00-1_R2C25   |
  | fd1d5774-5139-4644-81ac-1d5293d8410e | MS-00_R2C25     |
  | 9b2dc827-b04a-4def-b36c-b1ee11449705 | NAS             |
  | 3444ce43-99b1-482e-97ef-35d7bb15aecc | PGPOOL-00_R2C25 |
  | ee11f00a-6ab4-42b3-b86d-f1ee1966ca5f | TestVM          |
  | a1503971-ddcf-4740-bcae-9efc3afe2988 | UTILITY-R2C25   |
  +--------------------------------------+-----------------+

  
  NOVA BOOT COMMAND:
  root@node-16:/var/log/nova# nova boot --flavor MDN-8G-8CPU --block-device source=image,id=6be3e076-691e-48cf-b9ed-264de01965b8,dest=volume,bus=scsi,bootindex=0  --nic port-id=5f9c10e9-09b9-4116-9d3a-348882d78817 --nic port-id=1c74b4bd-aaf0-4681-a566-92fdf0f69b0c --nic port-id=d4e59190-24ad-466c-8551-1e29ed3416fa --nic port-id=c08fa16e-11da-49d7-b12c-291dfe836d4c CN-00

  
  ERROR:
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.InvalidBDM'> (HTTP 500) (Request-ID: req-94b9fea7-1f45-486c-9292-47b2a3e6dc56)

  
  NOVA-API.LOG
  2016-03-29 01:05:05.173 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "auth_strategy" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
  2016-03-29 01:05:05.174 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_auth_url" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
  2016-03-29 01:05:05.175 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_username" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
  2016-03-29 01:05:05.176 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_password" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
  2016-03-29 01:05:05.176 41670 WARNING oslo_config.cfg [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Option "admin_tenant_name" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] Unexpected exception in API method
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 611, in create
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     **create_kwargs)
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 149, in inner
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1581, in create
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1211, in _create_instance
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     instance_group, check_server_group_quota)
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1060, in _provision_instances
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     quotas.rollback()
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1023, in _provision_instances
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     num_instances, i, shutdown_terminate)
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1499, in create_db_entry_for_new_instance
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     instance.destroy()
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1495, in create_db_entry_for_new_instance
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     context, instance, instance_type, block_device_mapping)
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1331, in _validate_bdm
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions     raise exception.InvalidBDM(message=_("Images with "
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions InvalidBDM: Images with destination_type 'volume' need to have a non-zero size specified
  2016-03-29 01:05:06.475 41670 ERROR nova.api.openstack.extensions
  2016-03-29 01:05:06.480 41670 INFO nova.api.openstack.wsgi [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.InvalidBDM'>
  2016-03-29 01:05:06.481 41670 INFO nova.osapi_compute.wsgi.server [req-94b9fea7-1f45-486c-9292-47b2a3e6dc56 39849a4b3f814d30b31178358712944f d37e0a300f154eb099d40ef9cf362123 - - -] 192.168.0.2 "POST /v2/d37e0a300f154eb099d40ef9cf362123/os-volumes_boot HTTP/1.1" status: 500 len: 425 time: 2.1962600

  
  GLANCE-IMAGE SHOW: (you can see that the IMAGE is NOT zero bytes):
  root@node-16:/var/log/nova# glance image-show 6be3e076-691e-48cf-b9ed-264de01965b8
  +------------------+--------------------------------------+
  | Property         | Value                                |
  +------------------+--------------------------------------+
  | checksum         | e1177f3cb1625b36c55b7d171ba9d240     |
  | container_format | bare                                 |
  | created_at       | 2016-03-29T00:27:28Z                 |
  | disk_format      | qcow2                                |
  | id               | 6be3e076-691e-48cf-b9ed-264de01965b8 |
  | min_disk         | 0                                    |
  | min_ram          | 0                                    |
  | name             | CN-00_R2C25                          |
  | owner            | d37e0a300f154eb099d40ef9cf362123     |
  | protected        | False                                |
  | size             | 3437887488                           |
  | status           | active                               |
  | tags             | []                                   |
  | updated_at       | 2016-03-29T00:28:08Z                 |
  | virtual_size     | None                                 |
  | visibility       | public                               |
  +------------------+------------

  
  MORE INFO:

  root@node-16:/var/log/nova# nova --version
  2.30.2
  root@node-16:/var/log/nova#

  
  IMPOSSIBLE TO ADD SCRIPT - since it would mean pushing the image file to you.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1563141/+subscriptions
Follow ups

[Bug 1563141] Re: Nova boot Fails InvalidBDM from GLANCE IMAGE
From: Daniel Smith, 2016-03-29