yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1517143] [NEW] nova-api is crashed during creating instance

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Marian Horban <mhorban@xxxxxxxxxxxx>
Date: Tue, 17 Nov 2015 17:12:20 -0000
Reply-to: Bug 1517143 <1517143@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

If nova-api started by Apache(devstack config NOVA_USE_MOD_WSGI=True) it
is crashed during creating instance with stack trace:

e81a26529b4ae2 and project 8a88d9cf7ff046a9a8b55e90466dec12. Resources: ['instances', 'ram', 'cores'] _get_quotas /opt/stack/nova/nova/quota.py:401
2015-10-16 11:54:13.486 DEBUG nova.quota [req-eb2e7b74-e2b2-4248-86ab-9b672f447a06 tempest-ServersAdminTestJSON-1309219026 tempest-ServersAdminTestJSON-633645645] Quotas for project 8a88d9cf7ff046a9a8b55e9
0466dec12 and user 97caf8fda3ef497a8de81a26529b4ae2 after resource sync: {'instances': 10, 'ram': 51200, 'cores': 20} reserve /opt/stack/nova/nova/quota.py:573
2015-10-16 11:54:13.487 ERROR nova.api.openstack.extensions [req-eb2e7b74-e2b2-4248-86ab-9b672f447a06 tempest-ServersAdminTestJSON-1309219026 tempest-ServersAdminTestJSON-633645645] Unexpected exception in
 API method
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 478, in wrapped
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 602, in create
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     **create_kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/hooks.py", line 149, in inner
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1569, in create
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1192, in _create_instance
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     instance_group, check_server_group_quota)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 994, in _provision_instances
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     context, instance_type, min_count, max_count)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 390, in _check_num_instances_quota
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     project_id=project_id, user_id=user_id)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 213, in wrapper
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return fn(self, *args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/quotas.py", line 92, in reserve
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     **deltas)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/quota.py", line 1343, in reserve
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     user_id=user_id)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/quota.py", line 583, in reserve
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     project_id=project_id, user_id=user_id)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/api.py", line 1119, in quota_reserve
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     project_id=project_id, user_id=user_id)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 204, in wrapper
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     ectxt.value = e.inner_exc
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 137, in wrapper
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 3538, in quota_reserve
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     elevated = context.elevated()
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/context.py", line 200, in elevated
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y = _reconstruct(x, rv, 1, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     state = deepcopy(state, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y = copier(x, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y[deepcopy(key, memo)] = deepcopy(value, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y = _reconstruct(x, rv, 1, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     state = deepcopy(state, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y = copier(x, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y[deepcopy(key, memo)] = deepcopy(value, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y = _reconstruct(x, rv, 1, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 329, in _reconstruct
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y = callable(*args)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy_reg.py", line 93, in __newobj__
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return cls.__new__(cls, *args)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions TypeError: object.__new__(thread.lock) is not safe, use thread.lock.__new__()
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions 
2015-10-16 11:54:13.488 INFO nova.api.openstack.wsgi [req-eb2e7b74-e2b2-4248-86ab-9b672f447a06 tempest-ServersAdminTestJSON-1309219026 tempest-ServersAdminTestJSON-633645645] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>
2015-10-16 11:54:13.489 DEBUG nova.api.openstack.wsgi [req-eb2e7b74-e2b2-4248-86ab-9b672f447a06 tempest-ServersAdminTestJSON-1309219026 tempest-ServersAdminTestJSON-633645645] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1175

** Affects: nova
     Importance: Undecided
     Assignee: Marian Horban (mhorban)
         Status: New

** Affects: python-keystoneclient
     Importance: Undecided
     Assignee: Marian Horban (mhorban)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Marian Horban (mhorban)

** Also affects: python-keystoneclient
   Importance: Undecided
       Status: New

** Changed in: python-keystoneclient
     Assignee: (unassigned) => Marian Horban (mhorban)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1517143

Title:
  nova-api is crashed during creating instance

Status in OpenStack Compute (nova):
  New
Status in python-keystoneclient:
  New

Bug description:
  If nova-api started by Apache(devstack config NOVA_USE_MOD_WSGI=True)
  it is crashed during creating instance with stack trace:

  e81a26529b4ae2 and project 8a88d9cf7ff046a9a8b55e90466dec12. Resources: ['instances', 'ram', 'cores'] _get_quotas /opt/stack/nova/nova/quota.py:401
  2015-10-16 11:54:13.486 DEBUG nova.quota [req-eb2e7b74-e2b2-4248-86ab-9b672f447a06 tempest-ServersAdminTestJSON-1309219026 tempest-ServersAdminTestJSON-633645645] Quotas for project 8a88d9cf7ff046a9a8b55e9
  0466dec12 and user 97caf8fda3ef497a8de81a26529b4ae2 after resource sync: {'instances': 10, 'ram': 51200, 'cores': 20} reserve /opt/stack/nova/nova/quota.py:573
  2015-10-16 11:54:13.487 ERROR nova.api.openstack.extensions [req-eb2e7b74-e2b2-4248-86ab-9b672f447a06 tempest-ServersAdminTestJSON-1309219026 tempest-ServersAdminTestJSON-633645645] Unexpected exception in
   API method
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 478, in wrapped
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 602, in create
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     **create_kwargs)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/hooks.py", line 149, in inner
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1569, in create
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 1192, in _create_instance
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     instance_group, check_server_group_quota)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 994, in _provision_instances
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     context, instance_type, min_count, max_count)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 390, in _check_num_instances_quota
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     project_id=project_id, user_id=user_id)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 213, in wrapper
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return fn(self, *args, **kwargs)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/quotas.py", line 92, in reserve
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     **deltas)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/quota.py", line 1343, in reserve
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     user_id=user_id)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/quota.py", line 583, in reserve
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     project_id=project_id, user_id=user_id)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/api.py", line 1119, in quota_reserve
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     project_id=project_id, user_id=user_id)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 204, in wrapper
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     ectxt.value = e.inner_exc
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 137, in wrapper
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 3538, in quota_reserve
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     elevated = context.elevated()
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/context.py", line 200, in elevated
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y = _reconstruct(x, rv, 1, memo)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     state = deepcopy(state, memo)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y = copier(x, memo)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y[deepcopy(key, memo)] = deepcopy(value, memo)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y = _reconstruct(x, rv, 1, memo)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     state = deepcopy(state, memo)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y = copier(x, memo)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y[deepcopy(key, memo)] = deepcopy(value, memo)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y = _reconstruct(x, rv, 1, memo)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy.py", line 329, in _reconstruct
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     y = callable(*args)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/copy_reg.py", line 93, in __newobj__
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions     return cls.__new__(cls, *args)
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions TypeError: object.__new__(thread.lock) is not safe, use thread.lock.__new__()
  2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions 
  2015-10-16 11:54:13.488 INFO nova.api.openstack.wsgi [req-eb2e7b74-e2b2-4248-86ab-9b672f447a06 tempest-ServersAdminTestJSON-1309219026 tempest-ServersAdminTestJSON-633645645] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.TypeError'>
  2015-10-16 11:54:13.489 DEBUG nova.api.openstack.wsgi [req-eb2e7b74-e2b2-4248-86ab-9b672f447a06 tempest-ServersAdminTestJSON-1309219026 tempest-ServersAdminTestJSON-633645645] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <type 'exceptions.TypeError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1175

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1517143/+subscriptions
Follow ups

[Bug 1517143] Re: nova-api is crashed during creating instance
From: Boris Bobrov, 2015-11-27