yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #95770
[Bug 2107968] [NEW] Openstack cli not working right when nova compute quota reached
Public bug reported:
Hi,
While developing a loadtest on Openstack, we noticed that whenever we
continue to send a create request even though the quota is full, we
cannot run the openstack server list command for the relevant project.
root@ansible:/home/ansible# openstack server list --project project_1
HttpException: 500: Server Error for url: https://xxxxxxr:8774/v2.1/servers/detail?project_id=cebcf25ed92841c19071d97b3eb0a971&deleted=False&all_tenants=True, Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'KeyError'>
openstack server show 14acce64-d571-4fe1-ba24-238527bef5b4
+-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hostname | test-instance-140477 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | instance-00000607 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-076z19yn |
| OS-EXT-SRV-ATTR:root_device_name | None |
| | == |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| config_drive | |
| created | 2025-04-22T15:07:52Z |
| description | test-instance-140477 |
| fault | {'code': 413, 'created': '2025-04-22T15:07:53Z', 'message': 'Quota exceeded for ram: Requested 2048, but already used 8192 of 8192 ram'} |
| flavor | description=, disk='0', ephemeral='0', extra_specs.aggregate_instance_extra_specs:gpu='false', id='g1.small', is_disabled=, is_public='True', location=, |
| | name='g1.small', original_name='g1.small', ram='2048', rxtx_factor=, swap='0', vcpus='1' |
| hostId |
nova-api.log
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi [None req-5203360f-4e2e-4ee7-abfd-7ad2a92e96ed be99cabfed634654922be9f9cb58d75a 89cc6311ec74408b88c17eafa57ab891 - - default default] Unexpected
exception in API method: KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4'
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/wsgi.py", line 678, in wrapped
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi [Previous line repeated 2 more times]
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi servers = self._get_servers(req, is_detail=True)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi response = self._view_builder.detail(
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/views/servers.py", line 475, in detail
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi servers_dict = self._list_view(self.show, request, instances,
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/views/servers.py", line 535, in _list_view
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi provided_az=req_specs_dict[server.uuid])["server"]
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ~~~~~~~~~~~~~~^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4'
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi
2025-04-22 15:08:16.729 24 INFO nova.api.openstack.wsgi [None req-5203360f-4e2e-4ee7-abfd-7ad2a92e96ed be99cabfed634654922be9f9cb58d75a 89cc6311ec74408b88c17eafa57ab891 - - default default] HTTP except
ion thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'KeyError'>
** Affects: nova
Importance: Undecided
Status: New
** Description changed:
Hi,
While developing a loadtest on Openstack, we noticed that whenever we
continue to send a create request even though the quota is full, we
cannot run the openstack server list command for the relevant project.
-
root@ansible:/home/ansible# openstack server list --project project_1
- HttpException: 500: Server Error for url: https://int-api-osdev.portvmind.com.tr:8774/v2.1/servers/detail?project_id=cebcf25ed92841c19071d97b3eb0a971&deleted=False&all_tenants=True, Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
+ HttpException: 500: Server Error for url: https://xxxxxxr:8774/v2.1/servers/detail?project_id=cebcf25ed92841c19071d97b3eb0a971&deleted=False&all_tenants=True, Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'KeyError'>
openstack server show 14acce64-d571-4fe1-ba24-238527bef5b4
+-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hostname | test-instance-140477 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | instance-00000607 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-076z19yn |
| OS-EXT-SRV-ATTR:root_device_name | None |
| | == |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| config_drive | |
| created | 2025-04-22T15:07:52Z |
| description | test-instance-140477 |
| fault | {'code': 413, 'created': '2025-04-22T15:07:53Z', 'message': 'Quota exceeded for ram: Requested 2048, but already used 8192 of 8192 ram'} |
| flavor | description=, disk='0', ephemeral='0', extra_specs.aggregate_instance_extra_specs:gpu='false', id='g1.small', is_disabled=, is_public='True', location=, |
| | name='g1.small', original_name='g1.small', ram='2048', rxtx_factor=, swap='0', vcpus='1' |
| hostId |
-
-
nova-api.log
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi [None req-5203360f-4e2e-4ee7-abfd-7ad2a92e96ed be99cabfed634654922be9f9cb58d75a 89cc6311ec74408b88c17eafa57ab891 - - default default] Unexpected
- exception in API method: KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4'
+ exception in API method: KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4'
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/wsgi.py", line 678, in wrapped
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi [Previous line repeated 2 more times]
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi servers = self._get_servers(req, is_detail=True)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi response = self._view_builder.detail(
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/views/servers.py", line 475, in detail
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi servers_dict = self._list_view(self.show, request, instances,
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/views/servers.py", line 535, in _list_view
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi provided_az=req_specs_dict[server.uuid])["server"]
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ~~~~~~~~~~~~~~^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4'
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi
2025-04-22 15:08:16.729 24 INFO nova.api.openstack.wsgi [None req-5203360f-4e2e-4ee7-abfd-7ad2a92e96ed be99cabfed634654922be9f9cb58d75a 89cc6311ec74408b88c17eafa57ab891 - - default default] HTTP except
ion thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'KeyError'>
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2107968
Title:
Openstack cli not working right when nova compute quota reached
Status in OpenStack Compute (nova):
New
Bug description:
Hi,
While developing a loadtest on Openstack, we noticed that whenever we
continue to send a create request even though the quota is full, we
cannot run the openstack server list command for the relevant project.
root@ansible:/home/ansible# openstack server list --project project_1
HttpException: 500: Server Error for url: https://xxxxxxr:8774/v2.1/servers/detail?project_id=cebcf25ed92841c19071d97b3eb0a971&deleted=False&all_tenants=True, Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'KeyError'>
openstack server show 14acce64-d571-4fe1-ba24-238527bef5b4
+-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hostname | test-instance-140477 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | instance-00000607 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-076z19yn |
| OS-EXT-SRV-ATTR:root_device_name | None |
| | == |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| config_drive | |
| created | 2025-04-22T15:07:52Z |
| description | test-instance-140477 |
| fault | {'code': 413, 'created': '2025-04-22T15:07:53Z', 'message': 'Quota exceeded for ram: Requested 2048, but already used 8192 of 8192 ram'} |
| flavor | description=, disk='0', ephemeral='0', extra_specs.aggregate_instance_extra_specs:gpu='false', id='g1.small', is_disabled=, is_public='True', location=, |
| | name='g1.small', original_name='g1.small', ram='2048', rxtx_factor=, swap='0', vcpus='1' |
| hostId |
nova-api.log
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi [None req-5203360f-4e2e-4ee7-abfd-7ad2a92e96ed be99cabfed634654922be9f9cb58d75a 89cc6311ec74408b88c17eafa57ab891 - - default default] Unexpected
exception in API method: KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4'
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/wsgi.py", line 678, in wrapped
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/validation/__init__.py", line 286, in wrapper
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi [Previous line repeated 2 more times]
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi servers = self._get_servers(req, is_detail=True)
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi response = self._view_builder.detail(
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/views/servers.py", line 475, in detail
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi servers_dict = self._list_view(self.show, request, instances,
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/api/openstack/compute/views/servers.py", line 535, in _list_view
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi provided_az=req_specs_dict[server.uuid])["server"]
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi ~~~~~~~~~~~~~~^^^^^^^^^^^^^
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi KeyError: '14acce64-d571-4fe1-ba24-238527bef5b4'
2025-04-22 15:08:16.724 24 ERROR nova.api.openstack.wsgi
2025-04-22 15:08:16.729 24 INFO nova.api.openstack.wsgi [None req-5203360f-4e2e-4ee7-abfd-7ad2a92e96ed be99cabfed634654922be9f9cb58d75a 89cc6311ec74408b88c17eafa57ab891 - - default default] HTTP except
ion thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'KeyError'>
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2107968/+subscriptions
Follow ups
