yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2054409] [NEW] <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) after upgrade

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Francesco Di Nucci <2054409@xxxxxxxxxxxxxxxxxx>
Date: Tue, 20 Feb 2024 08:26:10 -0000
Reply-to: Bug 2054409 <2054409@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Description
===========
After upgrading from Antelope to Bobcat, I'm unable to manage instances.

Steps to reproduce
==================

Following the upgrade guide
https://docs.openstack.org/nova/2023.2/admin/upgrades.html

* Pre-existing installation, Antelope 2023.1
* Upgraded the controller node to Bobcat 2023.2
* Ran 'nova-manage api_db sync' and 'nova-manage db sync'
* Upgraded other services, 'nova-status upgrade check' reports
everything as succesful.
* 'nova service-list' does not report orphaned records
* Ran 'nova-manage db online_data_migrations'

Expected result
===============
Upgrade successful, able to use the OpenStack instance

Actual result
=============
The last step does not complete succesfully. Even after running
it multiple times, there still is a pending operation:

# nova-manage db online_data_migrations
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
Running batches of 50 until complete
1 rows matched query populate_instance_compute_id, 0 migrated
+-------------------------------------+--------------+-----------+
|              Migration              | Total Needed | Completed |
+-------------------------------------+--------------+-----------+
|     fill_virtual_interface_list     |      0       |     0     |
|         migrate_empty_ratio         |      0       |     0     |
|   migrate_quota_classes_to_api_db   |      0       |     0     |
|    migrate_quota_limits_to_api_db   |      0       |     0     |
|      migration_migrate_to_uuid      |      0       |     0     |
|          populate_dev_uuids         |      0       |     0     |
|     populate_instance_compute_id    |      1       |     0     |
| populate_missing_availability_zones |      0       |     0     |
|      populate_queued_for_delete     |      0       |     0     |
|           populate_user_id          |      0       |     0     |
|            populate_uuids           |      0       |     0     |
+-------------------------------------+--------------+-----------+

Listing instances on the dashboard now fails with the error:
Error: Unable to retrieve instances. Details
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/
and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) (Request-ID: req-2fa94a10-3168-41a5-8b0f-6d7499465ff1)

Environment
===========
1. Exact version of OpenStack you are running? Bobcat 2023.2

2. Which hypervisor did you use? Compute nodes are running
qemu-kvm 8.0.0 and libvirt 9.5.0

2. Which storage type did you use? Cinder has two backends, NFS and
dcache

3. Which networking type did you use? Neutron with OpenVSwitch

Logs & Configs
==============

/var/log/nova/nova-api.log

Trying to list instances:
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] Unexpected exception in API method: neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0']
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   [Previous line repeated 2 more times]
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     servers = self._get_servers(req, is_detail=True)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     response = self._view_builder.detail(
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 474, in detail
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     self._add_security_grps(request, list(servers_dict["servers"]),
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 684, in _add_security_grps
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     security_group_api.get_instances_security_groups_bindings(
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 541, in get_instances_security_groups_bindings
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ports = _get_ports_from_server_list(servers, neutron)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 490, in _get_ports_from_server_list
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ports.extend(neutron.list_ports(**search_opts).get('ports'))
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 815, in list_ports
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return self.list('ports', self.ports_path, retrieve_all,
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 372, in list
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     for r in self._pagination(collection, path, **params):
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 387, in _pagination
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     res = self.get(path, params=params)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 356, in get
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return self.retry_request("GET", action, body=body,
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 333, in retry_request
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return self.do_request(method, action, body=body,
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 297, in do_request
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     self._handle_fault_response(status_code, replybody, resp)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 272, in _handle_fault_response
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     exception_handler_v20(status_code, error_body)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 90, in exception_handler_v20
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     raise client_exc(message=error_message,
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request.
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0']
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi
2024-02-20 08:57:14.750 66460 INFO nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.InternalServerError'>

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  Description
  ===========
  After upgrading from Antelope to Bobcat, I'm unable to manage instances.
  
  Steps to reproduce
  ==================
  
  Following the upgrade guide
  https://docs.openstack.org/nova/2023.2/admin/upgrades.html
  
  * Pre-existing installation, Antelope 2023.1
  * Upgraded the controller node to Bobcat 2023.2
  * Ran 'nova-manage api_db sync' and 'nova-manage db sync'
  * Upgraded other services, 'nova-status upgrade check' reports
  everything as succesful.
  * 'nova service-list' does not report orphaned records
  * Ran 'nova-manage db online_data_migrations'
  
- 
- 
  Expected result
  ===============
- After the execution of the steps above, what should have
- happened if the issue wasn't present?
+ Upgrade successful, able to use the OpenStack instance
  
  Actual result
  =============
- The last step does not complete succesfully. Even after running 
+ The last step does not complete succesfully. Even after running
  it multiple times, there still is a pending operation:
  
  # nova-manage db online_data_migrations
  Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
  Running batches of 50 until complete
  1 rows matched query populate_instance_compute_id, 0 migrated
  +-------------------------------------+--------------+-----------+
  |              Migration              | Total Needed | Completed |
  +-------------------------------------+--------------+-----------+
  |     fill_virtual_interface_list     |      0       |     0     |
  |         migrate_empty_ratio         |      0       |     0     |
  |   migrate_quota_classes_to_api_db   |      0       |     0     |
  |    migrate_quota_limits_to_api_db   |      0       |     0     |
  |      migration_migrate_to_uuid      |      0       |     0     |
  |          populate_dev_uuids         |      0       |     0     |
  |     populate_instance_compute_id    |      1       |     0     |
  | populate_missing_availability_zones |      0       |     0     |
  |      populate_queued_for_delete     |      0       |     0     |
  |           populate_user_id          |      0       |     0     |
  |            populate_uuids           |      0       |     0     |
  +-------------------------------------+--------------+-----------+
  
  Listing instances on the dashboard now fails with the error:
  Error: Unable to retrieve instances. Details
- Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ 
- and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) (Request-ID: req-2fa94a10-3168-41a5-8b0f-6d7499465ff1) 
+ Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/
+ and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) (Request-ID: req-2fa94a10-3168-41a5-8b0f-6d7499465ff1)
  
  Environment
  ===========
  1. Exact version of OpenStack you are running? Bobcat 2023.2
  
  2. Which hypervisor did you use? Compute nodes are running
  qemu-kvm 8.0.0 and libvirt 9.5.0
  
  2. Which storage type did you use? Cinder has two backends, NFS and
  dcache
  
  3. Which networking type did you use? Neutron with OpenVSwitch
  
  Logs & Configs
  ==============
  
  /var/log/nova/nova-api.log
  
  Trying to list instances:
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] Unexpected exception in API method: neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request.
  Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0']
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   [Previous line repeated 2 more times]
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     servers = self._get_servers(req, is_detail=True)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     response = self._view_builder.detail(
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 474, in detail
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     self._add_security_grps(request, list(servers_dict["servers"]),
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 684, in _add_security_grps
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     security_group_api.get_instances_security_groups_bindings(
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 541, in get_instances_security_groups_bindings
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ports = _get_ports_from_server_list(servers, neutron)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 490, in _get_ports_from_server_list
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ports.extend(neutron.list_ports(**search_opts).get('ports'))
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 815, in list_ports
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return self.list('ports', self.ports_path, retrieve_all,
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 372, in list
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     for r in self._pagination(collection, path, **params):
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 387, in _pagination
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     res = self.get(path, params=params)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 356, in get
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return self.retry_request("GET", action, body=body,
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 333, in retry_request
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return self.do_request(method, action, body=body,
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 297, in do_request
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     self._handle_fault_response(status_code, replybody, resp)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 272, in _handle_fault_response
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     exception_handler_v20(status_code, error_body)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 90, in exception_handler_v20
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     raise client_exc(message=error_message,
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request.
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0']
- 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi 
+ 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi
  2024-02-20 08:57:14.750 66460 INFO nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'neutronclient.common.exceptions.InternalServerError'>

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2054409

Title:
  <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP
  500) after upgrade

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  After upgrading from Antelope to Bobcat, I'm unable to manage instances.

  Steps to reproduce
  ==================

  Following the upgrade guide
  https://docs.openstack.org/nova/2023.2/admin/upgrades.html

  * Pre-existing installation, Antelope 2023.1
  * Upgraded the controller node to Bobcat 2023.2
  * Ran 'nova-manage api_db sync' and 'nova-manage db sync'
  * Upgraded other services, 'nova-status upgrade check' reports
  everything as succesful.
  * 'nova service-list' does not report orphaned records
  * Ran 'nova-manage db online_data_migrations'

  Expected result
  ===============
  Upgrade successful, able to use the OpenStack instance

  Actual result
  =============
  The last step does not complete succesfully. Even after running
  it multiple times, there still is a pending operation:

  # nova-manage db online_data_migrations
  Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
  Running batches of 50 until complete
  1 rows matched query populate_instance_compute_id, 0 migrated
  +-------------------------------------+--------------+-----------+
  |              Migration              | Total Needed | Completed |
  +-------------------------------------+--------------+-----------+
  |     fill_virtual_interface_list     |      0       |     0     |
  |         migrate_empty_ratio         |      0       |     0     |
  |   migrate_quota_classes_to_api_db   |      0       |     0     |
  |    migrate_quota_limits_to_api_db   |      0       |     0     |
  |      migration_migrate_to_uuid      |      0       |     0     |
  |          populate_dev_uuids         |      0       |     0     |
  |     populate_instance_compute_id    |      1       |     0     |
  | populate_missing_availability_zones |      0       |     0     |
  |      populate_queued_for_delete     |      0       |     0     |
  |           populate_user_id          |      0       |     0     |
  |            populate_uuids           |      0       |     0     |
  +-------------------------------------+--------------+-----------+

  Listing instances on the dashboard now fails with the error:
  Error: Unable to retrieve instances. Details
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/
  and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) (Request-ID: req-2fa94a10-3168-41a5-8b0f-6d7499465ff1)

  Environment
  ===========
  1. Exact version of OpenStack you are running? Bobcat 2023.2

  2. Which hypervisor did you use? Compute nodes are running
  qemu-kvm 8.0.0 and libvirt 9.5.0

  2. Which storage type did you use? Cinder has two backends, NFS and
  dcache

  3. Which networking type did you use? Neutron with OpenVSwitch

  Logs & Configs
  ==============

  /var/log/nova/nova-api.log

  Trying to list instances:
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] Unexpected exception in API method: neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request.
  Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0']
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   [Previous line repeated 2 more times]
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     servers = self._get_servers(req, is_detail=True)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     response = self._view_builder.detail(
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 474, in detail
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     self._add_security_grps(request, list(servers_dict["servers"]),
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 684, in _add_security_grps
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     security_group_api.get_instances_security_groups_bindings(
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 541, in get_instances_security_groups_bindings
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ports = _get_ports_from_server_list(servers, neutron)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 490, in _get_ports_from_server_list
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ports.extend(neutron.list_ports(**search_opts).get('ports'))
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 815, in list_ports
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return self.list('ports', self.ports_path, retrieve_all,
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 372, in list
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     for r in self._pagination(collection, path, **params):
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 387, in _pagination
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     res = self.get(path, params=params)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 356, in get
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return self.retry_request("GET", action, body=body,
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 333, in retry_request
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return self.do_request(method, action, body=body,
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 297, in do_request
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     self._handle_fault_response(status_code, replybody, resp)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 272, in _handle_fault_response
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     exception_handler_v20(status_code, error_body)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 90, in exception_handler_v20
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     raise client_exc(message=error_message,
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request.
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0']
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi
  2024-02-20 08:57:14.750 66460 INFO nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'neutronclient.common.exceptions.InternalServerError'>

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2054409/+subscriptions
Follow ups

[Bug 2054409] Re: <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) after upgrade
From: Sylvain Bauza, 2024-03-12