yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2054409] Re: <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) after upgrade

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sylvain Bauza <2054409@xxxxxxxxxxxxxxxxxx>
Date: Tue, 12 Mar 2024 15:06:30 -0000
Reply-to: Bug 2054409 <2054409@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Are you sure you also upgraded Neutron ? Apparently, the Neutron
metadata agent tries to call the neutron server by RPC but the server
doesn't support the RPC version from the agent.

Doesn't look to me a problem from Nova, so I'll close this bug report
but please reopen it if you find some problem on Nova.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2054409

Title:
  <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP
  500) after upgrade

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Description
  ===========
  After upgrading from Antelope to Bobcat, I'm unable to manage instances.

  Steps to reproduce
  ==================

  Following the upgrade guide
  https://docs.openstack.org/nova/2023.2/admin/upgrades.html

  * Pre-existing installation, Antelope 2023.1
  * Upgraded the controller node to Bobcat 2023.2
  * Ran 'nova-manage api_db sync' and 'nova-manage db sync'
  * Upgraded other services, 'nova-status upgrade check' reports
  everything as succesful.
  * 'nova service-list' does not report orphaned records
  * Ran 'nova-manage db online_data_migrations'

  Expected result
  ===============
  Upgrade successful, able to use the OpenStack instance

  Actual result
  =============
  The last step does not complete succesfully. Even after running
  it multiple times, there still is a pending operation:

  # nova-manage db online_data_migrations
  Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
  Running batches of 50 until complete
  1 rows matched query populate_instance_compute_id, 0 migrated
  +-------------------------------------+--------------+-----------+
  |              Migration              | Total Needed | Completed |
  +-------------------------------------+--------------+-----------+
  |     fill_virtual_interface_list     |      0       |     0     |
  |         migrate_empty_ratio         |      0       |     0     |
  |   migrate_quota_classes_to_api_db   |      0       |     0     |
  |    migrate_quota_limits_to_api_db   |      0       |     0     |
  |      migration_migrate_to_uuid      |      0       |     0     |
  |          populate_dev_uuids         |      0       |     0     |
  |     populate_instance_compute_id    |      1       |     0     |
  | populate_missing_availability_zones |      0       |     0     |
  |      populate_queued_for_delete     |      0       |     0     |
  |           populate_user_id          |      0       |     0     |
  |            populate_uuids           |      0       |     0     |
  +-------------------------------------+--------------+-----------+

  Listing instances on the dashboard now fails with the error:
  Error: Unable to retrieve instances. Details
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/
  and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) (Request-ID: req-2fa94a10-3168-41a5-8b0f-6d7499465ff1)

  Environment
  ===========
  1. Exact version of OpenStack you are running? Bobcat 2023.2

  2. Which hypervisor did you use? Compute nodes are running
  qemu-kvm 8.0.0 and libvirt 9.5.0

  2. Which storage type did you use? Cinder has two backends, NFS and
  dcache

  3. Which networking type did you use? Neutron with OpenVSwitch

  Logs & Configs
  ==============

  /var/log/nova/nova-api.log

  Trying to list instances:
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] Unexpected exception in API method: neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request.
  Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0']
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   [Previous line repeated 2 more times]
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     servers = self._get_servers(req, is_detail=True)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     response = self._view_builder.detail(
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 474, in detail
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     self._add_security_grps(request, list(servers_dict["servers"]),
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 684, in _add_security_grps
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     security_group_api.get_instances_security_groups_bindings(
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 541, in get_instances_security_groups_bindings
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ports = _get_ports_from_server_list(servers, neutron)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 490, in _get_ports_from_server_list
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ports.extend(neutron.list_ports(**search_opts).get('ports'))
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 815, in list_ports
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return self.list('ports', self.ports_path, retrieve_all,
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 372, in list
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     for r in self._pagination(collection, path, **params):
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 387, in _pagination
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     res = self.get(path, params=params)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 356, in get
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return self.retry_request("GET", action, body=body,
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 333, in retry_request
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     return self.do_request(method, action, body=body,
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 297, in do_request
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     self._handle_fault_response(status_code, replybody, resp)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     ret = obj(*args, **kwargs)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 272, in _handle_fault_response
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     exception_handler_v20(status_code, error_body)
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 90, in exception_handler_v20
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi     raise client_exc(message=error_message,
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request.
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0']
  2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi
  2024-02-20 08:57:14.750 66460 INFO nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'neutronclient.common.exceptions.InternalServerError'>

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2054409/+subscriptions
References

[Bug 2054409] [NEW] <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) after upgrade
From: Francesco Di Nucci, 2024-02-20