yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #86311
[Bug 1931707] [NEW] "NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception" error while instantiating instance in Train RDO
Public bug reported:
Hi, Following ticket has been reported in https://bugzilla.redhat.com/show_bug.cgi?id=1964893
Based on their answer, verified the neutron command manually and it works fine.
The issue is occurring randomly and failing the instance creation
usecase.
Recently had failed 2 cases,
1) it failed once out of 72 runs
2021-06-09 10:00:49.391 3194 INFO heat.engine.resource [req-6d38438b-be89-4619-8517-9edc49bc7a40 - instTermProject-1 - default default] CREATE: Net "vm_network_3" [5776a857-4316-47f9-b3fa-3a1fbff39132] Stack "vapp-Service-060821210256-1" [c7fa51bb-2012-4f5c-aa6f-1720dd47a5ed]
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Traceback (most recent call last):
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 920, in _action_recorder
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource yield
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1033, in _do_action
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource step = next(subtask)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 982, in action_handler_task
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource done = check(handler_data)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/net.py", line 208, in check_create_complete
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource attributes = self._show_resource()
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/neutron.py", line 139, in _show_resource
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource res_info = client_method(*args)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 822, in show_network
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource return self.get(self.network_path % (network), params=_params)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource headers=headers, params=params)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource headers=headers, params=params)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource self._handle_fault_response(status_code, replybody, resp)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource exception_handler_v20(status_code, error_body)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource request_ids=request_ids)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Neutron server returns request_ids: ['req-8eb8d67b-1b44-4caf-8c5a-37a442494b31']
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource
2021-06-09 10:00:49.429 3194 INFO heat.engine.stack [req-6d38438b-be89-4619-8517-9edc49bc7a40 - instTermProject-1 - default default] Stack CREATE FAILED (vapp-Service-060821210256-1): Resource CREATE failed: Unauthorized: resources.vm_network_3: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
Neutron server returns request_ids: ['req-8eb8d67b-1b44-4caf-8c5a-37a442494b31']
2) it failed once out of 72 runs
2021-06-10 10:01:27.888 13501 INFO nova.compute.manager [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Took 8.84 seconds to spawn the instance on the hypervisor.
2021-06-10 10:01:28.034 13501 ERROR nova.network.neutronv2.api [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] Neutron client was not able to generate a valid admin token, please verify Neutron admin credential located in nova.conf: Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Failed to deallocate networks: NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Traceback (most recent call last):
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2746, in _cleanup_allocated_networks
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] self._deallocate_network(context, instance, requested_networks)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2002, in _deallocate_network
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] context, instance, requested_networks=requested_networks)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1570, in deallocate_for_instance
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] data = neutron.list_ports(**search_opts)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] **_params)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] for r in self._pagination(collection, path, **params):
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] res = self.get(path, params=params)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] headers=headers, params=params)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] headers=headers, params=params)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] self._handle_fault_response(status_code, replybody, resp)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 137, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] raise exception.NeutronAdminCredentialConfigurationInvalid()
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]
2021-06-10 10:01:28.154 13501 INFO nova.scheduler.client.report [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] Deleted allocation for instance f0dc327b-1416-400f-9846-86b9ef0dfc71
I did tried with neutron auth config from nova.conf manually and it works fine.
No similar issue observed with Queens openstack version.
Also many errors related to rabbitmq
2021-06-11 03:28:26.028 26179 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2021-06-11 03:28:26.034 26158 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2021-06-11 03:28:26.056 26179 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2021-06-11 03:28:26.057 26158 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
openstack-nova-novncproxy-20.6.0-1.el7.noarch
openstack-nova-api-20.6.0-1.el7.noarch
openstack-nova-common-20.6.0-1.el7.noarch
puppet-nova-15.7.0-1.el7.noarch
python2-nova-20.6.0-1.el7.noarch
openstack-nova-scheduler-20.6.0-1.el7.noarch
openstack-nova-conductor-20.6.0-1.el7.noarch
python2-novaclient-15.1.1-1.el7.noarch
openstack-neutron-common-15.3.2-1.el7.noarch
puppet-neutron-15.5.0-1.el7.noarch
python2-neutronclient-6.14.1-1.el7.noarch
openstack-neutron-metering-agent-15.3.2-1.el7.noarch
openstack-neutron-15.3.2-1.el7.noarch
openstack-neutron-openvswitch-15.3.2-1.el7.noarch
python2-neutron-lib-1.29.1-1.el7.noarch
openstack-neutron-ml2-15.3.2-1.el7.noarch
python2-neutron-15.3.2-1.el7.noarch
openstack-keystone-16.0.1-1.el7.noarch
python2-keystonemiddleware-7.0.1-2.el7.noarch
puppet-keystone-15.4.0-1.el7.noarch
python2-keystoneclient-3.21.0-2.el7.noarch
python2-keystoneauth1-3.17.3-1.el7.noarch
python2-keystone-16.0.1-1.el7.noarch
Could you please check and let know if any config needs to be changed?
Best Regards
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1931707
Title:
"NeutronAdminCredentialConfigurationInvalid: Networking client is
experiencing an unauthorized exception" error while instantiating
instance in Train RDO
Status in OpenStack Compute (nova):
New
Bug description:
Hi, Following ticket has been reported in https://bugzilla.redhat.com/show_bug.cgi?id=1964893
Based on their answer, verified the neutron command manually and it works fine.
The issue is occurring randomly and failing the instance creation
usecase.
Recently had failed 2 cases,
1) it failed once out of 72 runs
2021-06-09 10:00:49.391 3194 INFO heat.engine.resource [req-6d38438b-be89-4619-8517-9edc49bc7a40 - instTermProject-1 - default default] CREATE: Net "vm_network_3" [5776a857-4316-47f9-b3fa-3a1fbff39132] Stack "vapp-Service-060821210256-1" [c7fa51bb-2012-4f5c-aa6f-1720dd47a5ed]
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Traceback (most recent call last):
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 920, in _action_recorder
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource yield
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1033, in _do_action
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource step = next(subtask)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 982, in action_handler_task
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource done = check(handler_data)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/net.py", line 208, in check_create_complete
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource attributes = self._show_resource()
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/neutron.py", line 139, in _show_resource
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource res_info = client_method(*args)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 822, in show_network
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource return self.get(self.network_path % (network), params=_params)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource headers=headers, params=params)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource headers=headers, params=params)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource self._handle_fault_response(status_code, replybody, resp)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource exception_handler_v20(status_code, error_body)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource request_ids=request_ids)
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Neutron server returns request_ids: ['req-8eb8d67b-1b44-4caf-8c5a-37a442494b31']
2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource
2021-06-09 10:00:49.429 3194 INFO heat.engine.stack [req-6d38438b-be89-4619-8517-9edc49bc7a40 - instTermProject-1 - default default] Stack CREATE FAILED (vapp-Service-060821210256-1): Resource CREATE failed: Unauthorized: resources.vm_network_3: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
Neutron server returns request_ids: ['req-8eb8d67b-1b44-4caf-8c5a-37a442494b31']
2) it failed once out of 72 runs
2021-06-10 10:01:27.888 13501 INFO nova.compute.manager [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Took 8.84 seconds to spawn the instance on the hypervisor.
2021-06-10 10:01:28.034 13501 ERROR nova.network.neutronv2.api [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] Neutron client was not able to generate a valid admin token, please verify Neutron admin credential located in nova.conf: Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Failed to deallocate networks: NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Traceback (most recent call last):
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2746, in _cleanup_allocated_networks
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] self._deallocate_network(context, instance, requested_networks)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2002, in _deallocate_network
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] context, instance, requested_networks=requested_networks)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1570, in deallocate_for_instance
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] data = neutron.list_ports(**search_opts)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] **_params)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] for r in self._pagination(collection, path, **params):
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] res = self.get(path, params=params)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] headers=headers, params=params)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] headers=headers, params=params)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] self._handle_fault_response(status_code, replybody, resp)
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 137, in wrapper
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] raise exception.NeutronAdminCredentialConfigurationInvalid()
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]
2021-06-10 10:01:28.154 13501 INFO nova.scheduler.client.report [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] Deleted allocation for instance f0dc327b-1416-400f-9846-86b9ef0dfc71
I did tried with neutron auth config from nova.conf manually and it works fine.
No similar issue observed with Queens openstack version.
Also many errors related to rabbitmq
2021-06-11 03:28:26.028 26179 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2021-06-11 03:28:26.034 26158 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2021-06-11 03:28:26.056 26179 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
2021-06-11 03:28:26.057 26158 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
openstack-nova-novncproxy-20.6.0-1.el7.noarch
openstack-nova-api-20.6.0-1.el7.noarch
openstack-nova-common-20.6.0-1.el7.noarch
puppet-nova-15.7.0-1.el7.noarch
python2-nova-20.6.0-1.el7.noarch
openstack-nova-scheduler-20.6.0-1.el7.noarch
openstack-nova-conductor-20.6.0-1.el7.noarch
python2-novaclient-15.1.1-1.el7.noarch
openstack-neutron-common-15.3.2-1.el7.noarch
puppet-neutron-15.5.0-1.el7.noarch
python2-neutronclient-6.14.1-1.el7.noarch
openstack-neutron-metering-agent-15.3.2-1.el7.noarch
openstack-neutron-15.3.2-1.el7.noarch
openstack-neutron-openvswitch-15.3.2-1.el7.noarch
python2-neutron-lib-1.29.1-1.el7.noarch
openstack-neutron-ml2-15.3.2-1.el7.noarch
python2-neutron-15.3.2-1.el7.noarch
openstack-keystone-16.0.1-1.el7.noarch
python2-keystonemiddleware-7.0.1-2.el7.noarch
puppet-keystone-15.4.0-1.el7.noarch
python2-keystoneclient-3.21.0-2.el7.noarch
python2-keystoneauth1-3.17.3-1.el7.noarch
python2-keystone-16.0.1-1.el7.noarch
Could you please check and let know if any config needs to be changed?
Best Regards
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1931707/+subscriptions
Follow ups
