yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1931707] Re: "NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception" error while instantiating instance in Train RDO

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Artom Lifshitz <1931707@xxxxxxxxxxxxxxxxxx>
Date: Mon, 09 May 2022 18:37:37 -0000
Reply-to: Bug 1931707 <1931707@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Hello, thanks for the bug report!

I think given how inconsistently reproducible the issue was, we can
blame this on an environmental issue as opposed to a Nova bug. I'm not
sure there's anything further to be done at this time.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1931707

Title:
  "NeutronAdminCredentialConfigurationInvalid: Networking client is
  experiencing an unauthorized exception" error while instantiating
  instance in Train RDO

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Hi, Following ticket has been reported in https://bugzilla.redhat.com/show_bug.cgi?id=1964893
  Based on their answer, verified the neutron command manually and it works fine.

  The issue is occurring randomly and failing the instance creation
  usecase.

  Recently had failed 2 cases,

  1) it failed once out of 72 runs

  2021-06-09 10:00:49.391 3194 INFO heat.engine.resource [req-6d38438b-be89-4619-8517-9edc49bc7a40 - instTermProject-1 - default default] CREATE: Net "vm_network_3" [5776a857-4316-47f9-b3fa-3a1fbff39132] Stack "vapp-Service-060821210256-1" [c7fa51bb-2012-4f5c-aa6f-1720dd47a5ed]
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Traceback (most recent call last):
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 920, in _action_recorder
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource     yield
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1033, in _do_action
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource     step = next(subtask)
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 982, in action_handler_task
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource     done = check(handler_data)
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/net.py", line 208, in check_create_complete
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource     attributes = self._show_resource()
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/neutron.py", line 139, in _show_resource
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource     res_info = client_method(*args)
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 822, in show_network
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource     return self.get(self.network_path % (network), params=_params)
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource     headers=headers, params=params)
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource     headers=headers, params=params)
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource     request_ids=request_ids)
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Neutron server returns request_ids: ['req-8eb8d67b-1b44-4caf-8c5a-37a442494b31']
  2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource
  2021-06-09 10:00:49.429 3194 INFO heat.engine.stack [req-6d38438b-be89-4619-8517-9edc49bc7a40 - instTermProject-1 - default default] Stack CREATE FAILED (vapp-Service-060821210256-1): Resource CREATE failed: Unauthorized: resources.vm_network_3: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
  Neutron server returns request_ids: ['req-8eb8d67b-1b44-4caf-8c5a-37a442494b31']


  2) it failed once out of 72 runs

  2021-06-10 10:01:27.888 13501 INFO nova.compute.manager [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Took 8.84 seconds to spawn the instance on the hypervisor.
  2021-06-10 10:01:28.034 13501 ERROR nova.network.neutronv2.api [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] Neutron client was not able to generate a valid admin token, please verify Neutron admin credential located in nova.conf: Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Failed to deallocate networks: NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Traceback (most recent call last):
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2746, in _cleanup_allocated_networks
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     self._deallocate_network(context, instance, requested_networks)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2002, in _deallocate_network
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     context, instance, requested_networks=requested_networks)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1570, in deallocate_for_instance
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     data = neutron.list_ports(**search_opts)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     ret = obj(*args, **kwargs)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     **_params)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     ret = obj(*args, **kwargs)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     for r in self._pagination(collection, path, **params):
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     res = self.get(path, params=params)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     ret = obj(*args, **kwargs)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     headers=headers, params=params)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     ret = obj(*args, **kwargs)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     headers=headers, params=params)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     ret = obj(*args, **kwargs)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     self._handle_fault_response(status_code, replybody, resp)
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 137, in wrapper
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]     raise exception.NeutronAdminCredentialConfigurationInvalid()
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
  2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71]
  2021-06-10 10:01:28.154 13501 INFO nova.scheduler.client.report [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] Deleted allocation for instance f0dc327b-1416-400f-9846-86b9ef0dfc71

  
  I did tried with neutron auth config from nova.conf manually and it works fine.

  No similar issue observed with Queens openstack version.

  Also many errors related to rabbitmq

  2021-06-11 03:28:26.028 26179 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
  2021-06-11 03:28:26.034 26158 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
  2021-06-11 03:28:26.056 26179 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
  2021-06-11 03:28:26.057 26158 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer


  openstack-nova-novncproxy-20.6.0-1.el7.noarch
  openstack-nova-api-20.6.0-1.el7.noarch
  openstack-nova-common-20.6.0-1.el7.noarch
  puppet-nova-15.7.0-1.el7.noarch
  python2-nova-20.6.0-1.el7.noarch
  openstack-nova-scheduler-20.6.0-1.el7.noarch
  openstack-nova-conductor-20.6.0-1.el7.noarch
  python2-novaclient-15.1.1-1.el7.noarch

  openstack-neutron-common-15.3.2-1.el7.noarch
  puppet-neutron-15.5.0-1.el7.noarch
  python2-neutronclient-6.14.1-1.el7.noarch
  openstack-neutron-metering-agent-15.3.2-1.el7.noarch
  openstack-neutron-15.3.2-1.el7.noarch
  openstack-neutron-openvswitch-15.3.2-1.el7.noarch
  python2-neutron-lib-1.29.1-1.el7.noarch
  openstack-neutron-ml2-15.3.2-1.el7.noarch
  python2-neutron-15.3.2-1.el7.noarch

  openstack-keystone-16.0.1-1.el7.noarch
  python2-keystonemiddleware-7.0.1-2.el7.noarch
  puppet-keystone-15.4.0-1.el7.noarch
  python2-keystoneclient-3.21.0-2.el7.noarch
  python2-keystoneauth1-3.17.3-1.el7.noarch
  python2-keystone-16.0.1-1.el7.noarch


  Could you please check and let know if any config needs to be changed?

  Best Regards

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1931707/+subscriptions
References

[Bug 1931707] [NEW] "NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception" error while instantiating instance in Train RDO
From: startlearningnew, 2021-06-11