yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2012749] Re: Instance fails to initiate with <class 'amqp.exceptions.AccessRefused'> (HTTP 500)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Isa Fishman <2012749@xxxxxxxxxxxxxxxxxx>
Date: Mon, 27 Mar 2023 13:11:04 -0000
Reply-to: Bug 2012749 <2012749@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
This is an issue with the messaging service, make sure to configure it
correctly.

** Changed in: nova
     Assignee: (unassigned) => Isa Fishman (notisa)

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2012749

Title:
  Instance fails to initiate with <class
  'amqp.exceptions.AccessRefused'> (HTTP 500)

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Description: 
  Here's the log I was told to include.
  "
  2023-03-24 15:46:03.378 1889 INFO nova.osapi_compute.wsgi.server [req-e659e737-0124-4718-8578-b5c1637eb581 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1 HTTP/1.1" status: 200 len: 783 time: 0.0010614
  2023-03-24 15:46:03.422 1891 INFO nova.osapi_compute.wsgi.server [-] 10.116.0.3 "GET /v2.1 HTTP/1.1" status: 200 len: 783 time: 0.0015924
  2023-03-24 15:46:04.275 1891 INFO nova.osapi_compute.wsgi.server [req-b4845a8b-526b-4516-a0ab-72db01cb6a17 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1/flavors/detail HTTP/1.1" status: 200 len: 860 time: 0.5756984
  2023-03-24 15:46:04.283 1889 INFO nova.osapi_compute.wsgi.server [req-53f7f032-7d3e-41c3-b3c5-5359ed8d9de3 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1/os-availability-zone HTTP/1.1" status: 200 len: 495 time: 0.7366662
  2023-03-24 15:46:04.303 1891 INFO nova.osapi_compute.wsgi.server [req-d5a4c3ea-bcdc-4c0c-bf33-bd1aed4deff6 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1/flavors/9f4a158d-c81f-4e28-b9e0-aced44b0c7b7/os-extra_specs HTTP/1.1" status: 200 len: 417 time: 0.0216706
  2023-03-24 15:46:04.359 1892 INFO nova.osapi_compute.wsgi.server [req-b04cc473-774a-4f81-8ef8-c9156b68f2f6 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1/limits?reserved=1 HTTP/1.1" status: 200 len: 910 time: 0.8074837
  2023-03-24 15:46:04.587 1889 INFO nova.osapi_compute.wsgi.server [-] 10.116.0.3 "GET /v2.1 HTTP/1.1" status: 200 len: 783 time: 0.0016103
  2023-03-24 15:46:04.675 1889 WARNING oslo_db.sqlalchemy.utils [req-f308f20c-7deb-43e3-b545-f31932ffcac6 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] Unique keys not in sort_keys. The sorting order may be unstable.
  2023-03-24 15:46:04.680 1889 INFO nova.osapi_compute.wsgi.server [req-f308f20c-7deb-43e3-b545-f31932ffcac6 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1/os-keypairs HTTP/1.1" status: 200 len: 414 time: 0.5834584
  2023-03-24 15:46:05.056 1892 INFO nova.osapi_compute.wsgi.server [req-27886a59-07e2-4ba5-80f0-cf8b32e5d3c0 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1/servers/640e5601-460d-4ee9-81e7-d2c033ed4041 HTTP/1.1" status: 200 len: 2021 time: 1.0985825
  2023-03-24 15:46:05.359 1889 INFO nova.osapi_compute.wsgi.server [req-e659e737-0124-4718-8578-b5c1637eb581 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1 HTTP/1.1" status: 200 len: 783 time: 0.0013490
  2023-03-24 15:46:05.374 1891 INFO nova.osapi_compute.wsgi.server [req-bbe0eb79-33a2-45c5-b4b1-34574217f2c6 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1/os-server-groups HTTP/1.1" status: 200 len: 419 time: 0.3312027
  2023-03-24 15:46:11.048 1889 INFO nova.osapi_compute.wsgi.server [req-3739f405-f817-4ff1-8883-42710fb49dcb a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1/limits?reserved=1 HTTP/1.1" status: 200 len: 910 time: 0.2919271
  2023-03-24 15:46:11.051 1889 INFO nova.osapi_compute.wsgi.server [req-3739f405-f817-4ff1-8883-42710fb49dcb a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1 HTTP/1.1" status: 200 len: 783 time: 0.0014436
  2023-03-24 15:46:11.511 1890 INFO nova.osapi_compute.wsgi.server [req-f4996eb9-133a-488f-8461-2c23385deca3 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1/servers/640e5601-460d-4ee9-81e7-d2c033ed4041 HTTP/1.1" status: 200 len: 2021 time: 0.2696385
  2023-03-24 15:46:11.741 1889 INFO nova.osapi_compute.wsgi.server [req-3739f405-f817-4ff1-8883-42710fb49dcb a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1 HTTP/1.1" status: 200 len: 783 time: 0.0009189
  2023-03-24 15:46:18.815 1890 INFO nova.osapi_compute.wsgi.server [-] 10.116.0.3 "GET /v2.1 HTTP/1.1" status: 200 len: 783 time: 0.0011003
  2023-03-24 15:46:19.177 1890 WARNING oslo_config.cfg [req-b6e5cc67-e09b-4aaf-804a-2beade3a2174 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] Deprecated: Option "api_servers" from group "glance" is deprecated for removal (
  Support for image service configuration via standard keystoneauth1 Adapter
  options was added in the 17.0.0 Queens release. The api_servers option was
  retained temporarily to allow consumers time to cut over to a real load
  balancing solution.
  ).  Its value may be silently ignored in the future.
  2023-03-24 15:46:19.840 1891 INFO nova.osapi_compute.wsgi.server [req-4ce507fd-ff77-4611-8b00-3d2b1cb18a48 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1/limits?reserved=1 HTTP/1.1" status: 200 len: 910 time: 0.2121718
  2023-03-24 15:46:19.849 1891 INFO nova.osapi_compute.wsgi.server [req-4ce507fd-ff77-4611-8b00-3d2b1cb18a48 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] 10.116.0.3 "GET /v2.1 HTTP/1.1" status: 200 len: 783 time: 0.0012639
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi [req-b6e5cc67-e09b-4aaf-804a-2beade3a2174 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] Unexpected exception in API method: amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/api/openstack/wsgi.py", line 658, in wrapped
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/api/validation/__init__.py", line 110, in wrapper
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   [Previous line repeated 10 more times]
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/api/openstack/compute/servers.py", line 784, in create
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     instances, resv_id = self.compute_api.create(
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/compute/api.py", line 2149, in create
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return self._create_instance(
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/compute/api.py", line 1697, in _create_instance
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     self.compute_task_api.schedule_and_build_instances(
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/conductor/api.py", line 134, in schedule_and_build_instances
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     self.conductor_compute_rpcapi.schedule_and_build_instances(
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/nova/conductor/rpcapi.py", line 413, in schedule_and_build_instances
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     cctxt.cast(context, 'schedule_and_build_instances', **kw)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 155, in cast
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     self.transport._send(self.target, msg_ctxt, msg,
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return self._driver.send(target, ctxt, message,
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return self._send(target, ctxt, message, wait_for_reply, timeout,
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 653, in _send
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     with self._get_connection(rpc_common.PURPOSE_SEND, retry) as conn:
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 605, in _get_connection
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return rpc_common.ConnectionContext(self._connection_pool,
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/common.py", line 423, in __init__
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     self.connection = connection_pool.get(retry=retry)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/pool.py", line 98, in get
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return self.create(retry=retry)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/pool.py", line 135, in create
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return self.connection_cls(self.conf, self.url, purpose, retry=retry)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 648, in __init__
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     self.ensure_connection()
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 771, in ensure_connection
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     self.connection.ensure_connection(
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/kombu/connection.py", line 382, in ensure_connection
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     self._ensure_connection(*args, **kwargs)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/kombu/connection.py", line 434, in _ensure_connection
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return retry_over_time(
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/kombu/utils/functional.py", line 312, in retry_over_time
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return fun(*args, **kwargs)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/kombu/connection.py", line 878, in _connection_factory
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     self._connection = self._establish_connection()
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/kombu/connection.py", line 813, in _establish_connection
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     conn = self.transport.establish_connection()
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     conn.connect()
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/amqp/connection.py", line 329, in connect
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     self.drain_events(timeout=self.connect_timeout)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/amqp/connection.py", line 522, in drain_events
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     while not self.blocking_read(timeout):
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/amqp/connection.py", line 528, in blocking_read
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return self.on_inbound_frame(frame)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/amqp/method_framing.py", line 53, in on_frame
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     callback(channel, method_sig, buf, None)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/amqp/connection.py", line 534, in on_inbound_method
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     return self.channels[channel_id].dispatch_method(
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     listener(*args)
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3/dist-packages/amqp/connection.py", line 664, in _on_close
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi     raise error_for_code(reply_code, reply_text,
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
  2023-03-24 15:46:20.230 1890 ERROR nova.api.openstack.wsgi 
  2023-03-24 15:46:20.236 1890 INFO nova.api.openstack.wsgi [req-b6e5cc67-e09b-4aaf-804a-2beade3a2174 a829c22d15af4e919b25a6b8f49f6d35 629b0682bf9f4e878930b936f19b6ade - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'amqp.exceptions.AccessRefused'>
  "
  I used the recommend Ubuntu install guide, setup everything according to the guide, and then tried to initialize an instance, that's when it errored out.

  
  I'm running 2 servers, One is only running compute and the other one is running identity and such. Here is what the identity server is running:

  ii  nova-api                             3:25.1.0-0ubuntu1~cloud0                             all          OpenStack Compute - API frontend
  ii  nova-common                          3:25.1.0-0ubuntu1~cloud0                             all          OpenStack Compute - common files
  ii  nova-conductor                       3:25.1.0-0ubuntu1~cloud0                             all          OpenStack Compute - conductor service
  ii  nova-novncproxy                      3:25.1.0-0ubuntu1~cloud0                             all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                       3:25.1.0-0ubuntu1~cloud0                             all          OpenStack Compute - virtual machine scheduler
  ii  python3-nova                         3:25.1.0-0ubuntu1~cloud0                             all          OpenStack Compute Python 3 libraries
  ii  python3-novaclient                   2:17.6.0-0ubuntu1~cloud0                             all          client library for OpenStack Compute API - 3.x

  I'm just trying to configure this in the cloud before i do it at my
  home lab.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2012749/+subscriptions
References

[Bug 2012749] [NEW] Instance fails to initiate with <class 'amqp.exceptions.AccessRefused'> (HTTP 500)
From: Isa Fishman, 2023-03-24