yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1835637] [NEW] (404) NOT_FOUND - failed to perform operation on queue 'notifications.info' in vhost '/nova' due to timeout

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Satish Patel <satish.txt@xxxxxxxxx>
Date: Sun, 07 Jul 2019 00:04:52 -0000
Reply-to: Bug 1835637 <1835637@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Running stein version / deployed by openstack-ansible

============
[root@ostack-osa ~]# openstack keypair create --public-key ~/.ssh/id_rsa.pub magnum-key-bar -vvvv
Starting new HTTP connection (1): 172.28.8.9:8774
http://172.28.8.9:8774 "POST /v2.1/os-keypairs HTTP/1.1" 504 None
RESP: [504] Cache-Control: no-cache Connection: close Content-Type: text/html
RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged.
Unknown Error (HTTP 504)
clean_up CreateKeypair: Unknown Error (HTTP 504)
END return value: 1
=============


I have verified haproxy load-balancer working fine and nova-api
listening on 8774, when i checked logs on nova-api i found following
error


2019-07-06 19:35:31.200 56 ERROR oslo.messaging._drivers.impl_rabbit [req-242de4d8-9568-40d5-b4db-52d83a8d244f 63847837de444225accd1ae1db2b1f11 6297c04e9593466d9c6747874e379444 - default default] Failed to publish message to topic 'nova': Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.error' in vhost '/nova' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.error' in vhost '/nova' due to timeout
2019-07-06 19:35:31.202 56 ERROR oslo.messaging._drivers.impl_rabbit [req-242de4d8-9568-40d5-b4db-52d83a8d244f 63847837de444225accd1ae1db2b1f11 6297c04e9593466d9c6747874e379444 - default default] Unable to connect to AMQP server on 172.28.15.248:5671 after inf tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.error' in vhost '/nova' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.error' in vhost '/nova' due to timeout
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging [req-242de4d8-9568-40d5-b4db-52d83a8d244f 63847837de444225accd1ae1db2b1f11 6297c04e9593466d9c6747874e379444 - default default] Could not send notification to versioned_notifications. Payload={'_context_domain': None, '_context_request_id': 'req-242de4d8-9568-40d5-b4db-52d83a8d244f', '_context_global_request_id': None, '_context_quota_class': None, 'event_type': u'compute.exception', '_context_service_catalog': [{u'endpoints': [{u'adminURL': u'http://172.28.8.9:9311', u'region': u'RegionOne', u'internalURL': u'http://172.28.8.9:9311', u'publicURL': u'https://10.30.8.9:9311'}], u'type': u'key-manager', u'name': u'barbican'}, {u'endpoints': [{u'adminURL': u'http://172.28.8.9:8776/v3/6297c04e9593466d9c6747874e379444', u'region': u'RegionOne', u'internalURL': u'http://172.28.8.9:8776/v3/6297c04e9593466d9c6747874e379444', u'publicURL': u'https://10.30.8.9:8776/v3/6297c04e9593466d9c6747874e379444'}], u'type': u'volumev3', u'name': u'cinderv3'}, {u'endpoints': [{u'adminURL': u'http://172.28.8.9:8780', u'region': u'RegionOne', u'internalURL': u'http://172.28.8.9:8780', u'publicURL': u'https://10.30.8.9:8780'}], u'type': u'placement', u'name': u'placement'}, {u'endpoints': [{u'adminURL': u'http://172.28.8.9:9696', u'region': u'RegionOne', u'internalURL': u'http://172.28.8.9:9696', u'publicURL': u'https://10.30.8.9:9696'}], u'type': u'network', u'name': u'neutron'}, {u'endpoints': [{u'adminURL': u'http://172.28.8.9:9292', u'region': u'RegionOne', u'internalURL': u'http://172.28.8.9:9292', u'publicURL': u'https://10.30.8.9:9292'}], u'type': u'image', u'name': u'glance'}], 'timestamp': u'2019-07-06 23:34:28.146951', '_context_user': u'63847837de444225accd1ae1db2b1f11', '_unique_id': '36959759468747838bcf2cd94602da0a', '_context_resource_uuid': None, '_context_is_admin_project': True, '_context_read_deleted': 'no', '_context_user_id': u'63847837de444225accd1ae1db2b1f11', 'payload': {'nova_object.version': '1.1', 'nova_object.name': 'ExceptionPayload', 'nova_object.namespace': 'nova', 'nova_object.data': {'module_name': u'nova.objects.keypair', 'exception': u'KeyPairExists', 'traceback': u'Traceback (most recent call last):\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped\n    return f(self, context, *args, **kw)\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/api.py", line 5841, in import_key_pair\n    keypair.create()\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper\n    return fn(self, *args, **kwargs)\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/objects/keypair.py", line 173, in create\n    self._create()\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/objects/keypair.py", line 177, in _create\n    db_keypair = self._create_in_db(self._context, updates)\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/objects/keypair.py", line 136, in _create_in_db\n    return _create_in_db(context, values)\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1012, in wrapper\n    return fn(*args, **kwargs)\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/objects/keypair.py", line 73, in _create_in_db\n    raise exception.KeyPairExists(key_name=values[\'name\'])\nKeyPairExists: Key pair \'magnum-key-foo\' already exists.\n', 'exception_message': u"Key pair 'magnum-key-foo' already exists.", 'function_name': u'_create_in_db'}}, '_context_project_name': u'admin', '_context_system_scope': None, '_context_user_identity': u'63847837de444225accd1ae1db2b1f11 6297c04e9593466d9c6747874e379444 - default default', '_context_auth_token': 'gAAAAABdIS9H21rHUX7kQ4xW-2vUVbFX3f8PQPQwyuY-TahQRxthQQg5vWDLQSR70eiz3iZi7u2IYLLFLz3_S3k0-il-vqcBoWs4kOiBwHofPs1CUyMJAO_zhHE1QtnlNMto0vXZ4UyYFjuXJOY3e7NibpQLL26qIJbZehbhn1s02ZwKBetSVm8', '_context_show_deleted': False, '_context_tenant': u'6297c04e9593466d9c6747874e379444', '_context_roles': [u'admin', u'reader', u'member'], 'priority': 'ERROR', '_context_read_only': False, '_context_is_admin': True, '_context_project_id': u'6297c04e9593466d9c6747874e379444', '_context_project_domain': u'default', '_context_timestamp': '2019-07-06T23:31:20.611554', '_context_user_domain': u'default', '_context_user_name': u'admin', 'publisher_id': u'nova-api:ostack-infra-2-1-nova-api-container-86de0dda', 'message_id': u'e2727710-d0fa-47b5-a136-ed01a8f10b5d', '_context_project': u'6297c04e9593466d9c6747874e379444', '_context_remote_address': '172.28.8.7'}: MessageDeliveryFailure: Unable to connect to AMQP server on 172.28.15.248:5671 after inf tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.error' in vhost '/nova' due to timeout
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/notify/messaging.py", line 70, in notify
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     retry=retry)
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/transport.py", line 135, in _send_notification
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     retry=retry)
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 649, in send_notification
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     envelope=(version == 2.0), notify=True, retry=retry)
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 613, in _send
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     conn.notify_send(exchange, target.topic, msg, retry=retry)
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1230, in notify_send
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     exchange, msg, routing_key=topic, retry=retry)
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1086, in _ensure_publishing
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     self.ensure(method, retry=retry, error_callback=_error_callback)
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 755, in ensure
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     raise exceptions.MessageDeliveryFailure(msg)
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging MessageDeliveryFailure: Unable to connect to AMQP server on 172.28.15.248:5671 after inf tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.error' in vhost '/nova' due to timeout
2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1835637

Title:
  (404) NOT_FOUND - failed to perform operation on queue
  'notifications.info' in vhost '/nova' due to timeout

Status in OpenStack Compute (nova):
  New

Bug description:
  Running stein version / deployed by openstack-ansible

  ============
  [root@ostack-osa ~]# openstack keypair create --public-key ~/.ssh/id_rsa.pub magnum-key-bar -vvvv
  Starting new HTTP connection (1): 172.28.8.9:8774
  http://172.28.8.9:8774 "POST /v2.1/os-keypairs HTTP/1.1" 504 None
  RESP: [504] Cache-Control: no-cache Connection: close Content-Type: text/html
  RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged.
  Unknown Error (HTTP 504)
  clean_up CreateKeypair: Unknown Error (HTTP 504)
  END return value: 1
  =============


  I have verified haproxy load-balancer working fine and nova-api
  listening on 8774, when i checked logs on nova-api i found following
  error


  2019-07-06 19:35:31.200 56 ERROR oslo.messaging._drivers.impl_rabbit [req-242de4d8-9568-40d5-b4db-52d83a8d244f 63847837de444225accd1ae1db2b1f11 6297c04e9593466d9c6747874e379444 - default default] Failed to publish message to topic 'nova': Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.error' in vhost '/nova' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.error' in vhost '/nova' due to timeout
  2019-07-06 19:35:31.202 56 ERROR oslo.messaging._drivers.impl_rabbit [req-242de4d8-9568-40d5-b4db-52d83a8d244f 63847837de444225accd1ae1db2b1f11 6297c04e9593466d9c6747874e379444 - default default] Unable to connect to AMQP server on 172.28.15.248:5671 after inf tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.error' in vhost '/nova' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.error' in vhost '/nova' due to timeout
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging [req-242de4d8-9568-40d5-b4db-52d83a8d244f 63847837de444225accd1ae1db2b1f11 6297c04e9593466d9c6747874e379444 - default default] Could not send notification to versioned_notifications. Payload={'_context_domain': None, '_context_request_id': 'req-242de4d8-9568-40d5-b4db-52d83a8d244f', '_context_global_request_id': None, '_context_quota_class': None, 'event_type': u'compute.exception', '_context_service_catalog': [{u'endpoints': [{u'adminURL': u'http://172.28.8.9:9311', u'region': u'RegionOne', u'internalURL': u'http://172.28.8.9:9311', u'publicURL': u'https://10.30.8.9:9311'}], u'type': u'key-manager', u'name': u'barbican'}, {u'endpoints': [{u'adminURL': u'http://172.28.8.9:8776/v3/6297c04e9593466d9c6747874e379444', u'region': u'RegionOne', u'internalURL': u'http://172.28.8.9:8776/v3/6297c04e9593466d9c6747874e379444', u'publicURL': u'https://10.30.8.9:8776/v3/6297c04e9593466d9c6747874e379444'}], u'type': u'volumev3', u'name': u'cinderv3'}, {u'endpoints': [{u'adminURL': u'http://172.28.8.9:8780', u'region': u'RegionOne', u'internalURL': u'http://172.28.8.9:8780', u'publicURL': u'https://10.30.8.9:8780'}], u'type': u'placement', u'name': u'placement'}, {u'endpoints': [{u'adminURL': u'http://172.28.8.9:9696', u'region': u'RegionOne', u'internalURL': u'http://172.28.8.9:9696', u'publicURL': u'https://10.30.8.9:9696'}], u'type': u'network', u'name': u'neutron'}, {u'endpoints': [{u'adminURL': u'http://172.28.8.9:9292', u'region': u'RegionOne', u'internalURL': u'http://172.28.8.9:9292', u'publicURL': u'https://10.30.8.9:9292'}], u'type': u'image', u'name': u'glance'}], 'timestamp': u'2019-07-06 23:34:28.146951', '_context_user': u'63847837de444225accd1ae1db2b1f11', '_unique_id': '36959759468747838bcf2cd94602da0a', '_context_resource_uuid': None, '_context_is_admin_project': True, '_context_read_deleted': 'no', '_context_user_id': u'63847837de444225accd1ae1db2b1f11', 'payload': {'nova_object.version': '1.1', 'nova_object.name': 'ExceptionPayload', 'nova_object.namespace': 'nova', 'nova_object.data': {'module_name': u'nova.objects.keypair', 'exception': u'KeyPairExists', 'traceback': u'Traceback (most recent call last):\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped\n    return f(self, context, *args, **kw)\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/compute/api.py", line 5841, in import_key_pair\n    keypair.create()\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper\n    return fn(self, *args, **kwargs)\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/objects/keypair.py", line 173, in create\n    self._create()\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/objects/keypair.py", line 177, in _create\n    db_keypair = self._create_in_db(self._context, updates)\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/objects/keypair.py", line 136, in _create_in_db\n    return _create_in_db(context, values)\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1012, in wrapper\n    return fn(*args, **kwargs)\n  File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/nova/objects/keypair.py", line 73, in _create_in_db\n    raise exception.KeyPairExists(key_name=values[\'name\'])\nKeyPairExists: Key pair \'magnum-key-foo\' already exists.\n', 'exception_message': u"Key pair 'magnum-key-foo' already exists.", 'function_name': u'_create_in_db'}}, '_context_project_name': u'admin', '_context_system_scope': None, '_context_user_identity': u'63847837de444225accd1ae1db2b1f11 6297c04e9593466d9c6747874e379444 - default default', '_context_auth_token': 'gAAAAABdIS9H21rHUX7kQ4xW-2vUVbFX3f8PQPQwyuY-TahQRxthQQg5vWDLQSR70eiz3iZi7u2IYLLFLz3_S3k0-il-vqcBoWs4kOiBwHofPs1CUyMJAO_zhHE1QtnlNMto0vXZ4UyYFjuXJOY3e7NibpQLL26qIJbZehbhn1s02ZwKBetSVm8', '_context_show_deleted': False, '_context_tenant': u'6297c04e9593466d9c6747874e379444', '_context_roles': [u'admin', u'reader', u'member'], 'priority': 'ERROR', '_context_read_only': False, '_context_is_admin': True, '_context_project_id': u'6297c04e9593466d9c6747874e379444', '_context_project_domain': u'default', '_context_timestamp': '2019-07-06T23:31:20.611554', '_context_user_domain': u'default', '_context_user_name': u'admin', 'publisher_id': u'nova-api:ostack-infra-2-1-nova-api-container-86de0dda', 'message_id': u'e2727710-d0fa-47b5-a136-ed01a8f10b5d', '_context_project': u'6297c04e9593466d9c6747874e379444', '_context_remote_address': '172.28.8.7'}: MessageDeliveryFailure: Unable to connect to AMQP server on 172.28.15.248:5671 after inf tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.error' in vhost '/nova' due to timeout
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/notify/messaging.py", line 70, in notify
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     retry=retry)
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/transport.py", line 135, in _send_notification
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     retry=retry)
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 649, in send_notification
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     envelope=(version == 2.0), notify=True, retry=retry)
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 613, in _send
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     conn.notify_send(exchange, target.topic, msg, retry=retry)
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1230, in notify_send
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     exchange, msg, routing_key=topic, retry=retry)
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1086, in _ensure_publishing
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     self.ensure(method, retry=retry, error_callback=_error_callback)
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging   File "/openstack/venvs/nova-19.0.0.0rc3.dev6/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 755, in ensure
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging     raise exceptions.MessageDeliveryFailure(msg)
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging MessageDeliveryFailure: Unable to connect to AMQP server on 172.28.15.248:5671 after inf tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'versioned_notifications.error' in vhost '/nova' due to timeout
  2019-07-06 19:35:31.204 56 ERROR oslo_messaging.notify.messaging

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1835637/+subscriptions