yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1906483] [NEW] MessagingTimeout: Timed out waiting for a reply to message

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: men <1906483@xxxxxxxxxxxxxxxxxx>
Date: Wed, 02 Dec 2020 08:10:17 -0000
Reply-to: Bug 1906483 <1906483@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

openstack-q
nova  --debug get-vnc-console  450e3ab5-ee30-47a3-910c-f990c1528748  novnc


DEBUG (connectionpool:461) http://controller:8774 "POST /v2.1/servers/450e3ab5-ee30-47a3-910c-f990c1528748/remote-consoles HTTP/1.1" 500 216
DEBUG (session:548) RESP: [500] Connection: keep-alive Content-Length: 216 Content-Type: application/json; charset=UTF-8 Date: Wed, 02 Dec 2020 08:45:05 GMT Openstack-Api-Version: compute 2.60 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-60c91d85-3a68-4604-b985-29e7f9b4fd77 X-Openstack-Nova-Api-Version: 2.60 X-Openstack-Request-Id: req-60c91d85-3a68-4604-b985-29e7f9b4fd77
DEBUG (session:580) RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'oslo_messaging.exceptions.MessagingTimeout'>", "code": 500}}
DEBUG (session:943) POST call to compute for http://controller:8774/v2.1/servers/450e3ab5-ee30-47a3-910c-f990c1528748/remote-consoles used request id req-60c91d85-3a68-4604-b985-29e7f9b4fd77
DEBUG (shell:793) Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-60c91d85-3a68-4604-b985-29e7f9b4fd77)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 791, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 713, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 2501, in do_get_vnc_console
    data = server.get_vnc_console(args.console_type)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 107, in get_vnc_console
    return self.manager.get_vnc_console(self, console_type)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 872, in get_vnc_console
    return self.get_console_url(server, console_type)
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 393, in substitution
    return methods[-1].func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 960, in get_console_url
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 392, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-60c91d85-3a68-4604-b985-29e7f9b4fd77)
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-60c91d85-3a68-4604-b985-29e7f9b4fd77)



nova-api.log
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi [req-60c91d85-3a68-4604-b985-29e7f9b4fd77 f1c06c2e1b054888b6b2196262b86a55 ab1ab9dd58564456bc60557bf8def817 - default default] Unexpected exception in API method: MessagingTimeout: Timed out waiting for a reply to message ID 905fc41b16c943d091fe1ad13a61703f
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 788, in wrapped
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 108, in wrapper
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 108, in wrapper
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/remote_consoles.py", line 168, in create
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     output = handler(context, instance, console_type)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 196, in wrapped
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     return function(self, context, instance, *args, **kwargs)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 186, in inner
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3737, in get_vnc_console
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     access_url=connect_info['access_url'])
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/consoleauth/rpcapi.py", line 93, in authorize_console
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     return cctxt.call(ctxt, 'authorize_console', **msg_args)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     retry=self.retry)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     timeout=timeout, retry=retry)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 625, in send
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     retry=retry)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 614, in _send
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     result = self._waiter.wait(msg_id, timeout)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 506, in wait
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     message = self.waiters.get(msg_id, timeout=timeout)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 394, in get
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     'to message ID %s' % msg_id)
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi MessagingTimeout: Timed out waiting for a reply to message ID 905fc41b16c943d091fe1ad13a61703f
2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi
2020-12-02 16:45:05.797 1294 INFO nova.api.openstack.wsgi [req-60c91d85-3a68-4604-b985-29e7f9b4fd77 f1c06c2e1b054888b6b2196262b86a55 ab1ab9dd58564456bc60557bf8def817 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2020-12-02 16:45:05.798 1294 INFO nova.osapi_compute.wsgi.server [req-60c91d85-3a68-4604-b985-29e7f9b4fd77 f1c06c2e1b054888b6b2196262b86a55 ab1ab9dd58564456bc60557bf8def817 - default default] 172.16.21.100 "POST /v2.1/servers/450e3ab5-ee30-47a3-910c-f990c1528748/remote-consoles HTTP/1.1" status: 500 len: 651 time: 60.1551988


There is no problem in creating the machine. Getting VNC failed

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1906483

Title:
  MessagingTimeout: Timed out waiting for a reply to message

Status in OpenStack Compute (nova):
  New

Bug description:
  openstack-q
  nova  --debug get-vnc-console  450e3ab5-ee30-47a3-910c-f990c1528748  novnc


  DEBUG (connectionpool:461) http://controller:8774 "POST /v2.1/servers/450e3ab5-ee30-47a3-910c-f990c1528748/remote-consoles HTTP/1.1" 500 216
  DEBUG (session:548) RESP: [500] Connection: keep-alive Content-Length: 216 Content-Type: application/json; charset=UTF-8 Date: Wed, 02 Dec 2020 08:45:05 GMT Openstack-Api-Version: compute 2.60 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-60c91d85-3a68-4604-b985-29e7f9b4fd77 X-Openstack-Nova-Api-Version: 2.60 X-Openstack-Request-Id: req-60c91d85-3a68-4604-b985-29e7f9b4fd77
  DEBUG (session:580) RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'oslo_messaging.exceptions.MessagingTimeout'>", "code": 500}}
  DEBUG (session:943) POST call to compute for http://controller:8774/v2.1/servers/450e3ab5-ee30-47a3-910c-f990c1528748/remote-consoles used request id req-60c91d85-3a68-4604-b985-29e7f9b4fd77
  DEBUG (shell:793) Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-60c91d85-3a68-4604-b985-29e7f9b4fd77)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 791, in main
      OpenStackComputeShell().main(argv)
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 713, in main
      args.func(self.cs, args)
    File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 2501, in do_get_vnc_console
      data = server.get_vnc_console(args.console_type)
    File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 107, in get_vnc_console
      return self.manager.get_vnc_console(self, console_type)
    File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 872, in get_vnc_console
      return self.get_console_url(server, console_type)
    File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 393, in substitution
      return methods[-1].func(obj, *args, **kwargs)
    File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 960, in get_console_url
      resp, body = self.api.client.post(url, body=body)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 392, in post
      return self.request(url, 'POST', **kwargs)
    File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
      raise exceptions.from_response(resp, body, url, method)
  ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-60c91d85-3a68-4604-b985-29e7f9b4fd77)
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-60c91d85-3a68-4604-b985-29e7f9b4fd77)



  
  nova-api.log
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi [req-60c91d85-3a68-4604-b985-29e7f9b4fd77 f1c06c2e1b054888b6b2196262b86a55 ab1ab9dd58564456bc60557bf8def817 - default default] Unexpected exception in API method: MessagingTimeout: Timed out waiting for a reply to message ID 905fc41b16c943d091fe1ad13a61703f
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 788, in wrapped
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 108, in wrapper
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 108, in wrapper
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/remote_consoles.py", line 168, in create
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     output = handler(context, instance, console_type)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 196, in wrapped
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     return function(self, context, instance, *args, **kwargs)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 186, in inner
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3737, in get_vnc_console
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     access_url=connect_info['access_url'])
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/consoleauth/rpcapi.py", line 93, in authorize_console
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     return cctxt.call(ctxt, 'authorize_console', **msg_args)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     retry=self.retry)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     timeout=timeout, retry=retry)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 625, in send
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     retry=retry)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 614, in _send
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     result = self._waiter.wait(msg_id, timeout)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 506, in wait
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     message = self.waiters.get(msg_id, timeout=timeout)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 394, in get
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi     'to message ID %s' % msg_id)
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi MessagingTimeout: Timed out waiting for a reply to message ID 905fc41b16c943d091fe1ad13a61703f
  2020-12-02 16:45:05.793 1294 ERROR nova.api.openstack.wsgi
  2020-12-02 16:45:05.797 1294 INFO nova.api.openstack.wsgi [req-60c91d85-3a68-4604-b985-29e7f9b4fd77 f1c06c2e1b054888b6b2196262b86a55 ab1ab9dd58564456bc60557bf8def817 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  2020-12-02 16:45:05.798 1294 INFO nova.osapi_compute.wsgi.server [req-60c91d85-3a68-4604-b985-29e7f9b4fd77 f1c06c2e1b054888b6b2196262b86a55 ab1ab9dd58564456bc60557bf8def817 - default default] 172.16.21.100 "POST /v2.1/servers/450e3ab5-ee30-47a3-910c-f990c1528748/remote-consoles HTTP/1.1" status: 500 len: 651 time: 60.1551988

  
  There is no problem in creating the machine. Getting VNC failed

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1906483/+subscriptions
Follow ups

[Bug 1906483] Re: MessagingTimeout: Timed out waiting for a reply to message
From: men, 2020-12-03