yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1777382] [NEW] class 'oslo_messaging.exceptions.MessagingTimeout'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: deb <1777382@xxxxxxxxxxxxxxxxxx>
Date: Mon, 18 Jun 2018 04:12:16 -0000
Reply-to: Bug 1777382 <1777382@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

/var/log/nova/nova-api.log is as follows

2018-06-18 08:39:14.619 22686 INFO nova.osapi_compute.wsgi.server [req-dca907e2-f894-4bee-953e-2701dbb1c20c 508b406d37c34ac499f41e29a81a088d d54b97ab58014004be21ea398033a716 - default default] 192.168.133.35 "GET /v2.1/d54b97ab58014004be21ea398033a716/flavors/small HTTP/1.1" status: 404 len: 432 time: 0.2995820
2018-06-18 08:39:14.949 22686 INFO nova.osapi_compute.wsgi.server [req-3500f964-c761-46a3-baef-bd15e5542272 508b406d37c34ac499f41e29a81a088d d54b97ab58014004be21ea398033a716 - default default] 192.168.133.35 "GET /v2.1/d54b97ab58014004be21ea398033a716/flavors HTTP/1.1" status: 200 len: 689 time: 0.3250749
2018-06-18 08:39:15.329 22686 INFO nova.osapi_compute.wsgi.server [req-084d5aec-6c16-401a-97f9-48322d9ae2f7 508b406d37c34ac499f41e29a81a088d d54b97ab58014004be21ea398033a716 - default default] 192.168.133.35 "GET /v2.1/d54b97ab58014004be21ea398033a716/flavors/49a9f23e-3f31-4005-9da5-b4b523a1c736 HTTP/1.1" status: 200 len: 857 time: 0.3730719
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions [req-85af8d4d-b517-4d7b-9155-9093d4ba3464 508b406d37c34ac499f41e29a81a088d d54b97ab58014004be21ea398033a716 - default default] Unexpected exception in API method
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 631, in create
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     **create_kwargs)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1535, in create
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1128, in _create_instance
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     reservation_id, max_count)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 824, in _validate_and_build_base_options
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     requested_networks, max_count)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 439, in _check_requested_networks
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     max_count)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 391, in validate_networks
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     requested_networks)
20ss 'oslo_messaging.exceptions.MessagingTimeout'>
18-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 214, in validate_networks
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return self.client.call(ctxt, 'validate_networks', networks=networks)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 428, in call
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return self.prepare().call(ctxt, method, **kwargs)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     retry=self.retry)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     retry=retry)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 453, in _send
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     result = self._waiter.wait(msg_id, timeout)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in wait
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     message = self.waiters.get(msg_id, timeout=timeout)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     'to message ID %s' % msg_id)
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID a8099e5f3654407da89066257aa45969
2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions
2018-06-18 08:40:17.002 22686 INFO nova.api.openstack.wsgi [req-85af8d4d-b517-4d7b-9155-9093d4ba3464 508b406d37c34ac499f41e29a81a088d d54b97ab58014004be21ea398033a716 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2018-06-18 08:40:17.005 22686 INFO nova.osapi_compute.wsgi.server [req-85af8d4d-b517-4d7b-9155-9093d4ba3464 508b406d37c34ac499f41e29a81a088d d54b97ab58014004be21ea398033a716 - default default] 192.168.133.35 "POST /v2.1/d54b97ab58014004be21ea398033a716/servers HTTP/1.1" status: 500 len: 583 time: 60.9287138

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1777382

Title:
  class 'oslo_messaging.exceptions.MessagingTimeout'

Status in OpenStack Compute (nova):
  New

Bug description:
  /var/log/nova/nova-api.log is as follows

  2018-06-18 08:39:14.619 22686 INFO nova.osapi_compute.wsgi.server [req-dca907e2-f894-4bee-953e-2701dbb1c20c 508b406d37c34ac499f41e29a81a088d d54b97ab58014004be21ea398033a716 - default default] 192.168.133.35 "GET /v2.1/d54b97ab58014004be21ea398033a716/flavors/small HTTP/1.1" status: 404 len: 432 time: 0.2995820
  2018-06-18 08:39:14.949 22686 INFO nova.osapi_compute.wsgi.server [req-3500f964-c761-46a3-baef-bd15e5542272 508b406d37c34ac499f41e29a81a088d d54b97ab58014004be21ea398033a716 - default default] 192.168.133.35 "GET /v2.1/d54b97ab58014004be21ea398033a716/flavors HTTP/1.1" status: 200 len: 689 time: 0.3250749
  2018-06-18 08:39:15.329 22686 INFO nova.osapi_compute.wsgi.server [req-084d5aec-6c16-401a-97f9-48322d9ae2f7 508b406d37c34ac499f41e29a81a088d d54b97ab58014004be21ea398033a716 - default default] 192.168.133.35 "GET /v2.1/d54b97ab58014004be21ea398033a716/flavors/49a9f23e-3f31-4005-9da5-b4b523a1c736 HTTP/1.1" status: 200 len: 857 time: 0.3730719
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions [req-85af8d4d-b517-4d7b-9155-9093d4ba3464 508b406d37c34ac499f41e29a81a088d d54b97ab58014004be21ea398033a716 - default default] Unexpected exception in API method
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 338, in wrapped
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 631, in create
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     **create_kwargs)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1535, in create
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1128, in _create_instance
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     reservation_id, max_count)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 824, in _validate_and_build_base_options
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     requested_networks, max_count)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 439, in _check_requested_networks
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     max_count)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 391, in validate_networks
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     requested_networks)
  20ss 'oslo_messaging.exceptions.MessagingTimeout'>
  18-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 214, in validate_networks
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return self.client.call(ctxt, 'validate_networks', networks=networks)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 428, in call
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     return self.prepare().call(ctxt, method, **kwargs)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     retry=self.retry)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     retry=retry)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 453, in _send
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     result = self._waiter.wait(msg_id, timeout)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in wait
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     message = self.waiters.get(msg_id, timeout=timeout)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions     'to message ID %s' % msg_id)
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID a8099e5f3654407da89066257aa45969
  2018-06-18 08:40:16.997 22686 ERROR nova.api.openstack.extensions
  2018-06-18 08:40:17.002 22686 INFO nova.api.openstack.wsgi [req-85af8d4d-b517-4d7b-9155-9093d4ba3464 508b406d37c34ac499f41e29a81a088d d54b97ab58014004be21ea398033a716 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  2018-06-18 08:40:17.005 22686 INFO nova.osapi_compute.wsgi.server [req-85af8d4d-b517-4d7b-9155-9093d4ba3464 508b406d37c34ac499f41e29a81a088d d54b97ab58014004be21ea398033a716 - default default] 192.168.133.35 "POST /v2.1/d54b97ab58014004be21ea398033a716/servers HTTP/1.1" status: 500 len: 583 time: 60.9287138

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1777382/+subscriptions
Follow ups

[Bug 1777382] Re: class 'oslo_messaging.exceptions.MessagingTimeout'
From: Matt Riedemann, 2018-06-18