yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1622630] Re: Failed to launch instance due to "Unexpected API error"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Mon, 12 Sep 2016 17:18:51 -0000
Reply-to: Bug 1622630 <1622630@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
You're hitting messaging timeouts, which means there is something wrong
with rabbitmq or the configuration between services to talk to rabbitmq,
check your setup. This is not a nova bug, it's a problem in your
deployment/configuration.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1622630

Title:
  Failed to launch instance due to "Unexpected API error"

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Installed openstack Mitaka on ubuntu 16.04 (s390x) following 
  http://docs.openstack.org/mitaka/install-guide-ubuntu/index.html

  Configured neutron and cinder according to my local environment

  created an image - now attempting to start an instance - that fails
  with an error message in horizon: (in German: "der Server kann nicht
  erstellt werden")

  Checked syslog and found:

  Sep 12 16:23:14 s42lp12 nova-compute[3644]: 2016-09-12 16:23:14.859 3644 INFO nova.compute.resource_tracker [req-1781920a-3d2c-4a77-bf96-3df4b579fd23 - - - - -] Compute_service record updated for s42lp12:s42lp12.boeblingen.de.ibm.com
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions [req-a5c08333-24fe-419c-869d-fdc374a444a3 8f3671f652524ce3862bfcc3840cb363 adf169e55d0d4949b2831809bf0f6900 - - -] Unexpected except
  ion in API method
  Sep 12 16:23:44 s42lp12 rsyslogd-2007: action 'action 10' suspended, next retry is Mon Sep 12 16:25:14 2016 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 629, in create
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     **create_kwargs)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1562, in create
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1145, in _create_instance
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     reservation_id, max_count)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 834, in _validate_and_build_base_options
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     requested_networks, max_count)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 448, in _check_requested_networks
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     max_count)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 49, in wrapped
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     return func(self, context, *args, **kwargs)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 399, in validate_networks
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     requested_networks)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 212, in validate_networks
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     return self.client.call(ctxt, 'validate_networks', networks=networks)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 413, in call
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     return self.prepare().call(ctxt, method, **kwargs)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     retry=self.retry)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     retry=retry)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     result = self._waiter.wait(msg_id, timeout)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     message = self.waiters.get(msg_id, timeout=timeout)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions     'to message ID %s' % msg_id)
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID dcc01f7bfdb047ebb53e8609def83df5
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.684 4407 INFO nova.api.openstack.wsgi [req-a5c08333-24fe-419c-869d-fdc374a444a3 8f3671f652524ce3862bfcc3840cb363 adf169e55d0d4949b2831809bf0f6900 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  Sep 12 16:23:44 s42lp12 nova-api[3695]: <class 'oslo_messaging.exceptions.MessagingTimeout'>
  Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.685 4407 INFO nova.osapi_compute.wsgi.server [req-a5c08333-24fe-419c-869d-fdc374a444a3 8f3671f652524ce3862bfcc3840cb363 adf169e55d0d4949b2831809bf0f6900 - - -] 9.152.110.201 "POST /v2.1/adf169e55d0d4949b2831809bf0f6900/servers HTTP/1.1" status: 500 len: 518 time: 60.1876330

  Nova API log shows exactly these messages, hence I do not attach
  separately.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1622630/+subscriptions
References

[Bug 1622630] [NEW] Failed to launch instance due to "Unexpected API error"
From: Holger Smolinski, 2016-09-12