yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #56358
[Bug 1622630] [NEW] Failed to launch instance due to "Unexpected API error"
Public bug reported:
Installed openstack Mitaka on ubuntu 16.04 (s390x) following
http://docs.openstack.org/mitaka/install-guide-ubuntu/index.html
Configured neutron and cinder according to my local environment
created an image - now attempting to start an instance - that fails with
an error message in horizon: (in German: "der Server kann nicht erstellt
werden")
Checked syslog and found:
Sep 12 16:23:14 s42lp12 nova-compute[3644]: 2016-09-12 16:23:14.859 3644 INFO nova.compute.resource_tracker [req-1781920a-3d2c-4a77-bf96-3df4b579fd23 - - - - -] Compute_service record updated for s42lp12:s42lp12.boeblingen.de.ibm.com
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions [req-a5c08333-24fe-419c-869d-fdc374a444a3 8f3671f652524ce3862bfcc3840cb363 adf169e55d0d4949b2831809bf0f6900 - - -] Unexpected except
ion in API method
Sep 12 16:23:44 s42lp12 rsyslogd-2007: action 'action 10' suspended, next retry is Mon Sep 12 16:25:14 2016 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions Traceback (most recent call last):
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 629, in create
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions **create_kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1562, in create
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1145, in _create_instance
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions reservation_id, max_count)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 834, in _validate_and_build_base_options
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions requested_networks, max_count)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 448, in _check_requested_networks
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions max_count)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 49, in wrapped
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return func(self, context, *args, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 399, in validate_networks
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions requested_networks)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 212, in validate_networks
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return self.client.call(ctxt, 'validate_networks', networks=networks)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 413, in call
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return self.prepare().call(ctxt, method, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions retry=self.retry)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions retry=retry)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions result = self._waiter.wait(msg_id, timeout)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions message = self.waiters.get(msg_id, timeout=timeout)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions 'to message ID %s' % msg_id)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID dcc01f7bfdb047ebb53e8609def83df5
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.684 4407 INFO nova.api.openstack.wsgi [req-a5c08333-24fe-419c-869d-fdc374a444a3 8f3671f652524ce3862bfcc3840cb363 adf169e55d0d4949b2831809bf0f6900 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
Sep 12 16:23:44 s42lp12 nova-api[3695]: <class 'oslo_messaging.exceptions.MessagingTimeout'>
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.685 4407 INFO nova.osapi_compute.wsgi.server [req-a5c08333-24fe-419c-869d-fdc374a444a3 8f3671f652524ce3862bfcc3840cb363 adf169e55d0d4949b2831809bf0f6900 - - -] 9.152.110.201 "POST /v2.1/adf169e55d0d4949b2831809bf0f6900/servers HTTP/1.1" status: 500 len: 518 time: 60.1876330
Nova API log shows exactly these messages, hence I do not attach
separately.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1622630
Title:
Failed to launch instance due to "Unexpected API error"
Status in OpenStack Compute (nova):
New
Bug description:
Installed openstack Mitaka on ubuntu 16.04 (s390x) following
http://docs.openstack.org/mitaka/install-guide-ubuntu/index.html
Configured neutron and cinder according to my local environment
created an image - now attempting to start an instance - that fails
with an error message in horizon: (in German: "der Server kann nicht
erstellt werden")
Checked syslog and found:
Sep 12 16:23:14 s42lp12 nova-compute[3644]: 2016-09-12 16:23:14.859 3644 INFO nova.compute.resource_tracker [req-1781920a-3d2c-4a77-bf96-3df4b579fd23 - - - - -] Compute_service record updated for s42lp12:s42lp12.boeblingen.de.ibm.com
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions [req-a5c08333-24fe-419c-869d-fdc374a444a3 8f3671f652524ce3862bfcc3840cb363 adf169e55d0d4949b2831809bf0f6900 - - -] Unexpected except
ion in API method
Sep 12 16:23:44 s42lp12 rsyslogd-2007: action 'action 10' suspended, next retry is Mon Sep 12 16:25:14 2016 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions Traceback (most recent call last):
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 629, in create
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions **create_kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1562, in create
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1145, in _create_instance
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions reservation_id, max_count)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 834, in _validate_and_build_base_options
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions requested_networks, max_count)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 448, in _check_requested_networks
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions max_count)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 49, in wrapped
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return func(self, context, *args, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 399, in validate_networks
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions requested_networks)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 212, in validate_networks
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return self.client.call(ctxt, 'validate_networks', networks=networks)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 413, in call
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions return self.prepare().call(ctxt, method, **kwargs)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions retry=self.retry)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions retry=retry)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions result = self._waiter.wait(msg_id, timeout)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions message = self.waiters.get(msg_id, timeout=timeout)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions 'to message ID %s' % msg_id)
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID dcc01f7bfdb047ebb53e8609def83df5
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.683 4407 ERROR nova.api.openstack.extensions
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.684 4407 INFO nova.api.openstack.wsgi [req-a5c08333-24fe-419c-869d-fdc374a444a3 8f3671f652524ce3862bfcc3840cb363 adf169e55d0d4949b2831809bf0f6900 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
Sep 12 16:23:44 s42lp12 nova-api[3695]: <class 'oslo_messaging.exceptions.MessagingTimeout'>
Sep 12 16:23:44 s42lp12 nova-api[3695]: 2016-09-12 16:23:44.685 4407 INFO nova.osapi_compute.wsgi.server [req-a5c08333-24fe-419c-869d-fdc374a444a3 8f3671f652524ce3862bfcc3840cb363 adf169e55d0d4949b2831809bf0f6900 - - -] 9.152.110.201 "POST /v2.1/adf169e55d0d4949b2831809bf0f6900/servers HTTP/1.1" status: 500 len: 518 time: 60.1876330
Nova API log shows exactly these messages, hence I do not attach
separately.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1622630/+subscriptions
Follow ups