yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1563029] Re: Nova instance fail to launch: "Unexpected API Error."

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Augustina Ragwitz <1563029@xxxxxxxxxxxxxxxxxx>
Date: Tue, 29 Mar 2016 01:32:57 -0000
Reply-to: Bug 1563029 <1563029@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
I moved this over to Kolla since this looks specific to that project and
I'm not working in that repo, so I can't reproduce this. If this turns
out to be a Nova specific issue, please update the bug with Nova-
specific reproduction steps and we'll take a look at it.

** Also affects: kolla
   Importance: Undecided
       Status: New

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1563029

Title:
  Nova instance fail to launch: "Unexpected API Error."

Status in kolla:
  New
Status in OpenStack Compute (nova):
  Invalid

Bug description:
  This is a Kolla installation of openstack. The docker images build
  well and some services work great; e.g cinder, glance and neutron.

  Nova seems to have a problem that is prevent provisioning instances.
  The manila-api logs below says:

  "Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1070"


  1. The GIT commit level is:

  lsorrillo@ptest:~/git/kolla$ git log -1
  commit 90321d0497f14bda370908610cfc44296a940523
  Merge: 328a7e8 a789346
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Sun Mar 27 23:42:47 2016 +0000

      Merge "Fix gate to use world writeable docker socket"
  lsorrillo@ptest:~/git/kolla$

  RPM version:

  lsorrillo@ptest:~$
  lsorrillo@ptest:~$ docker exec -ti nova_api bash
  bash-4.2$
  bash-4.2$
  bash-4.2$
  bash-4.2$ rpm -qa | grep nova
  python-novaclient-3.3.1-0.20160303082753.d63800d.el7.centos.noarch
  openstack-nova-common-13.0.0.0b4-0.20160304162843.c5a45a2.el7.centos.noarch
  python-nova-13.0.0.0b4-0.20160304162843.c5a45a2.el7.centos.noarch
  openstack-nova-api-13.0.0.0b4-0.20160304162843.c5a45a2.el7.centos.noarch
  bash-4.2$

  2.

  The following error is seen in the nova-api.log:

  2016-03-28 19:53:40.086 29 DEBUG oslo.messaging._drivers.impl_rabbit [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Connecting to AMQP server on 192.168.0.121:5672 __init__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:535
  2016-03-28 19:53:40.132 29 DEBUG oslo.messaging._drivers.impl_rabbit [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Connected to AMQP server on 192.168.0.121:5672 via [amqp] client __init__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:562
  2016-03-28 19:53:40.148 29 DEBUG oslo.messaging._drivers.pool [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Pool creating new connection create /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py:109
  2016-03-28 19:53:40.151 29 DEBUG oslo.messaging._drivers.impl_rabbit [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Connecting to AMQP server on 192.168.0.121:5672 __init__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:535
  2016-03-28 19:53:40.205 29 DEBUG oslo.messaging._drivers.impl_rabbit [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Connected to AMQP server on 192.168.0.121:5672 via [amqp] client __init__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:562
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Unexpected exception in API method
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 630, in create
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     **create_kwargs)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1556, in create
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1139, in _create_instance
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     reservation_id, max_count)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 834, in _validate_and_build_base_options
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     requested_networks, max_count)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 448, in _check_requested_networks
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     max_count)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 49, in wrapped
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     return func(self, context, *args, **kwargs)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 399, in validate_networks
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     requested_networks)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 212, in validate_networks
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     return self.client.call(ctxt, 'validate_networks', networks=networks)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 413, in call
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     return self.prepare().call(ctxt, method, **kwargs)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     retry=self.retry)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     retry=retry)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     result = self._waiter.wait(msg_id, timeout)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     message = self.waiters.get(msg_id, timeout=timeout)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions     'to message ID %s' % msg_id)
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID 73fdd3b38da34bf7a19b08a086fb26f8
  2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions
  2016-03-28 19:54:40.229 29 INFO nova.api.openstack.wsgi [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'>
  2016-03-28 19:54:40.231 29 DEBUG nova.api.openstack.wsgi [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1070
  2016-03-28 19:54:40.237 29 INFO nova.osapi_compute.wsgi.server [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 483, in handle_one_response
      write(b''.join(towrite))
    File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 426, in write
      _writelines(towrite)
    File "/usr/lib64/python2.7/socket.py", line 334, in writelines
      self.flush()
    File "/usr/lib64/python2.7/socket.py", line 303, in flush
      self._sock.sendall(view[write_offset:write_offset+buffer_size])
    File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 377, in sendall
      tail = self.send(data, flags)
    File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send
      total_sent += fd.send(data[total_sent:], flags)
  error: [Errno 104] Connection reset by peer

  2016-03-28 19:54:40.238 29 INFO nova.osapi_compute.wsgi.server [req-
  88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84
  9e4b40d958e741599adc497b0a9d955a - - -] 10.10.10.254,192.168.0.121
  "POST /v2/9e4b40d958e741599adc497b0a9d955a/servers HTTP/1.1" status:
  500 len: 0 time: 60.8730421

  3.  git clone https://git.openstack.org/openstack/kolla
  git checkout -b ltshome

  and make the following changes:

  lsorrillo@ptest:~/git/kolla$ git status
  # On branch ltshome
  # Changes not staged for commit:
  #   (use "git add <file>..." to update what will be committed)
  #   (use "git checkout -- <file>..." to discard changes in working directory)
  #
  #	modified:   ansible/group_vars/all.yml
  #	modified:   etc/kolla/globals.yml
  #	modified:   etc/kolla/passwords.yml
  #
  # Untracked files:
  #   (use "git add <file>..." to include in what will be committed)
  #
  #	etc/kolla/passwords.yml.orig
  no changes added to commit (use "git add" and/or "git commit -a")
  lsorrillo@ptest:~/git/kolla$

  Here are diffs of the modified files above:

  lsorrillo@ptest:~/git/kolla$ git diff  ansible/group_vars/all.yml
  diff --git a/ansible/group_vars/all.yml b/ansible/group_vars/all.yml
  index a0ab95a..6ec956e 100644
  --- a/ansible/group_vars/all.yml
  +++ b/ansible/group_vars/all.yml
  @@ -176,7 +176,7 @@ enable_rabbitmq: "yes"
   # Additional optional OpenStack services are specified here
   enable_ceph: "no"
   enable_ceph_rgw: "no"
  -enable_cinder: "no"
  +enable_cinder: "yes"
   enable_heat: "yes"
   enable_horizon: "yes"
   enable_swift: "no"
  @@ -184,9 +184,9 @@ enable_murano: "no"
   enable_ironic: "no"
   enable_magnum: "no"
   enable_mistral: "no"
  -enable_central_logging: "no"
  +enable_central_logging: "yes"
   enable_mongodb: "no"
  -enable_manila: "no"
  +enable_manila: "yes"

   ironic_keystone_user: "ironic"
   neutron_keystone_user: "neutron"
  lsorrillo@ptest:~/git/kolla$

  lsorrillo@ptest:~/git/kolla$ git diff etc/kolla/globals.yml
  diff --git a/etc/kolla/globals.yml b/etc/kolla/globals.yml
  index ec26126..2afea72 100644
  --- a/etc/kolla/globals.yml
  +++ b/etc/kolla/globals.yml
  @@ -55,14 +55,14 @@ kolla_internal_vip_address: "10.10.10.254"
   # This interface is what all your api services will be bound to by default.
   # Additionally, all vxlan/tunnel and storage network traffic will go over this
   # interface by default. This interface must contain an IPv4 address.
  -network_interface: "eth0"
  +network_interface: "enp2s0f1"

   # These can be adjusted for even more customization. The default is the same as
   # the 'network_interface'. These interfaces must container an IPv4 address.
  -#kolla_external_vip_interface: "{{ network_interface }}"
  -#api_interface: "{{ network_interface }}"
  -#storage_interface: "{{ network_interface }}"
  -#tunnel_interface: "{{ network_interface }}"
  +kolla_external_vip_interface: "{{ network_interface }}"
  +api_interface: "{{ network_interface }}"
  +storage_interface: "{{ network_interface }}"
  +tunnel_interface: "{{ network_interface }}"

   # This is the raw interface given to neutron as its external network port. Even
   # though an IP address can exist on this interface, it will be unusable in most
  @@ -80,7 +80,7 @@ neutron_external_interface: "eth1"
   # To provide encryption and authentication on the kolla_external_vip_interface,
   # TLS can be enabled.  When TLS is enabled, certificates must be provided to
   # allow clients to perform authentication.
  -#kolla_enable_tls_external: "no"
  +kolla_enable_tls_external: "no"
   #kolla_external_fqdn_cert: "{{ node_config_directory }}/certificates/haproxy.pem"

  @@ -89,13 +89,13 @@ neutron_external_interface: "eth1"
   ####################
   # Use these options to set the various log levels across all OpenStack projects
   # Valid options are [ True, False ]
  -#openstack_logging_debug: "False"
  +openstack_logging_debug: "True"

   # Valid options are [ novnc, spice ]
   #nova_console: "novnc"

   # OpenStack services can be enabled or disabled with these options
  -#enable_cinder: "no"
  +enable_cinder: "yes"
   #enable_heat: "yes"
   #enable_magnum: "no"
   #enable_swift: "no"
  @@ -118,7 +118,7 @@ neutron_external_interface: "eth1"
   #######################################
   # Manila - Shared File Systems Options
   #######################################
  -#manila_enable_dhss: "yes"
  +manila_enable_dhss: "yes"

   #######################################
  lsorrillo@ptest:~/git/kolla$

  git diff etc/kolla/passwords.yml
  xxx
  xxx

To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla/+bug/1563029/+subscriptions
References

[Bug 1563029] [NEW] Nova intance fail to launch: Unexpected API Error.
From: Lawrence Sorrillo, 2016-03-28