← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1615582] Re: Error: Unable to create the server. Unexpected API Error.

 

Reopening this bug, going through an upgrade from liberty -> mitaka and
getting this bug

** Changed in: nova
       Status: Expired => New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1615582

Title:
  Error: Unable to create the server. Unexpected API Error.

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  After upgrading fresh OpenStack Liberty installation to Mitaka on Trusty with Juju, I am no longer able to create Nova instances.

  Steps to reproduce
  ==================
  * Install OpenStack Liberty with Juju from the attached "bundle.yaml" file.
  * Upgrade OpenStack Liberty to Mitaka by executing the following commands:

  juju upgrade-charm keystone
  juju upgrade-charm ceilometer
  juju upgrade-charm ceilometer-agent
  juju upgrade-charm ceph
  juju upgrade-charm ceph-osd
  juju upgrade-charm cinder
  juju upgrade-charm glance
  juju upgrade-charm nova-cloud-controller
  juju upgrade-charm nova-compute
  juju upgrade-charm neutron-api
  juju upgrade-charm neutron-gateway
  juju upgrade-charm openstack-dashboard
  juju set-config ceph source="cloud:trusty-mitaka"
  juju set-config ceph-osd source="cloud:trusty-mitaka"
  juju set-config keystone openstack-origin="cloud:trusty-mitaka"
  juju ssh keystone/0 sudo keystone-manage db_sync
  juju set-config ceilometer openstack-origin="cloud:trusty-mitaka"
  juju set-config ceilometer-agent openstack-origin="cloud:trusty-mitaka"
  juju set-config cinder openstack-origin="cloud:trusty-mitaka"
  juju ssh cinder/0 sudo cinder-manage db sync
  juju set-config glance openstack-origin="cloud:trusty-mitaka"
  juju ssh glance/0 sudo glance-manage db_sync
  juju set-config nova-cloud-controller openstack-origin="cloud:trusty-mitaka"
  juju ssh nova-cloud-controller/0 sudo nova-manage db sync
  juju set-config nova-compute openstack-origin="cloud:trusty-mitaka"
  juju set-config neutron-api openstack-origin="cloud:trusty-mitaka"
  juju ssh neutron-api/0 sudo neutron-db-manage upgrade heads
  juju set-config neutron-gateway openstack-origin="cloud:trusty-mitaka"
  juju set-config openstack-dashboard openstack-origin="cloud:trusty-mitaka"
  juju ssh ceph/0 sudo reboot
  juju ssh ceph/1 sudo reboot
  juju ssh ceph/2 sudo reboot
  juju ssh ceph-osd/0 sudo service ceph restart
  juju ssh ceph-osd/1 sudo service ceph restart
  juju ssh ceph-osd/2 sudo service ceph restart
  juju ssh keystone/0 sudo reboot
  juju ssh keystone/1 sudo reboot
  juju ssh keystone/2 sudo reboot
  juju ssh ceilometer/0 sudo reboot
  juju ssh ceilometer/1 sudo reboot
  juju ssh ceilometer/2 sudo reboot
  juju ssh ceilometer-agent/0 sudo service ceilometer-agent-compute restart
  juju ssh ceilometer-agent/1 sudo service ceilometer-agent-compute restart
  juju ssh ceilometer-agent/2 sudo service ceilometer-agent-compute restart
  juju ssh cinder/0 sudo reboot
  juju ssh cinder/1 sudo reboot
  juju ssh cinder/2 sudo reboot
  juju ssh glance/0 sudo reboot
  juju ssh glance/1 sudo reboot
  juju ssh glance/2 sudo reboot
  juju ssh nova-cloud-controller/0 sudo reboot
  juju ssh nova-cloud-controller/1 sudo reboot
  juju ssh nova-cloud-controller/2 sudo reboot
  juju ssh nova-compute/0 sudo service nova-compute restart
  juju ssh nova-compute/1 sudo service nova-compute restart
  juju ssh nova-compute/2 sudo service nova-compute restart
  juju ssh neutron-api/0 sudo reboot
  juju ssh neutron-api/1 sudo reboot
  juju ssh neutron-api/2 sudo reboot
  juju ssh neutron-gateway/0 sudo service neutron-dhcp-agent restart
  juju ssh neutron-gateway/0 sudo service neutron-lbaas-agent restart
  juju ssh neutron-gateway/0 sudo service neutron-metadata-agent restart
  juju ssh neutron-gateway/0 sudo service neutron-metering-agent restart
  juju ssh neutron-gateway/0 sudo service neutron-openvswitch-agent restart
  juju ssh neutron-gateway/0 sudo service neutron-vpn-agent restart
  juju ssh nova-compute/0 sudo service neutron-openvswitch-agent restart
  juju ssh nova-compute/1 sudo service neutron-openvswitch-agent restart
  juju ssh nova-compute/2 sudo service neutron-openvswitch-agent restart
  juju ssh openstack-dashboard/0 sudo reboot
  juju ssh openstack-dashboard/1 sudo reboot
  juju ssh openstack-dashboard/2 sudo reboot

  * Attempt to create Nova instance.

  Expected result
  ===============
  Nova instance being created.

  Actual result
  =============
  * Nova instance not being created.
  * The following error messages being displayed:
  ** from GUI:

     Error: Unable to create the server.

  ** from CLI:

     Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
     <type 'exceptions.AttributeError'> (HTTP 500) (Request-ID: req-c035d518-50c0-4cab-913e-9a5263392a2a)

  Environment
  ===========
  1. Exact version of OpenStack you are running.

  ubuntu@tkurek-maas:~$ juju ssh nova-cloud-controller/0 sudo dpkg-query -l | grep nova
  ii  nova-api-os-compute                           2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute - OpenStack Compute API frontend
  ii  nova-cert                                     2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute - certificate management
  ii  nova-common                                   2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute - common files
  ii  nova-conductor                                2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute - conductor service
  ii  nova-consoleauth                              2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                               2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                                2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute - virtual machine scheduler
  ii  python-nova                                   2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute Python libraries
  ii  python-novaclient                             2:3.3.1-2~cloud0            all                         client library for OpenStack Compute API - Python 2.7
  Connection to 10.230.126.81 closed.
  ubuntu@tkurek-maas:~$ juju ssh nova-compute/0 sudo dpkg-query -l | grep nova
  ii  nova-common                                   2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute - common files
  ii  nova-compute                                  2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute - compute node base
  ii  nova-compute-kvm                              2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                          2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute - compute node libvirt support
  ii  python-nova                                   2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute Python libraries
  ii  python-novaclient                             2:3.3.1-2~cloud0            all                         client library for OpenStack Compute API - Python 2.7
  Connection to 10.230.126.54 closed.

  2. Which hypervisor did you use?

  KVM

     What's the version of that?

  ubuntu@tkurek-maas:~$ juju ssh nova-compute/0 sudo dpkg-query -l | grep kvm
  ii  nova-compute-kvm                              2:13.1.0-0ubuntu1~cloud0    all                         OpenStack Compute - compute node (KVM)
  Connection to 10.230.126.54 closed.

  3. Which storage type did you use?

  Ceph

     What's the version of that?

  ubuntu@tkurek-maas:~$ juju ssh ceph/0 sudo dpkg-query -l | grep ceph
  ii  ceph                                          10.2.2-0ubuntu0.16.04.2~clo amd64                       distributed storage and file system
  ii  ceph-common                                   10.2.2-0ubuntu0.16.04.2~clo amd64                       common utilities to mount and interact with a ceph storage cluster
  ii  libcephfs1                                    10.2.2-0ubuntu0.16.04.2~clo amd64                       Ceph distributed file system client library
  ii  python-ceph                                   10.2.2-0ubuntu0.16.04.2~clo all                         Meta-package for python libraries for the Ceph libraries
  ii  python-cephfs                                 10.2.2-0ubuntu0.16.04.2~clo amd64                       Python libraries for the Ceph libcephfs library
  Connection to 10.230.126.65 closed.
  ubuntu@tkurek-maas:~$ juju ssh ceph-osd/0 sudo dpkg-query -l | grep ceph
  ii  ceph                                          10.2.2-0ubuntu0.16.04.2~clo amd64                       distributed storage and file system
  ii  ceph-common                                   10.2.2-0ubuntu0.16.04.2~clo amd64                       common utilities to mount and interact with a ceph storage cluster
  ii  libcephfs1                                    10.2.2-0ubuntu0.16.04.2~clo amd64                       Ceph distributed file system client library
  ii  python-ceph                                   10.2.2-0ubuntu0.16.04.2~clo all                         Meta-package for python libraries for the Ceph libraries
  ii  python-cephfs                                 10.2.2-0ubuntu0.16.04.2~clo amd64                       Python libraries for the Ceph libcephfs library
  Connection to 10.230.126.54 closed.

  4. Which networking type did you use?

  Neutron with Openvswitch

  5. Additional information:

  ubuntu@tkurek-maas:~$ juju ssh nova-cloud-controller/0 sudo dpkg -S /usr/lib/python2.7/dist-packages/dogpile/cache/region.py
  python-dogpile.cache: /usr/lib/python2.7/dist-packages/dogpile/cache/region.py
  Connection to 10.230.126.81 closed.
  ubuntu@tkurek-maas:~$ juju ssh nova-cloud-controller/0 sudo dpkg-query -l python-dogpile.cache
  Desired=Unknown/Install/Remove/Purge/Hold
  | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
  |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
  ||/ Name                                          Version                     Architecture                Description
  +++-=============================================-===========================-===========================-===============================================================================================
  ii  python-dogpile.cache                          0.5.7-1ubuntu1~cloud0       all                         caching front-end based on the Dogpile lock - Python 2.x
  Connection to 10.230.126.81 closed.

  Logs & Configs
  ==============
  Attached "/var/log/nova/nova-api-os-compute.log" file errors:

  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions [req-b2f38657-7eae-4714-a1e1-0217950e7e29 d12595ae6b2d418eaf9ffd209fa25479 64c87895e12b4f4fb0bfd979b1ac70a4 - - -] Unexpected exception in API method
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 629, in create
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     **create_kwargs)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1562, in create
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1175, in _create_instance
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     instance_group, check_server_group_quota, filter_properties)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1033, in _provision_instances
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     quotas.rollback()
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     self.force_reraise()
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 988, in _provision_instances
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     num_instances, i, shutdown_terminate)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1445, in create_db_entry_for_new_instance
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     instance.create()
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 223, in wrapper
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     return fn(self, *args, **kwargs)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 471, in create
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     self._load_ec2_ids()
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 818, in _load_ec2_ids
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     self.ec2_ids = objects.EC2Ids.get_by_instance(self._context, self)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/objects/ec2.py", line 218, in get_by_instance
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     ec2_ids = cls._get_ec2_ids(context, instance)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/objects/ec2.py", line 204, in _get_ec2_ids
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     instance.image_ref)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/ec2/ec2utils.py", line 128, in glance_id_to_ec2_id
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     image_id = glance_id_to_id(context, glance_id)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/api/ec2/ec2utils.py", line 48, in memoizer
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     value = _CACHE.get(key)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/nova/cache_utils.py", line 151, in get
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     value = self.region.get(key)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 456, in get
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     expiration_time, ignore_expiration)(value)
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 475, in value_fn
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions     current_time - value.metadata["ct"] > expiration_time:
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions AttributeError: 'int' object has no attribute 'metadata'
  2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1615582/+subscriptions


References