yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #68210
[Bug 1615582] Re: Liberty -> Mitaka Upgrade: Error: Unable to create the server. Unexpected API Error.
[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]
** Changed in: nova
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1615582
Title:
Liberty -> Mitaka Upgrade: Error: Unable to create the server.
Unexpected API Error.
Status in OpenStack Compute (nova):
Expired
Bug description:
Description
===========
After upgrading fresh OpenStack Liberty installation to Mitaka on Trusty with Juju, I am no longer able to create Nova instances.
Steps to reproduce
==================
* Install OpenStack Liberty with Juju from the attached "bundle.yaml" file.
* Upgrade OpenStack Liberty to Mitaka by executing the following commands:
juju upgrade-charm keystone
juju upgrade-charm ceilometer
juju upgrade-charm ceilometer-agent
juju upgrade-charm ceph
juju upgrade-charm ceph-osd
juju upgrade-charm cinder
juju upgrade-charm glance
juju upgrade-charm nova-cloud-controller
juju upgrade-charm nova-compute
juju upgrade-charm neutron-api
juju upgrade-charm neutron-gateway
juju upgrade-charm openstack-dashboard
juju set-config ceph source="cloud:trusty-mitaka"
juju set-config ceph-osd source="cloud:trusty-mitaka"
juju set-config keystone openstack-origin="cloud:trusty-mitaka"
juju ssh keystone/0 sudo keystone-manage db_sync
juju set-config ceilometer openstack-origin="cloud:trusty-mitaka"
juju set-config ceilometer-agent openstack-origin="cloud:trusty-mitaka"
juju set-config cinder openstack-origin="cloud:trusty-mitaka"
juju ssh cinder/0 sudo cinder-manage db sync
juju set-config glance openstack-origin="cloud:trusty-mitaka"
juju ssh glance/0 sudo glance-manage db_sync
juju set-config nova-cloud-controller openstack-origin="cloud:trusty-mitaka"
juju ssh nova-cloud-controller/0 sudo nova-manage db sync
juju set-config nova-compute openstack-origin="cloud:trusty-mitaka"
juju set-config neutron-api openstack-origin="cloud:trusty-mitaka"
juju ssh neutron-api/0 sudo neutron-db-manage upgrade heads
juju set-config neutron-gateway openstack-origin="cloud:trusty-mitaka"
juju set-config openstack-dashboard openstack-origin="cloud:trusty-mitaka"
juju ssh ceph/0 sudo reboot
juju ssh ceph/1 sudo reboot
juju ssh ceph/2 sudo reboot
juju ssh ceph-osd/0 sudo service ceph restart
juju ssh ceph-osd/1 sudo service ceph restart
juju ssh ceph-osd/2 sudo service ceph restart
juju ssh keystone/0 sudo reboot
juju ssh keystone/1 sudo reboot
juju ssh keystone/2 sudo reboot
juju ssh ceilometer/0 sudo reboot
juju ssh ceilometer/1 sudo reboot
juju ssh ceilometer/2 sudo reboot
juju ssh ceilometer-agent/0 sudo service ceilometer-agent-compute restart
juju ssh ceilometer-agent/1 sudo service ceilometer-agent-compute restart
juju ssh ceilometer-agent/2 sudo service ceilometer-agent-compute restart
juju ssh cinder/0 sudo reboot
juju ssh cinder/1 sudo reboot
juju ssh cinder/2 sudo reboot
juju ssh glance/0 sudo reboot
juju ssh glance/1 sudo reboot
juju ssh glance/2 sudo reboot
juju ssh nova-cloud-controller/0 sudo reboot
juju ssh nova-cloud-controller/1 sudo reboot
juju ssh nova-cloud-controller/2 sudo reboot
juju ssh nova-compute/0 sudo service nova-compute restart
juju ssh nova-compute/1 sudo service nova-compute restart
juju ssh nova-compute/2 sudo service nova-compute restart
juju ssh neutron-api/0 sudo reboot
juju ssh neutron-api/1 sudo reboot
juju ssh neutron-api/2 sudo reboot
juju ssh neutron-gateway/0 sudo service neutron-dhcp-agent restart
juju ssh neutron-gateway/0 sudo service neutron-lbaas-agent restart
juju ssh neutron-gateway/0 sudo service neutron-metadata-agent restart
juju ssh neutron-gateway/0 sudo service neutron-metering-agent restart
juju ssh neutron-gateway/0 sudo service neutron-openvswitch-agent restart
juju ssh neutron-gateway/0 sudo service neutron-vpn-agent restart
juju ssh nova-compute/0 sudo service neutron-openvswitch-agent restart
juju ssh nova-compute/1 sudo service neutron-openvswitch-agent restart
juju ssh nova-compute/2 sudo service neutron-openvswitch-agent restart
juju ssh openstack-dashboard/0 sudo reboot
juju ssh openstack-dashboard/1 sudo reboot
juju ssh openstack-dashboard/2 sudo reboot
* Attempt to create Nova instance.
Expected result
===============
Nova instance being created.
Actual result
=============
* Nova instance not being created.
* The following error messages being displayed:
** from GUI:
Error: Unable to create the server.
** from CLI:
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.AttributeError'> (HTTP 500) (Request-ID: req-c035d518-50c0-4cab-913e-9a5263392a2a)
Environment
===========
1. Exact version of OpenStack you are running.
ubuntu@tkurek-maas:~$ juju ssh nova-cloud-controller/0 sudo dpkg-query -l | grep nova
ii nova-api-os-compute 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - OpenStack Compute API frontend
ii nova-cert 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - certificate management
ii nova-common 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-conductor 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - conductor service
ii nova-consoleauth 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Connection to 10.230.126.81 closed.
ubuntu@tkurek-maas:~$ juju ssh nova-compute/0 sudo dpkg-query -l | grep nova
ii nova-common 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Connection to 10.230.126.54 closed.
2. Which hypervisor did you use?
KVM
What's the version of that?
ubuntu@tkurek-maas:~$ juju ssh nova-compute/0 sudo dpkg-query -l | grep kvm
ii nova-compute-kvm 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
Connection to 10.230.126.54 closed.
3. Which storage type did you use?
Ceph
What's the version of that?
ubuntu@tkurek-maas:~$ juju ssh ceph/0 sudo dpkg-query -l | grep ceph
ii ceph 10.2.2-0ubuntu0.16.04.2~clo amd64 distributed storage and file system
ii ceph-common 10.2.2-0ubuntu0.16.04.2~clo amd64 common utilities to mount and interact with a ceph storage cluster
ii libcephfs1 10.2.2-0ubuntu0.16.04.2~clo amd64 Ceph distributed file system client library
ii python-ceph 10.2.2-0ubuntu0.16.04.2~clo all Meta-package for python libraries for the Ceph libraries
ii python-cephfs 10.2.2-0ubuntu0.16.04.2~clo amd64 Python libraries for the Ceph libcephfs library
Connection to 10.230.126.65 closed.
ubuntu@tkurek-maas:~$ juju ssh ceph-osd/0 sudo dpkg-query -l | grep ceph
ii ceph 10.2.2-0ubuntu0.16.04.2~clo amd64 distributed storage and file system
ii ceph-common 10.2.2-0ubuntu0.16.04.2~clo amd64 common utilities to mount and interact with a ceph storage cluster
ii libcephfs1 10.2.2-0ubuntu0.16.04.2~clo amd64 Ceph distributed file system client library
ii python-ceph 10.2.2-0ubuntu0.16.04.2~clo all Meta-package for python libraries for the Ceph libraries
ii python-cephfs 10.2.2-0ubuntu0.16.04.2~clo amd64 Python libraries for the Ceph libcephfs library
Connection to 10.230.126.54 closed.
4. Which networking type did you use?
Neutron with Openvswitch
5. Additional information:
ubuntu@tkurek-maas:~$ juju ssh nova-cloud-controller/0 sudo dpkg -S /usr/lib/python2.7/dist-packages/dogpile/cache/region.py
python-dogpile.cache: /usr/lib/python2.7/dist-packages/dogpile/cache/region.py
Connection to 10.230.126.81 closed.
ubuntu@tkurek-maas:~$ juju ssh nova-cloud-controller/0 sudo dpkg-query -l python-dogpile.cache
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=============================================-===========================-===========================-===============================================================================================
ii python-dogpile.cache 0.5.7-1ubuntu1~cloud0 all caching front-end based on the Dogpile lock - Python 2.x
Connection to 10.230.126.81 closed.
Logs & Configs
==============
Attached "/var/log/nova/nova-api-os-compute.log" file errors:
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions [req-b2f38657-7eae-4714-a1e1-0217950e7e29 d12595ae6b2d418eaf9ffd209fa25479 64c87895e12b4f4fb0bfd979b1ac70a4 - - -] Unexpected exception in API method
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 629, in create
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions **create_kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1562, in create
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1175, in _create_instance
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions instance_group, check_server_group_quota, filter_properties)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1033, in _provision_instances
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions quotas.rollback()
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions self.force_reraise()
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 988, in _provision_instances
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions num_instances, i, shutdown_terminate)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1445, in create_db_entry_for_new_instance
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions instance.create()
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 223, in wrapper
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 471, in create
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions self._load_ec2_ids()
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 818, in _load_ec2_ids
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions self.ec2_ids = objects.EC2Ids.get_by_instance(self._context, self)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/ec2.py", line 218, in get_by_instance
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions ec2_ids = cls._get_ec2_ids(context, instance)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/ec2.py", line 204, in _get_ec2_ids
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions instance.image_ref)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/ec2/ec2utils.py", line 128, in glance_id_to_ec2_id
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions image_id = glance_id_to_id(context, glance_id)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/ec2/ec2utils.py", line 48, in memoizer
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions value = _CACHE.get(key)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/cache_utils.py", line 151, in get
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions value = self.region.get(key)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 456, in get
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions expiration_time, ignore_expiration)(value)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 475, in value_fn
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions current_time - value.metadata["ct"] > expiration_time:
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions AttributeError: 'int' object has no attribute 'metadata'
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1615582/+subscriptions
References
