openstack team mailing list archive

["Mark A. Nye" <marknye@xxxxxxxxxxxxxxxxxxxxx>]

I was finally able to determine the cause of this problem. A bug report
with full description has been submitted here:
https://bugs.launchpad.net/glance/+bug/1198566

best,
Mark


On Thu, Jun 27, 2013 at 7:57 PM, Mark A. Nye
<marknye@xxxxxxxxxxxxxxxxxxxxx>wrote:

>
> Hello,
>
> We have a working two host Folsom (2012.2.3 on Ubuntu 12.04) OpenStack
> cluster with a controller and compute node. This afternoon we added a
> second compute node, but attempts to spawn instances on the new node fail
> with a glance client exception (see example log below).
>
> I've triple-checked our new nova.conf and api-paste.ini files, which are
> identical to what we have on the working compute node, except for the local
> IP address change to metadata_host, vncserver_proxyclient_address, and
> my_ip local.
>
> When I run "nova-manage service list" the nova-compute and nova-network
> services on all three machines report as status :-).
>
> The ONLY significant difference I'm seeing is that the new compute node is
> running the 2012.2.4 version of nova-network, nova-compute,
> and nova-api-metadata. I wouldn't expect this to be a problem, but maybe a
> bug or incompatibility was introduced in 2012.2.4? I'd like to try rolling
> back to 2012.2.3, but I can't find a copy of the older .deb packages.
>
> Am I missing something obvious? Can anyone offer a suggestion?
>
> best,
> Mark
>
>
>
>
>
> 2013-06-28 02:12:47 5290 ERROR nova.openstack.common.rpc.amqp [-]
> Exception during message handling
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp Traceback
> (most recent call last):
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
> 276, in _process_data
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     rval =
> self.proxy.dispatch(ctxt, version, method, **args)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
> line 145, in dispatch
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     return
> getattr(proxyobj, method)(ctxt, **kwargs)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> temp_level, payload)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> self.gen.next()
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     return
> f(*args, **kw)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 176, in
> decorated_function
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     pass
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> self.gen.next()
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 162, in
> decorated_function
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     return
> function(self, context, *args, **kwargs)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 197, in
> decorated_function
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> kwargs['instance']['uuid'], e, sys.exc_info())
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> self.gen.next()
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 191, in
> decorated_function
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     return
> function(self, context, *args, **kwargs)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 839, in
> run_instance
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> do_run_instance()
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 803, in inner
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     retval =
> f(*args, **kwargs)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 838, in
> do_run_instance
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> admin_password, is_first_time, instance)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 529, in
> _run_instance
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> self._set_instance_error_state(context, instance['uuid'])
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> self.gen.next()
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 517, in
> _run_instance
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> is_first_time, request_spec, filter_properties)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 503, in
> _run_instance
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> injected_files, admin_password)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 756, in
> _spawn
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> block_device_info)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> temp_level, payload)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> self.gen.next()
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     return
> f(*args, **kw)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1099,
> in spawn
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> admin_pass=admin_password)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1365,
> in _create_image
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> project_id=instance['project_id'])
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line
> 131, in cache
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     *args,
> **kwargs)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line
> 178, in create_image
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> prepare_template(target=base, *args, **kwargs)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 795, in inner
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     retval =
> f(*args, **kwargs)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line
> 122, in call_if_not_exists
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> fetch_func(target=target, *args, **kwargs)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 440, in
> fetch_image
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> images.fetch_to_raw(context, image_id, target, user_id, project_id)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87, in
> fetch_to_raw
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> fetch(context, image_href, path_tmp, user_id, project_id)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 82, in fetch
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> image_service.download(context, image_id, image_file)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 222, in
> download
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> _reraise_translated_image_exception(image_id)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 220, in
> download
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> image_chunks = self._client.call(context, 1, 'data', image_id)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 138, in call
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     return
> getattr(client.images, method)(*args, **kwargs)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 108, in
> data
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     resp,
> body = self.api.raw_request('GET', '/v1/images/%s' % image_id)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 191,
> in raw_request
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     return
> self._http_request(url, method, **kwargs)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 158,
> in _http_request
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp     raise
> exc.from_response(resp)
> 2013-06-28 02:12:47 5290 TRACE nova.openstack.common.rpc.amqp
> HTTPInternalServerError: HTTPInternalServerError (HTTP 500)
>