openstack team mailing list archive

[Antonio Messina <antonio.s.messina@xxxxxxxxx>]

Hi all,

We have a folsom installation on ubuntu, and we are having trouble
creating snapshots. Sometimes (quite often indeed) the process fails
and the snapshot image goes in "killed" status. This happen with both
the Web frontend and nova image-create client. The funny thing is that
creation of a snapshot on the very same instance can fail a few times,
but it will eventually succeed.

The nova-compute.log shows:

2013-02-11 17:03:54 DEBUG nova.utils
[req-a789eaef-d7f7-4f41-807a-a7b910471a0b
c5c6eb3e0ea8456c98acd07100a627af c4f9ddbce4f8444c896722e7c672d945]
Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf
qemu-img snapshot -c 656baf75ff104768bce7169233cb76d0
/var/lib/nova/instances/instance-0000045c/disk execute
/usr/lib/python2.7/dist-packages/nova/utils.py:183
2013-02-11 17:03:55 DEBUG nova.utils
[req-a789eaef-d7f7-4f41-807a-a7b910471a0b
c5c6eb3e0ea8456c98acd07100a627af c4f9ddbce4f8444c896722e7c672d945]
Result was 0 execute
/usr/lib/python2.7/dist-packages/nova/utils.py:199
2013-02-11 17:03:55 DEBUG nova.utils
[req-a789eaef-d7f7-4f41-807a-a7b910471a0b
c5c6eb3e0ea8456c98acd07100a627af c4f9ddbce4f8444c896722e7c672d945]
Running cmd (subprocess): qemu-img convert -f qcow2 -O qcow2 -s
656baf75ff104768bce7169233cb76d0
/var/lib/nova/instances/instance-0000045c/disk
/var/lib/nova/instances/snapshots/tmpQYvmFD/656baf75ff104768bce7169233cb76d0
execute /usr/lib/python2.7/dist-packages/nova/utils.py:183
2013-02-11 17:03:58 DEBUG nova.utils
[req-a789eaef-d7f7-4f41-807a-a7b910471a0b
c5c6eb3e0ea8456c98acd07100a627af c4f9ddbce4f8444c896722e7c672d945]
Result was 0 execute
/usr/lib/python2.7/dist-packages/nova/utils.py:199
2013-02-11 17:03:58 DEBUG nova.utils
[req-a789eaef-d7f7-4f41-807a-a7b910471a0b
c5c6eb3e0ea8456c98acd07100a627af c4f9ddbce4f8444c896722e7c672d945]
Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf
qemu-img snapshot -d 656baf75ff104768bce7169233cb76d0
/var/lib/nova/instances/instance-0000045c/disk execute
/usr/lib/python2.7/dist-packages/nova/utils.py:183
2013-02-11 17:03:58 DEBUG nova.utils
[req-a789eaef-d7f7-4f41-807a-a7b910471a0b
c5c6eb3e0ea8456c98acd07100a627af c4f9ddbce4f8444c896722e7c672d945]
Result was 0 execute
/usr/lib/python2.7/dist-packages/nova/utils.py:199
2013-02-11 17:04:01 DEBUG nova.utils
[req-a789eaef-d7f7-4f41-807a-a7b910471a0b
c5c6eb3e0ea8456c98acd07100a627af c4f9ddbce4f8444c896722e7c672d945]
Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf
tee /sys/class/net/vnet0/brport/hairpin_mode execute
/usr/lib/python2.7/dist-packages/nova/utils.py:183
2013-02-11 17:04:01 DEBUG nova.utils
[req-a789eaef-d7f7-4f41-807a-a7b910471a0b
c5c6eb3e0ea8456c98acd07100a627af c4f9ddbce4f8444c896722e7c672d945]
Result was 0 execute
/usr/lib/python2.7/dist-packages/nova/utils.py:199
2013-02-11 17:04:30 DEBUG nova.utils
[req-a789eaef-d7f7-4f41-807a-a7b910471a0b
c5c6eb3e0ea8456c98acd07100a627af c4f9ddbce4f8444c896722e7c672d945] Got
semaphore "compute_resources" for method "update_usage"... inner
/usr/lib/python2.7/dist-packages/nova/utils.py:721
2013-02-11 17:04:30 24056 ERROR nova.openstack.common.rpc.amqp [-]
Exception during message handling
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
Traceback (most recent call last):
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
line 276, in _process_data
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 145, in dispatch
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
return getattr(proxyobj, method)(ctxt, **kwargs)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
wrapped
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
temp_level, payload)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in
wrapped
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
return f(*args, **kw)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 175,
in decorated_function
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp     pass
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 161,
in decorated_function
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
return function(self, context, *args, **kwargs)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 196,
in decorated_function
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
kwargs['instance']['uuid'], e, sys.exc_info())
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 190,
in decorated_function
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
return function(self, context, *args, **kwargs)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1174,
in snapshot_instance
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
self.driver.snapshot(context, instance, image_id)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
wrapped
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
temp_level, payload)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in
wrapped
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
return f(*args, **kw)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
882, in snapshot
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp     image_file)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 253, in
update
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
_reraise_translated_image_exception(image_id)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 251, in
update
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
image_id, **image_meta)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 138, in
call
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
return getattr(client.images, method)(*args, **kwargs)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line
253, in update
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
'PUT', url, headers=hdrs, body=image_data)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line
191, in raw_request
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
return self._http_request(url, method, **kwargs)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line
158, in _http_request
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
raise exc.from_response(resp)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp
Invalid: HTTPBadRequest (HTTP 400)
2013-02-11 17:04:30 24056 TRACE nova.openstack.common.rpc.amqp

Thank in advance for your help
.a.

-- 
antonio.s.messina@xxxxxxxxx
GC3: Grid Computing Competence Center
http://www.gc3.uzh.ch/
University of Zurich
Winterthurerstrasse 190
CH-8057 Zurich Switzerland