openstack team mailing list archive

["Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>]

That's confused me, "Unexpected error while running command" will give us what command is running when this error occured, you may check the log more carefully.
and you also can migrate the VM directly by libvirt command line like below:
virsh  migrate --live --persistent --undefinesource --verbose --domain ${instance-uuid} --desturi qemu+tcp://${the-dest-host-ip}/system
then see the output result.

2013-07-26



Wangpan



发件人：Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
发送时间：2013-07-26 09:30
主题：Re: Re: Re: Re: Re: [Openstack] Configuring Live Migration
收件人："Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>
抄送："openstack@xxxxxxxxxxxxxxxxxxx"<openstack@xxxxxxxxxxxxxxxxxxx>

When I enter the action to migrate, the only two errors showen at this period are these two.



2013/7/25 Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>

There may have more error logs at this minute 2013-07-25 17:28,
you can get the backtrace near the error log you post.

2013-07-26



Wangpan



发件人：Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
发送时间：2013-07-26 04:48
主题：Re: Re: Re: Re: [Openstack] Configuring Live Migration
收件人："Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>,"openstack@xxxxxxxxxxxxxxxxxxx"<openstack@xxxxxxxxxxxxxxxxxxx>
抄送：

Hello guys, I'm getting problems when trying to migrate a VM. 


I'm getting this error on my log:


2013-07-25 17:28:31.582 ERROR nova.compute.manager [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 919f13d2-8116-4714-b2ec-d2e0f3d3c910] Unexpected error while running command.
Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state to ERROR
2013-07-25 17:28:32.212 ERROR nova.openstack.common.rpc.amqp [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling




The user Wangpan was helping me, but I guess his hour is different from mine, can anybody five me an idea?


Thank you.


Guilherme.



2013/7/25 Guilherme Russi <luisguilherme.cr@xxxxxxxxx>

Hello Wangpan, I've looking at the logs and I saw one error with my ssh port, and then I fixed the port, but now I'm getting this error:  


2013-07-25 14:09:50.388 ERROR nova.compute.manager [req-6930a2e2-60e7-421f-a560-26a626ecebcb c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: d268d24b-96fc-4450-88d4-19ba83cd8892] Unexpected error while running command.
Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state to ERROR
2013-07-25 14:09:51.005 ERROR nova.openstack.common.rpc.amqp [req-6930a2e2-60e7-421f-a560-26a626ecebcb c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling


What "host key verification"? Where should I put this key?


Thank you.


Guilherme.



2013/7/25 Guilherme Russi <luisguilherme.cr@xxxxxxxxx>

Typed wrong here 
Em 25/07/2013 00:42, "Wangpan" <hzwangpan@xxxxxxxxxxxxxxxx> escreveu:

chmod 777 -R /var/lib/nova/instances 
not the /var/log/nova/instances

2013-07-25



Wangpan



发件人：Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
发送时间：2013-07-25 10:39
主题：Re: Re: Re: [Openstack] Configuring Live Migration
收件人："Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>
抄送：

I've done chmod 777 -R /var/log/nova/instances 


Does it work?



2013/7/24 Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>

I saw this error "Permission denied: '/var/lib/nova/instances/'" in your logs.
you may need to chown the instances path  '/var/lib/nova/instances/'

2013-07-25



Wangpan



发件人：Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
发送时间：2013-07-25 10:08
主题：Re: Re: [Openstack] Configuring Live Migration
收件人："Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>
抄送：

Compute Node 01: 


cat /var/log/nova/nova-*.log | grep -E ERROR
2013-07-24 08:03:57.262 1820 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 08:43:57.655 1820 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 09:23:57.816 1820 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 10:03:57.993 1820 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 10:20:21.767 ERROR nova.compute.manager [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Instance failed to spawn
2013-07-24 10:20:29.326 1820 ERROR nova.virt.libvirt.driver [-] [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] During wait destroy, instance disappeared.
2013-07-24 10:20:29.511 ERROR nova.compute.manager [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn\n    admin_pass=admin_password)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree\n    os.makedirs(path)\n', '  File "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name, mode)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'\n"]
2013-07-24 10:33:17.279 ERROR nova.compute.manager [req-f10c5f14-b3ad-4c13-b20f-c53f57379163 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Instance failed to spawn
2013-07-24 10:33:36.673 1820 ERROR nova.virt.libvirt.driver [-] [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] During wait destroy, instance disappeared.
2013-07-24 10:33:36.837 ERROR nova.compute.manager [req-f10c5f14-b3ad-4c13-b20f-c53f57379163 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn\n    admin_pass=admin_password)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree\n    os.makedirs(path)\n', '  File "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name, mode)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'\n"]
2013-07-24 10:40:26.722 ERROR nova.compute.manager [req-5d0289b3-36d7-4046-996c-a1decb922066 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Instance failed to spawn
2013-07-24 10:40:39.402 1820 ERROR nova.virt.libvirt.driver [-] [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] During wait destroy, instance disappeared.
2013-07-24 10:40:39.530 ERROR nova.compute.manager [req-5d0289b3-36d7-4046-996c-a1decb922066 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn\n    admin_pass=admin_password)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree\n    os.makedirs(path)\n', '  File "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name, mode)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'\n"]
2013-07-24 10:43:58.194 1820 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 11:03:19.657 ERROR nova.virt.libvirt.driver [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission denied: '/var/lib/nova/instances/.directio.test''
2013-07-24 11:03:19.658 ERROR nova.compute.manager [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Instance failed to spawn
2013-07-24 11:03:32.036 1685 ERROR nova.virt.libvirt.driver [-] [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] During wait destroy, instance disappeared.
2013-07-24 11:03:32.170 ERROR nova.compute.manager [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510, in spawn\n    block_device_info=block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326, in to_xml\n    disk_info, rescue, block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244, in get_guest_config\n    inst_type):\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051, in get_guest_storage_config\n    inst_type)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013, in get_guest_disk_config\n    self.disk_cachemode,\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355, in disk_cachemode\n    if not self._supports_direct_io(CONF.instances_path):\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676, in _supports_direct_io\n    {\'path\': dirpath, \'ex\': str(e)})\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663, in _supports_direct_io\n    f = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/.directio.test'\n"]
2013-07-24 11:22:38.103 ERROR nova.compute.manager [req-26522972-c8be-4be5-83d1-f45412718d94 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Instance failed to spawn
2013-07-24 11:22:50.791 1685 ERROR nova.virt.libvirt.driver [-] [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] During wait destroy, instance disappeared.
2013-07-24 11:22:50.925 ERROR nova.compute.manager [req-26522972-c8be-4be5-83d1-f45412718d94 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn\n    admin_pass=admin_password)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree\n    os.makedirs(path)\n', '  File "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name, mode)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'\n"]
2013-07-24 11:26:00.874 ERROR nova.compute.manager [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Instance failed to spawn
2013-07-24 11:27:07.567 ERROR nova.compute.manager [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn\n    admin_pass=admin_password)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree\n    os.makedirs(path)\n', '  File "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name, mode)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'\n"]
2013-07-24 11:27:08.045 ERROR nova.openstack.common.rpc.amqp [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
2013-07-24 11:34:02.320 1685 ERROR nova.virt.libvirt.driver [-] [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] During wait destroy, instance disappeared.
2013-07-24 11:38:23.618 ERROR nova.compute.manager [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 69a2c512-bea1-46de-91be-c33a07799981] Instance failed to spawn
2013-07-24 11:38:49.087 1685 ERROR nova.virt.libvirt.driver [-] [instance: 69a2c512-bea1-46de-91be-c33a07799981] During wait destroy, instance disappeared.
2013-07-24 11:38:49.220 ERROR nova.compute.manager [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 69a2c512-bea1-46de-91be-c33a07799981] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn\n    admin_pass=admin_password)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '  File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree\n    os.makedirs(path)\n', '  File "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name, mode)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'\n"]
2013-07-24 11:41:15.017 1685 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 12:45:36.353 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 13:13:09.224 ERROR nova.virt.libvirt.driver [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission denied: '/var/lib/nova/instances/.directio.test''
2013-07-24 13:13:09.225 ERROR nova.compute.manager [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Instance failed to spawn
2013-07-24 13:13:21.636 1694 ERROR nova.virt.libvirt.driver [-] [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] During wait destroy, instance disappeared.
2013-07-24 13:13:21.729 ERROR nova.compute.manager [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510, in spawn\n    block_device_info=block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326, in to_xml\n    disk_info, rescue, block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244, in get_guest_config\n    inst_type):\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051, in get_guest_storage_config\n    inst_type)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013, in get_guest_disk_config\n    self.disk_cachemode,\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355, in disk_cachemode\n    if not self._supports_direct_io(CONF.instances_path):\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676, in _supports_direct_io\n    {\'path\': dirpath, \'ex\': str(e)})\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663, in _supports_direct_io\n    f = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/.directio.test'\n"]
2013-07-24 13:25:36.702 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 14:05:37.095 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 14:45:37.319 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 14:47:05.364 ERROR nova.compute.manager [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Instance failed to spawn
2013-07-24 14:47:26.986 ERROR nova.compute.manager [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network\n    domain = self._create_domain(xml, instance=instance)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n    result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n    rv = meth(*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags\n    if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: internal error cannot load AppArmor profile 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'\n"]
2013-07-24 15:25:37.532 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 16:05:37.847 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 16:14:39.249 ERROR nova.compute.manager [req-e0716278-f864-4f70-962a-bf74559fc4c9 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Instance failed to spawn
2013-07-24 16:14:54.459 ERROR nova.compute.manager [req-e0716278-f864-4f70-962a-bf74559fc4c9 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network\n    domain = self._create_domain(xml, instance=instance)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n    result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n    rv = meth(*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags\n    if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: internal error cannot load AppArmor profile 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'\n"]
2013-07-24 16:43:47.486 ERROR nova.compute.manager [req-5d9b9575-a6eb-4324-b975-78dba69c24ef c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Instance failed to spawn
2013-07-24 16:43:58.434 ERROR nova.compute.manager [req-5d9b9575-a6eb-4324-b975-78dba69c24ef c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network\n    domain = self._create_domain(xml, instance=instance)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n    result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n    rv = meth(*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags\n    if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: internal error cannot load AppArmor profile 'libvirt-00497f97-a0fe-4a51-b7f6-6901d0d34791'\n"]
2013-07-24 16:46:43.408 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 17:26:43.627 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 17:47:46.003 ERROR nova.compute.manager [req-370ac716-203d-43aa-a146-709297c715b3 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 8a4e0c84-07ec-40ab-9585-5433dff92b64] Instance failed to spawn
2013-07-24 17:47:55.194 ERROR nova.compute.manager [req-370ac716-203d-43aa-a146-709297c715b3 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 8a4e0c84-07ec-40ab-9585-5433dff92b64] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network\n    domain = self._create_domain(xml, instance=instance)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n    result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n    rv = meth(*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags\n    if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: internal error cannot load AppArmor profile 'libvirt-8a4e0c84-07ec-40ab-9585-5433dff92b64'\n"]
2013-07-24 17:53:42.789 ERROR nova.compute.manager [req-153949aa-d5a7-473f-b79b-38616ae77441 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Instance failed to spawn
2013-07-24 17:54:05.061 ERROR nova.compute.manager [req-153949aa-d5a7-473f-b79b-38616ae77441 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network\n    domain = self._create_domain(xml, instance=instance)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n    result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n    rv = meth(*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags\n    if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: internal error cannot load AppArmor profile 'libvirt-d3bfc215-7e24-40d2-b6f9-a1542a2b9d05'\n"]
2013-07-24 18:05:44.677 ERROR nova.compute.manager [req-60486a96-54a1-4472-846b-b57209c68921 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 005f7640-a8ce-429a-ba91-85e7db1d86ac] Instance failed to spawn
2013-07-24 18:06:14.408 ERROR nova.compute.manager [req-60486a96-54a1-4472-846b-b57209c68921 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 005f7640-a8ce-429a-ba91-85e7db1d86ac] Error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n    set_access_ip=set_access_ip)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n    LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n    self.gen.next()\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn\n    block_device_info)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network\n    domain = self._create_domain(xml, instance=instance)\n', '  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n    result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n    rv = meth(*args,**kwargs)\n', '  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags\n    if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: internal error cannot load AppArmor profile 'libvirt-005f7640-a8ce-429a-ba91-85e7db1d86ac'\n"]




cat /var/log/nova/nova-*.log | grep -E TRACE
2013-07-24 08:03:57.262 1820 TRACE nova.manager Traceback (most recent call last):
2013-07-24 08:03:57.262 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 08:03:57.262 1820 TRACE nova.manager     task(self, context)
2013-07-24 08:03:57.262 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 08:03:57.262 1820 TRACE nova.manager     storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 08:03:57.262 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 08:03:57.262 1820 TRACE nova.manager     retval = f(*args, **kwargs)
2013-07-24 08:03:57.262 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 08:03:57.262 1820 TRACE nova.manager     with open(id_path, 'w') as f:
2013-07-24 08:03:57.262 1820 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 08:03:57.262 1820 TRACE nova.manager 
2013-07-24 08:43:57.655 1820 TRACE nova.manager Traceback (most recent call last):
2013-07-24 08:43:57.655 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 08:43:57.655 1820 TRACE nova.manager     task(self, context)
2013-07-24 08:43:57.655 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 08:43:57.655 1820 TRACE nova.manager     storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 08:43:57.655 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 08:43:57.655 1820 TRACE nova.manager     retval = f(*args, **kwargs)
2013-07-24 08:43:57.655 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 08:43:57.655 1820 TRACE nova.manager     with open(id_path, 'w') as f:
2013-07-24 08:43:57.655 1820 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 08:43:57.655 1820 TRACE nova.manager 
2013-07-24 09:23:57.816 1820 TRACE nova.manager Traceback (most recent call last):
2013-07-24 09:23:57.816 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 09:23:57.816 1820 TRACE nova.manager     task(self, context)
2013-07-24 09:23:57.816 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 09:23:57.816 1820 TRACE nova.manager     storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 09:23:57.816 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 09:23:57.816 1820 TRACE nova.manager     retval = f(*args, **kwargs)
2013-07-24 09:23:57.816 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 09:23:57.816 1820 TRACE nova.manager     with open(id_path, 'w') as f:
2013-07-24 09:23:57.816 1820 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 09:23:57.816 1820 TRACE nova.manager 
2013-07-24 10:03:57.993 1820 TRACE nova.manager Traceback (most recent call last):
2013-07-24 10:03:57.993 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 10:03:57.993 1820 TRACE nova.manager     task(self, context)
2013-07-24 10:03:57.993 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 10:03:57.993 1820 TRACE nova.manager     storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 10:03:57.993 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 10:03:57.993 1820 TRACE nova.manager     retval = f(*args, **kwargs)
2013-07-24 10:03:57.993 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 10:03:57.993 1820 TRACE nova.manager     with open(id_path, 'w') as f:
2013-07-24 10:03:57.993 1820 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 10:03:57.993 1820 TRACE nova.manager 
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Traceback (most recent call last):
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]     block_device_info)
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]     admin_pass=admin_password)
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]     fileutils.ensure_tree(basepath(suffix=''))
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]     os.makedirs(path)
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]   File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]     mkdir(name, mode)
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] 
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Traceback (most recent call last):
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]     block_device_info)
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]     admin_pass=admin_password)
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]     fileutils.ensure_tree(basepath(suffix=''))
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]     os.makedirs(path)
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]   File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]     mkdir(name, mode)
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] 
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Traceback (most recent call last):
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]     block_device_info)
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]     admin_pass=admin_password)
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]     fileutils.ensure_tree(basepath(suffix=''))
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]     os.makedirs(path)
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]   File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]     mkdir(name, mode)
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] 
2013-07-24 10:43:58.194 1820 TRACE nova.manager Traceback (most recent call last):
2013-07-24 10:43:58.194 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 10:43:58.194 1820 TRACE nova.manager     task(self, context)
2013-07-24 10:43:58.194 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 10:43:58.194 1820 TRACE nova.manager     storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 10:43:58.194 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 10:43:58.194 1820 TRACE nova.manager     retval = f(*args, **kwargs)
2013-07-24 10:43:58.194 1820 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 10:43:58.194 1820 TRACE nova.manager     with open(id_path, 'w') as f:
2013-07-24 10:43:58.194 1820 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 10:43:58.194 1820 TRACE nova.manager 
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Traceback (most recent call last):
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     block_device_info)
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510, in spawn
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     block_device_info=block_device_info)
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326, in to_xml
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     disk_info, rescue, block_device_info)
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244, in get_guest_config
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     inst_type):
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051, in get_guest_storage_config
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     inst_type)
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013, in get_guest_disk_config
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     self.disk_cachemode,
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355, in disk_cachemode
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     if not self._supports_direct_io(CONF.instances_path):
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676, in _supports_direct_io
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     {'path': dirpath, 'ex': str(e)})
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     self.gen.next()
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663, in _supports_direct_io
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     f = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/.directio.test'
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] 
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Traceback (most recent call last):
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]     block_device_info)
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]     admin_pass=admin_password)
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]     fileutils.ensure_tree(basepath(suffix=''))
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]     os.makedirs(path)
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]   File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]     mkdir(name, mode)
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] 
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Traceback (most recent call last):
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]     block_device_info)
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]     admin_pass=admin_password)
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]     fileutils.ensure_tree(basepath(suffix=''))
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]     os.makedirs(path)
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]   File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]     mkdir(name, mode)
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] 
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in decorated_function
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     pass
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in decorated_function
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1210, in run_instance
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     do_run_instance()
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1209, in do_run_instance
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     admin_password, is_first_time, node, instance)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 865, in _run_instance
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     self._set_instance_error_state(context, instance['uuid'])
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 857, in _run_instance
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     filter_properties, bdms)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 897, in _reschedule_or_reraise
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     self._log_original_error(exc_info, instance_uuid)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 892, in _reschedule_or_reraise
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     self._shutdown_instance(context, instance, bdms)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1222, in _shutdown_instance
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     network_info = self._get_instance_nw_info(context, instance)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 667, in _get_instance_nw_info
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     instance, conductor_api=self.conductor_api)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 359, in get_instance_nw_info
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     result = self._get_instance_nw_info(context, instance, networks)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 367, in _get_instance_nw_info
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     nw_info = self._build_network_info_model(context, instance, networks)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 778, in _build_network_info_model
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     data = client.list_ports(**search_opts)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     ret = self.function(instance, *args, **kwargs)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 255, in list_ports
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     **_params)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 996, in list
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     for r in self._pagination(collection, path, **params):
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 1009, in _pagination
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     res = self.get(path, params=params)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 982, in get
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 967, in retry_request
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     headers=headers, params=params)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     self._handle_fault_response(status_code, replybody)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     exception_handler_v20(status_code, des_error_body)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp     message=message)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp QuantumClientException: Request Timeout
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp 
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] Traceback (most recent call last):
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981]     block_device_info)
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981]     admin_pass=admin_password)
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981]     fileutils.ensure_tree(basepath(suffix=''))
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981]     os.makedirs(path)
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981]   File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981]     mkdir(name, mode)
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] 
2013-07-24 11:41:15.017 1685 TRACE nova.manager Traceback (most recent call last):
2013-07-24 11:41:15.017 1685 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 11:41:15.017 1685 TRACE nova.manager     task(self, context)
2013-07-24 11:41:15.017 1685 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 11:41:15.017 1685 TRACE nova.manager     storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 11:41:15.017 1685 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 11:41:15.017 1685 TRACE nova.manager     retval = f(*args, **kwargs)
2013-07-24 11:41:15.017 1685 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 11:41:15.017 1685 TRACE nova.manager     with open(id_path, 'w') as f:
2013-07-24 11:41:15.017 1685 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 11:41:15.017 1685 TRACE nova.manager 
2013-07-24 12:45:36.353 1694 TRACE nova.manager Traceback (most recent call last):
2013-07-24 12:45:36.353 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 12:45:36.353 1694 TRACE nova.manager     task(self, context)
2013-07-24 12:45:36.353 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 12:45:36.353 1694 TRACE nova.manager     storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 12:45:36.353 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 12:45:36.353 1694 TRACE nova.manager     retval = f(*args, **kwargs)
2013-07-24 12:45:36.353 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 12:45:36.353 1694 TRACE nova.manager     with open(id_path, 'w') as f:
2013-07-24 12:45:36.353 1694 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 12:45:36.353 1694 TRACE nova.manager 
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Traceback (most recent call last):
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]     block_device_info)
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510, in spawn
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]     block_device_info=block_device_info)
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326, in to_xml
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]     disk_info, rescue, block_device_info)
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244, in get_guest_config
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]     inst_type):
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051, in get_guest_storage_config
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]     inst_type)
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013, in get_guest_disk_config
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]     self.disk_cachemode,
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355, in disk_cachemode
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]     if not self._supports_direct_io(CONF.instances_path):
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676, in _supports_direct_io
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]     {'path': dirpath, 'ex': str(e)})
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]     self.gen.next()
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663, in _supports_direct_io
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]     f = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/.directio.test'
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] 
2013-07-24 13:25:36.702 1694 TRACE nova.manager Traceback (most recent call last):
2013-07-24 13:25:36.702 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 13:25:36.702 1694 TRACE nova.manager     task(self, context)
2013-07-24 13:25:36.702 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 13:25:36.702 1694 TRACE nova.manager     storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 13:25:36.702 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 210, in inner
2013-07-24 13:25:36.702 1694 TRACE nova.manager     fileutils.ensure_tree(local_lock_path)
2013-07-24 13:25:36.702 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 13:25:36.702 1694 TRACE nova.manager     os.makedirs(path)
2013-07-24 13:25:36.702 1694 TRACE nova.manager   File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 13:25:36.702 1694 TRACE nova.manager     mkdir(name, mode)
2013-07-24 13:25:36.702 1694 TRACE nova.manager OSError: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 13:25:36.702 1694 TRACE nova.manager 
2013-07-24 14:05:37.095 1694 TRACE nova.manager Traceback (most recent call last):
2013-07-24 14:05:37.095 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 14:05:37.095 1694 TRACE nova.manager     task(self, context)
2013-07-24 14:05:37.095 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 14:05:37.095 1694 TRACE nova.manager     storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 14:05:37.095 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 210, in inner
2013-07-24 14:05:37.095 1694 TRACE nova.manager     fileutils.ensure_tree(local_lock_path)
2013-07-24 14:05:37.095 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 14:05:37.095 1694 TRACE nova.manager     os.makedirs(path)
2013-07-24 14:05:37.095 1694 TRACE nova.manager   File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 14:05:37.095 1694 TRACE nova.manager     mkdir(name, mode)
2013-07-24 14:05:37.095 1694 TRACE nova.manager OSError: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 14:05:37.095 1694 TRACE nova.manager 
2013-07-24 14:45:37.319 1694 TRACE nova.manager Traceback (most recent call last):
2013-07-24 14:45:37.319 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 14:45:37.319 1694 TRACE nova.manager     task(self, context)
2013-07-24 14:45:37.319 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 14:45:37.319 1694 TRACE nova.manager     storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 14:45:37.319 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 210, in inner
2013-07-24 14:45:37.319 1694 TRACE nova.manager     fileutils.ensure_tree(local_lock_path)
2013-07-24 14:45:37.319 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 14:45:37.319 1694 TRACE nova.manager     os.makedirs(path)
2013-07-24 14:45:37.319 1694 TRACE nova.manager   File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 14:45:37.319 1694 TRACE nova.manager     mkdir(name, mode)
2013-07-24 14:45:37.319 1694 TRACE nova.manager OSError: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 14:45:37.319 1694 TRACE nova.manager 
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Traceback (most recent call last):
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     block_device_info)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     block_device_info)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     domain = self._create_domain(xml, instance=instance)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     domain.createWithFlags(launch_flags)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     rv = execute(f,*args,**kwargs)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     rv = meth(*args,**kwargs)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] libvirtError: internal error cannot load AppArmor profile 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] 
2013-07-24 15:25:37.532 1694 TRACE nova.manager Traceback (most recent call last):
2013-07-24 15:25:37.532 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 15:25:37.532 1694 TRACE nova.manager     task(self, context)
2013-07-24 15:25:37.532 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 15:25:37.532 1694 TRACE nova.manager     storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 15:25:37.532 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 210, in inner
2013-07-24 15:25:37.532 1694 TRACE nova.manager     fileutils.ensure_tree(local_lock_path)
2013-07-24 15:25:37.532 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 15:25:37.532 1694 TRACE nova.manager     os.makedirs(path)
2013-07-24 15:25:37.532 1694 TRACE nova.manager   File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 15:25:37.532 1694 TRACE nova.manager     mkdir(name, mode)
2013-07-24 15:25:37.532 1694 TRACE nova.manager OSError: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 15:25:37.532 1694 TRACE nova.manager 
2013-07-24 16:05:37.847 1694 TRACE nova.manager Traceback (most recent call last):
2013-07-24 16:05:37.847 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 16:05:37.847 1694 TRACE nova.manager     task(self, context)
2013-07-24 16:05:37.847 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 16:05:37.847 1694 TRACE nova.manager     storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 16:05:37.847 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 210, in inner
2013-07-24 16:05:37.847 1694 TRACE nova.manager     fileutils.ensure_tree(local_lock_path)
2013-07-24 16:05:37.847 1694 TRACE nova.manager   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 16:05:37.847 1694 TRACE nova.manager     os.makedirs(path)
2013-07-24 16:05:37.847 1694 TRACE nova.manager   File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 16:05:37.847 1694 TRACE nova.manager     mkdir(name, mode)
2013-07-24 16:05:37.847 1694 TRACE nova.manager OSError: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 16:05:37.847 1694 TRACE nova.manager 
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Traceback (most recent call last):
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]     block_device_info)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]     block_device_info)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]     domain = self._create_domain(xml, instance=instance)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]     domain.createWithFlags(launch_flags)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]     rv = execute(f,*args,**kwargs)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]     rv = meth(*args,**kwargs)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] libvirtError: internal error cannot load AppArmor profile 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] 
2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Traceback (most recent call last):
2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791]     block_device_info)
2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn
2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791]   
...