openstack team mailing list archive

Thread
Date
Re: Configuring Live Migration

To: Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
From: Filipe Manco <filipe.manco@xxxxxxxxx>
Date: Fri, 26 Jul 2013 10:53:01 +0100
Cc: "openstack@xxxxxxxxxxxxxxxxxxx" <openstack@xxxxxxxxxxxxxxxxxxx>
In-reply-to: <CAKz5Ux-umx3LGKQbRMAM0TGnr=nv3NvpBfXbHfSDtUtwyB79xw@mail.gmail.com>
Hi

This error means that you cannot ssh into the new compute node because it's
public key cannot be verified. Add the public keys of the compute nodes to
/etc/ssh/know_hosts file (or the nova one, usually under
/var/lib/nova/.ssh/) and the problem should be gone.

Regards

Filipe Manco
http://about.me/fmanco


2013/7/26 Guilherme Russi <luisguilherme.cr@xxxxxxxxx>

> Ok, I'll look more carefully the log and it's cool this command from
> libvirt, I didn't know it, I'll try it and return to you.
>
> Thank you again.
>
>
> 2013/7/25 Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>
>
>> **
>> That's confused me, "Unexpected error while running command" will give
>> us what command is running when this error occured, you may check the log
>> more carefully.
>> and you also can migrate the VM directly by libvirt command line like
>> below:
>> virsh  migrate --live --persistent --undefinesource --verbose
>> --domain ${instance-uuid} --desturi qemu+tcp://${the-dest-host-ip}/system
>> then see the output result.
>>
>> 2013-07-26
>>  ------------------------------
>>  Wangpan
>>  ------------------------------
>>  *发件人：*Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>> *发送时间：*2013-07-26 09:30
>> *主题：*Re: Re: Re: Re: Re: [Openstack] Configuring Live Migration
>> *收件人：*"Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>
>> *抄送：*"openstack@xxxxxxxxxxxxxxxxxxx"<openstack@xxxxxxxxxxxxxxxxxxx>
>>
>>  When I enter the action to migrate, the only two errors showen at this
>> period are these two.
>>
>>
>> 2013/7/25 Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>
>>
>>> **
>>> There may have more error logs at this minute 2013-07-25 17:28,
>>> you can get the backtrace near the error log you post.
>>>
>>> 2013-07-26
>>>  ------------------------------
>>>  Wangpan
>>>  ------------------------------
>>>  *发件人：*Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>> *发送时间：*2013-07-26 04:48
>>> *主题：*Re: Re: Re: Re: [Openstack] Configuring Live Migration
>>> *收件人：*"Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>,"
>>> openstack@xxxxxxxxxxxxxxxxxxx"<openstack@xxxxxxxxxxxxxxxxxxx>
>>> *抄送：*
>>>
>>>  Hello guys, I'm getting problems when trying to migrate a VM.
>>>
>>> I'm getting this error on my log:
>>>
>>>  2013-07-25 17:28:31.582 ERROR nova.compute.manager
>>> [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4 c402785616534f2096b34ce132b7d3f2
>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>> 919f13d2-8116-4714-b2ec-d2e0f3d3c910] Unexpected error while running
>>> command.
>>> Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state
>>> to ERROR
>>> 2013-07-25 17:28:32.212 ERROR nova.openstack.common.rpc.amqp
>>> [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4 c402785616534f2096b34ce132b7d3f2
>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>
>>>
>>> The user Wangpan was helping me, but I guess his hour is different from
>>> mine, can anybody five me an idea?
>>>
>>> Thank you.
>>>
>>> Guilherme.
>>>
>>>
>>> 2013/7/25 Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>>
>>>> Hello Wangpan, I've looking at the logs and I saw one error with my ssh
>>>> port, and then I fixed the port, but now I'm getting this error:
>>>>
>>>>  2013-07-25 14:09:50.388 ERROR nova.compute.manager
>>>> [req-6930a2e2-60e7-421f-a560-26a626ecebcb c402785616534f2096b34ce132b7d3f2
>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>> d268d24b-96fc-4450-88d4-19ba83cd8892] Unexpected error while running
>>>> command.
>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state
>>>> to ERROR
>>>> 2013-07-25 14:09:51.005 ERROR nova.openstack.common.rpc.amqp
>>>> [req-6930a2e2-60e7-421f-a560-26a626ecebcb c402785616534f2096b34ce132b7d3f2
>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>
>>>> What "host key verification"? Where should I put this key?
>>>>
>>>> Thank you.
>>>>
>>>> Guilherme.
>>>>
>>>>
>>>> 2013/7/25 Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>>>
>>>>> Typed wrong here
>>>>> Em 25/07/2013 00:42, "Wangpan" <hzwangpan@xxxxxxxxxxxxxxxx> escreveu:
>>>>>
>>>>>> **
>>>>>> chmod 777 -R /var/lib/nova/instances
>>>>>> not the /var/log/nova/instances
>>>>>>
>>>>>> 2013-07-25
>>>>>>  ------------------------------
>>>>>>  Wangpan
>>>>>>  ------------------------------
>>>>>>  *发件人：*Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>>>>> *发送时间：*2013-07-25 10:39
>>>>>> *主题：*Re: Re: Re: [Openstack] Configuring Live Migration
>>>>>> *收件人：*"Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>
>>>>>> *抄送：*
>>>>>>
>>>>>>  I've done chmod 777 -R /var/log/nova/instances
>>>>>>
>>>>>> Does it work?
>>>>>>
>>>>>>
>>>>>> 2013/7/24 Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>
>>>>>>
>>>>>>> **
>>>>>>> I saw this error "Permission denied: '/var/lib/nova/instances/'" in
>>>>>>> your logs.
>>>>>>> you may need to chown the instances path  '/var/lib/nova/instances/'
>>>>>>>
>>>>>>> 2013-07-25
>>>>>>>  ------------------------------
>>>>>>>  Wangpan
>>>>>>>  ------------------------------
>>>>>>>  *发件人：*Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>>>>>> *发送时间：*2013-07-25 10:08
>>>>>>> *主题：*Re: Re: [Openstack] Configuring Live Migration
>>>>>>> *收件人：*"Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>
>>>>>>> *抄送：*
>>>>>>>
>>>>>>>  Compute Node 01:
>>>>>>>
>>>>>>>  cat /var/log/nova/nova-*.log | grep -E ERROR
>>>>>>> 2013-07-24 08:03:57.262 1820 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 08:43:57.655 1820 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 09:23:57.816 1820 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 10:03:57.993 1820 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 10:20:21.767 ERROR nova.compute.manager
>>>>>>> [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Instance failed to spawn
>>>>>>> 2013-07-24 10:20:29.326 1820 ERROR nova.virt.libvirt.driver [-]
>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] During wait destroy,
>>>>>>> instance disappeared.
>>>>>>> 2013-07-24 10:20:29.511 ERROR nova.compute.manager
>>>>>>> [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'\n"]
>>>>>>> 2013-07-24 10:33:17.279 ERROR nova.compute.manager
>>>>>>> [req-f10c5f14-b3ad-4c13-b20f-c53f57379163 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] Instance failed to spawn
>>>>>>> 2013-07-24 10:33:36.673 1820 ERROR nova.virt.libvirt.driver [-]
>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] During wait destroy,
>>>>>>> instance disappeared.
>>>>>>> 2013-07-24 10:33:36.837 ERROR nova.compute.manager
>>>>>>> [req-f10c5f14-b3ad-4c13-b20f-c53f57379163 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'\n"]
>>>>>>> 2013-07-24 10:40:26.722 ERROR nova.compute.manager
>>>>>>> [req-5d0289b3-36d7-4046-996c-a1decb922066 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Instance failed to spawn
>>>>>>> 2013-07-24 10:40:39.402 1820 ERROR nova.virt.libvirt.driver [-]
>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] During wait destroy,
>>>>>>> instance disappeared.
>>>>>>> 2013-07-24 10:40:39.530 ERROR nova.compute.manager
>>>>>>> [req-5d0289b3-36d7-4046-996c-a1decb922066 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'\n"]
>>>>>>> 2013-07-24 10:43:58.194 1820 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 11:03:19.657 ERROR nova.virt.libvirt.driver
>>>>>>> [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Error on '/var/lib/nova/instances' while
>>>>>>> checking direct I/O: '[Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/.directio.test''
>>>>>>> 2013-07-24 11:03:19.658 ERROR nova.compute.manager
>>>>>>> [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Instance failed to spawn
>>>>>>> 2013-07-24 11:03:32.036 1685 ERROR nova.virt.libvirt.driver [-]
>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] During wait destroy,
>>>>>>> instance disappeared.
>>>>>>> 2013-07-24 11:03:32.170 ERROR nova.compute.manager
>>>>>>> [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>> in spawn\n    block_device_info=block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>> in to_xml\n    disk_info, rescue, block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>> in get_guest_config\n    inst_type):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>> in get_guest_storage_config\n    inst_type)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>> in get_guest_disk_config\n    self.disk_cachemode,\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>> in disk_cachemode\n    if not
>>>>>>> self._supports_direct_io(CONF.instances_path):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>> in _supports_direct_io\n    {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>  self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>> in _supports_direct_io\n    f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>> 2013-07-24 11:22:38.103 ERROR nova.compute.manager
>>>>>>> [req-26522972-c8be-4be5-83d1-f45412718d94 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Instance failed to spawn
>>>>>>> 2013-07-24 11:22:50.791 1685 ERROR nova.virt.libvirt.driver [-]
>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] During wait destroy,
>>>>>>> instance disappeared.
>>>>>>> 2013-07-24 11:22:50.925 ERROR nova.compute.manager
>>>>>>> [req-26522972-c8be-4be5-83d1-f45412718d94 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'\n"]
>>>>>>> 2013-07-24 11:26:00.874 ERROR nova.compute.manager
>>>>>>> [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Instance failed to spawn
>>>>>>> 2013-07-24 11:27:07.567 ERROR nova.compute.manager
>>>>>>> [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'\n"]
>>>>>>> 2013-07-24 11:27:08.045 ERROR nova.openstack.common.rpc.amqp
>>>>>>> [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>>> 2013-07-24 11:34:02.320 1685 ERROR nova.virt.libvirt.driver [-]
>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] During wait destroy,
>>>>>>> instance disappeared.
>>>>>>> 2013-07-24 11:38:23.618 ERROR nova.compute.manager
>>>>>>> [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] Instance failed to spawn
>>>>>>> 2013-07-24 11:38:49.087 1685 ERROR nova.virt.libvirt.driver [-]
>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] During wait destroy,
>>>>>>> instance disappeared.
>>>>>>> 2013-07-24 11:38:49.220 ERROR nova.compute.manager
>>>>>>> [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'\n"]
>>>>>>> 2013-07-24 11:41:15.017 1685 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 12:45:36.353 1694 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 13:13:09.224 ERROR nova.virt.libvirt.driver
>>>>>>> [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Error on '/var/lib/nova/instances' while
>>>>>>> checking direct I/O: '[Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/.directio.test''
>>>>>>> 2013-07-24 13:13:09.225 ERROR nova.compute.manager
>>>>>>> [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] Instance failed to spawn
>>>>>>> 2013-07-24 13:13:21.636 1694 ERROR nova.virt.libvirt.driver [-]
>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] During wait destroy,
>>>>>>> instance disappeared.
>>>>>>> 2013-07-24 13:13:21.729 ERROR nova.compute.manager
>>>>>>> [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>> in spawn\n    block_device_info=block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>> in to_xml\n    disk_info, rescue, block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>> in get_guest_config\n    inst_type):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>> in get_guest_storage_config\n    inst_type)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>> in get_guest_disk_config\n    self.disk_cachemode,\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>> in disk_cachemode\n    if not
>>>>>>> self._supports_direct_io(CONF.instances_path):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>> in _supports_direct_io\n    {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>  self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>> in _supports_direct_io\n    f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>> 2013-07-24 13:25:36.702 1694 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lock/nova'
>>>>>>> 2013-07-24 14:05:37.095 1694 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lock/nova'
>>>>>>> 2013-07-24 14:45:37.319 1694 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lock/nova'
>>>>>>> 2013-07-24 14:47:05.364 ERROR nova.compute.manager
>>>>>>> [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Instance failed to spawn
>>>>>>> 2013-07-24 14:47:26.986 ERROR nova.compute.manager
>>>>>>> [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>> instance=instance)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>> internal error cannot load AppArmor profile
>>>>>>> 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'\n"]
>>>>>>> 2013-07-24 15:25:37.532 1694 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lock/nova'
>>>>>>> 2013-07-24 16:05:37.847 1694 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lock/nova'
>>>>>>> 2013-07-24 16:14:39.249 ERROR nova.compute.manager
>>>>>>> [req-e0716278-f864-4f70-962a-bf74559fc4c9 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913] Instance failed to spawn
>>>>>>> 2013-07-24 16:14:54.459 ERROR nova.compute.manager
>>>>>>> [req-e0716278-f864-4f70-962a-bf74559fc4c9 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>> instance=instance)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>> internal error cannot load AppArmor profile
>>>>>>> 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'\n"]
>>>>>>> 2013-07-24 16:43:47.486 ERROR nova.compute.manager
>>>>>>> [req-5d9b9575-a6eb-4324-b975-78dba69c24ef c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791] Instance failed to spawn
>>>>>>> 2013-07-24 16:43:58.434 ERROR nova.compute.manager
>>>>>>> [req-5d9b9575-a6eb-4324-b975-78dba69c24ef c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>> instance=instance)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>> internal error cannot load AppArmor profile
>>>>>>> 'libvirt-00497f97-a0fe-4a51-b7f6-6901d0d34791'\n"]
>>>>>>> 2013-07-24 16:46:43.408 1694 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lock/nova'
>>>>>>> 2013-07-24 17:26:43.627 1694 ERROR nova.manager [-] Error during
>>>>>>> ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied:
>>>>>>> '/var/lock/nova'
>>>>>>> 2013-07-24 17:47:46.003 ERROR nova.compute.manager
>>>>>>> [req-370ac716-203d-43aa-a146-709297c715b3 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 8a4e0c84-07ec-40ab-9585-5433dff92b64] Instance failed to spawn
>>>>>>> 2013-07-24 17:47:55.194 ERROR nova.compute.manager
>>>>>>> [req-370ac716-203d-43aa-a146-709297c715b3 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 8a4e0c84-07ec-40ab-9585-5433dff92b64] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>> instance=instance)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>> internal error cannot load AppArmor profile
>>>>>>> 'libvirt-8a4e0c84-07ec-40ab-9585-5433dff92b64'\n"]
>>>>>>> 2013-07-24 17:53:42.789 ERROR nova.compute.manager
>>>>>>> [req-153949aa-d5a7-473f-b79b-38616ae77441 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Instance failed to spawn
>>>>>>> 2013-07-24 17:54:05.061 ERROR nova.compute.manager
>>>>>>> [req-153949aa-d5a7-473f-b79b-38616ae77441 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>> instance=instance)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>> internal error cannot load AppArmor profile
>>>>>>> 'libvirt-d3bfc215-7e24-40d2-b6f9-a1542a2b9d05'\n"]
>>>>>>> 2013-07-24 18:05:44.677 ERROR nova.compute.manager
>>>>>>> [req-60486a96-54a1-4472-846b-b57209c68921 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 005f7640-a8ce-429a-ba91-85e7db1d86ac] Instance failed to spawn
>>>>>>> 2013-07-24 18:06:14.408 ERROR nova.compute.manager
>>>>>>> [req-60486a96-54a1-4472-846b-b57209c68921 c402785616534f2096b34ce132b7d3f2
>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>> 005f7640-a8ce-429a-ba91-85e7db1d86ac] Error: ['Traceback (most recent call
>>>>>>> last):\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>> instance=instance)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>> internal error cannot load AppArmor profile
>>>>>>> 'libvirt-005f7640-a8ce-429a-ba91-85e7db1d86ac'\n"]
>>>>>>>
>>>>>>>
>>>>>>>  cat /var/log/nova/nova-*.log | grep -E TRACE
>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager Traceback (most
>>>>>>> recent call last):
>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>> periodic_tasks
>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager     task(self,
>>>>>>> context)
>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>> _run_image_cache_manager_pass
>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>> 228, in inner
>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager     retval =
>>>>>>> f(*args, **kwargs)
>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>> register_storage_use
>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager     with
>>>>>>> open(id_path, 'w') as f:
>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager IOError: [Errno 13]
>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager Traceback (most
>>>>>>> recent call last):
>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>> periodic_tasks
>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager     task(self,
>>>>>>> context)
>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>> _run_image_cache_manager_pass
>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>> 228, in inner
>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager     retval =
>>>>>>> f(*args, **kwargs)
>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>> register_storage_use
>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager     with
>>>>>>> open(id_path, 'w') as f:
>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager IOError: [Errno 13]
>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager Traceback (most
>>>>>>> recent call last):
>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>> periodic_tasks
>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager     task(self,
>>>>>>> context)
>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>> _run_image_cache_manager_pass
>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>> 228, in inner
>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager     retval =
>>>>>>> f(*args, **kwargs)
>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>> register_storage_use
>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager     with
>>>>>>> open(id_path, 'w') as f:
>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager IOError: [Errno 13]
>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager Traceback (most
>>>>>>> recent call last):
>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>> periodic_tasks
>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager     task(self,
>>>>>>> context)
>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>> _run_image_cache_manager_pass
>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>> 228, in inner
>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager     retval =
>>>>>>> f(*args, **kwargs)
>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>> register_storage_use
>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager     with
>>>>>>> open(id_path, 'w') as f:
>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager IOError: [Errno 13]
>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Traceback (most recent call last):
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc]     block_device_info)
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>> in spawn
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc]     admin_pass=admin_password)
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>> in _create_image
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc]     os.makedirs(path)
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc]   File "/usr/lib/python2.7/os.py",
>>>>>>> line 157, in makedirs
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc]     mkdir(name, mode)
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] OSError: [Errno 13] Permission
>>>>>>> denied: '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'
>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance:
>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] Traceback (most recent call last):
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd]     block_device_info)
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>> in spawn
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd]     admin_pass=admin_password)
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>> in _create_image
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd]     os.makedirs(path)
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd]   File "/usr/lib/python2.7/os.py",
>>>>>>> line 157, in makedirs
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd]     mkdir(name, mode)
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] OSError: [Errno 13] Permission
>>>>>>> denied: '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'
>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance:
>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Traceback (most recent call last):
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]     block_device_info)
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>> in spawn
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]     admin_pass=admin_password)
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>> in _create_image
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]     os.makedirs(path)
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]   File "/usr/lib/python2.7/os.py",
>>>>>>> line 157, in makedirs
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]     mkdir(name, mode)
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] OSError: [Errno 13] Permission
>>>>>>> denied: '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'
>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance:
>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager Traceback (most
>>>>>>> recent call last):
>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>> periodic_tasks
>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager     task(self,
>>>>>>> context)
>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>> _run_image_cache_manager_pass
>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>> 228, in inner
>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager     retval =
>>>>>>> f(*args, **kwargs)
>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>> register_storage_use
>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager     with
>>>>>>> open(id_path, 'w') as f:
>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager IOError: [Errno 13]
>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Traceback (most recent call last):
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     block_device_info)
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>> in spawn
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>> block_device_info=block_device_info)
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>> in to_xml
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     disk_info, rescue,
>>>>>>> block_device_info)
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>> in get_guest_config
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     inst_type):
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>> in get_guest_storage_config
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     inst_type)
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>> in get_guest_disk_config
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     self.disk_cachemode,
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>> in disk_cachemode
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     if not
>>>>>>> self._supports_direct_io(CONF.instances_path):
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>> in _supports_direct_io
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     {'path': dirpath, 'ex': str(e)})
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File
>>>>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     self.gen.next()
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>> in _supports_direct_io
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     f = os.open(testfile, os.O_CREAT
>>>>>>> | os.O_WRONLY | os.O_DIRECT)
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] OSError: [Errno 13] Permission
>>>>>>> denied: '/var/lib/nova/instances/.directio.test'
>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance:
>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Traceback (most recent call last):
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]     block_device_info)
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>> in spawn
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]     admin_pass=admin_password)
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>> in _create_image
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]     os.makedirs(path)
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]   File "/usr/lib/python2.7/os.py",
>>>>>>> line 157, in makedirs
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]     mkdir(name, mode)
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] OSError: [Errno 13] Permission
>>>>>>> denied: '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'
>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance:
>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Traceback (most recent call last):
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b]     block_device_info)
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>> in spawn
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b]     admin_pass=admin_password)
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>> in _create_image
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b]     os.makedirs(path)
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b]   File "/usr/lib/python2.7/os.py",
>>>>>>> line 157, in makedirs
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b]     mkdir(name, mode)
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] OSError: [Errno 13] Permission
>>>>>>> denied: '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'
>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance:
>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> Traceback (most recent call last):
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>>>>>> line 430, in _process_data
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   rval = self.proxy.dispatch(ctxt, version, method, **args)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>> line 133, in dispatch
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   return getattr(proxyobj, method)(ctxt, **kwargs)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>>>>>> wrapped
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   temp_level, payload)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   self.gen.next()
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>>>>>> wrapped
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   return f(self, context, *args, **kw)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209,
>>>>>>> in decorated_function
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   pass
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   self.gen.next()
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195,
>>>>>>> in decorated_function
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   return function(self, context, *args, **kwargs)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260,
>>>>>>> in decorated_function
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   function(self, context, *args, **kwargs)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237,
>>>>>>> in decorated_function
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   e, sys.exc_info())
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   self.gen.next()
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224,
>>>>>>> in decorated_function
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   return function(self, context, *args, **kwargs)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1210,
>>>>>>> in run_instance
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   do_run_instance()
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py",
>>>>>>> line 242, in inner
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   retval = f(*args, **kwargs)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1209,
>>>>>>> in do_run_instance
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   admin_password, is_first_time, node, instance)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 865,
>>>>>>> in _run_instance
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   self._set_instance_error_state(context, instance['uuid'])
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   self.gen.next()
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 857,
>>>>>>> in _run_instance
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   filter_properties, bdms)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 897,
>>>>>>> in _reschedule_or_reraise
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   self._log_original_error(exc_info, instance_uuid)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   self.gen.next()
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 892,
>>>>>>> in _reschedule_or_reraise
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   self._shutdown_instance(context, instance, bdms)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1222,
>>>>>>> in _shutdown_instance
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   network_info = self._get_instance_nw_info(context, instance)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 667,
>>>>>>> in _get_instance_nw_info
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   instance, conductor_api=self.conductor_api)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line
>>>>>>> 359, in get_instance_nw_info
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   result = self._get_instance_nw_info(context, instance, networks)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line
>>>>>>> 367, in _get_instance_nw_info
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   nw_info = self._build_network_info_model(context, instance, networks)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line
>>>>>>> 778, in _build_network_info_model
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   data = client.list_ports(**search_opts)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line
>>>>>>> 107, in with_params
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   ret = self.function(instance, *args, **kwargs)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line
>>>>>>> 255, in list_ports
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   **_params)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line
>>>>>>> 996, in list
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   for r in self._pagination(collection, path, **params):
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line
>>>>>>> 1009, in _pagination
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   res = self.get(path, params=params)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line
>>>>>>> 982, in get
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   headers=headers, params=params)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line
>>>>>>> 967, in retry_request
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   headers=headers, params=params)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line
>>>>>>> 912, in do_request
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   self._handle_fault_response(status_code, replybody)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line
>>>>>>> 893, in _handle_fault_response
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   exception_handler_v20(status_code, des_error_body)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line
>>>>>>> 87, in exception_handler_v20
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>>   message=message)
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> QuantumClientException: Request Timeout
>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] Traceback (most recent call last):
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981]     block_device_info)
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>> in spawn
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981]     admin_pass=admin_password)
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>> in _create_image
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981]
>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981]     os.makedirs(path)
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981]   File "/usr/lib/python2.7/os.py",
>>>>>>> line 157, in makedirs
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981]     mkdir(name, mode)
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] OSError: [Errno 13] Permission
>>>>>>> denied: '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'
>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance:
>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981]
>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager Traceback (most
>>>>>>> recent call last):
>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>> periodic_tasks
>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager     task(self,
>>>>>>> context)
>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>> _run_image_cache_manager_pass
>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager
>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>> 228, in inner
>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager     retval =
>>>>>>> f(*args, **kwargs)
>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>> register_storage_use
>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager     with
>>>>>>> open(id_path, 'w') as f:
>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager IOError: [Errno 13]
>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager
>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager Traceback (most
>>>>>>> recent call last):
>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>> periodic_tasks
>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager     task(self,
>>>>>>> context)
>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>> _run_image_cache_manager_pass
>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager
>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>> 228, in inner
>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager     retval =
>>>>>>> f(*args, **kwargs)
>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>> register_storage_use
>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager     with
>>>>>>> open(id_path, 'w') as f:
>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager IOError: [Errno 13]
>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] Traceback (most recent call last):
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]     block_device_info)
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>> in spawn
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]
>>>>>>> block_device_info=block_device_info)
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>> in to_xml
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]     disk_info, rescue,
>>>>>>> block_device_info)
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>> in get_guest_config
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]     inst_type):
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>> in get_guest_storage_config
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]     inst_type)
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>> in get_guest_disk_config
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]     self.disk_cachemode,
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>> in disk_cachemode
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]     if not
>>>>>>> self._supports_direct_io(CONF.instances_path):
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>> in _supports_direct_io
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]     {'path': dirpath, 'ex': str(e)})
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]   File
>>>>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]     self.gen.next()
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>> in _supports_direct_io
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]     f = os.open(testfile, os.O_CREAT
>>>>>>> | os.O_WRONLY | os.O_DIRECT)
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] OSError: [Errno 13] Permission
>>>>>>> denied: '/var/lib/nova/instances/.directio.test'
>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance:
>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb]
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager Traceback (most
>>>>>>> recent call last):
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>> periodic_tasks
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager     task(self,
>>>>>>> context)
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>> _run_image_cache_manager_pass
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager
>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>> 210, in inner
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager
>>>>>>> fileutils.ensure_tree(local_lock_path)
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager     os.makedirs(path)
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager     mkdir(name, mode)
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager OSError: [Errno 13]
>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager Traceback (most
>>>>>>> recent call last):
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>> periodic_tasks
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager     task(self,
>>>>>>> context)
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>> _run_image_cache_manager_pass
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager
>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>> 210, in inner
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager
>>>>>>> fileutils.ensure_tree(local_lock_path)
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager     os.makedirs(path)
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager     mkdir(name, mode)
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager OSError: [Errno 13]
>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager Traceback (most
>>>>>>> recent call last):
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>> periodic_tasks
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager     task(self,
>>>>>>> context)
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>> _run_image_cache_manager_pass
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager
>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>> 210, in inner
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager
>>>>>>> fileutils.ensure_tree(local_lock_path)
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager     os.makedirs(path)
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager     mkdir(name, mode)
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager OSError: [Errno 13]
>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Traceback (most recent call last):
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     block_device_info)
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>> in spawn
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     block_device_info)
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>> in _create_domain_and_network
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     domain = self._create_domain(xml,
>>>>>>> instance=instance)
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>> in _create_domain
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]
>>>>>>> domain.createWithFlags(launch_flags)
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     result =
>>>>>>> proxy_call(self._autowrap, f, *args, **kwargs)
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>> proxy_call
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     rv = execute(f,*args,**kwargs)
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     rv = meth(*args,**kwargs)
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]     if ret == -1: raise libvirtError
>>>>>>> ('virDomainCreateWithFlags() failed', dom=self)
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601] libvirtError: internal error cannot
>>>>>>> load AppArmor profile 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'
>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance:
>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601]
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager Traceback (most
>>>>>>> recent call last):
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>> periodic_tasks
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager     task(self,
>>>>>>> context)
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>> _run_image_cache_manager_pass
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager
>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>> 210, in inner
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager
>>>>>>> fileutils.ensure_tree(local_lock_path)
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager     os.makedirs(path)
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager     mkdir(name, mode)
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager OSError: [Errno 13]
>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager Traceback (most
>>>>>>> recent call last):
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>> periodic_tasks
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager     task(self,
>>>>>>> context)
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>> _run_image_cache_manager_pass
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager
>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>> 210, in inner
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager
>>>>>>> fileutils.ensure_tree(local_lock_path)
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>> 29, in ensure_tree
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager     os.makedirs(path)
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager   File
>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager     mkdir(name, mode)
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager OSError: [Errno 13]
>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913] Traceback (most recent call last):
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]     block_device_info)
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>> in spawn
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]     block_device_info)
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>> in _create_domain_and_network
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]     domain = self._create_domain(xml,
>>>>>>> instance=instance)
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>> in _create_domain
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]
>>>>>>> domain.createWithFlags(launch_flags)
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]     result =
>>>>>>> proxy_call(self._autowrap, f, *args, **kwargs)
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>> proxy_call
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]     rv = execute(f,*args,**kwargs)
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]     rv = meth(*args,**kwargs)
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]     if ret == -1: raise libvirtError
>>>>>>> ('virDomainCreateWithFlags() failed', dom=self)
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913] libvirtError: internal error cannot
>>>>>>> load AppArmor profile 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'
>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance:
>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913]
>>>>>>> 2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance:
>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791] Traceback (most recent call last):
>>>>>>> 2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance:
>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>> _spawn
>>>>>>> 2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance:
>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791]     block_device_info)
>>>>>>> 2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance:
>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791]   File
>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>> in spawn
>>>>>>> 2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance:
>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791]
>>>>>>>
>>>>>> ...
>>>>>
>>>>>
>>>>
>>>
>>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
Follow ups

Re: Configuring Live Migration
From: Guilherme Russi, 2013-07-26
References

Configuring Live Migration
From: Guilherme Russi, 2013-07-24
Re: Configuring Live Migration
From: Wangpan, 2013-07-25
Re: Configuring Live Migration
From: Guilherme Russi, 2013-07-25
Re: Configuring Live Migration
From: Wangpan, 2013-07-25
Re: Configuring Live Migration
From: Guilherme Russi, 2013-07-25
Re: Configuring Live Migration
From: Wangpan, 2013-07-26
Re: Configuring Live Migration
From: Guilherme Russi, 2013-07-26
Re: Configuring Live Migration
From: Wangpan, 2013-07-26
Re: Configuring Live Migration
From: Guilherme Russi, 2013-07-26