openstack team mailing list archive
-
openstack team
-
Mailing list archive
-
Message #25498
Re: Configuring Live Migration
About the keys it's up to you. You can use the same or create a new one.
Using the same is easier.
But the nova user public key should go into
/var/lib/nova/.ssh/authorized_keys not into the know_hosts file. The easier
way (as you only have two compute nodes) to add the keys to the know_hosts
is to manually ssh and accept the key.
Filipe Manco
http://about.me/fmanco
2013/7/26 Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
> Ok, Must I create another ssh key to communicate both compute nodes or the
> current key can work? I've put the plublic key from both compute node in
> the know_hosts. Which permission should I apply? 644?
>
> Thank you.
>
>
> 2013/7/26 Filipe Manco <filipe.manco@xxxxxxxxx>
>
>> First you need to check why this "Failed to add the host to the list of
>> known hosts (/var/lib/nova/.ssh/known_hosts)." happens. It maybe file
>> permissions or so. Then you need to generate an ssh key for the user nova
>> and add it to the other compute nodes on the authorized_keys file (the
>> password is not really necessary as soon as you have the keys in place).
>>
>> Filipe Manco
>> http://about.me/fmanco
>>
>>
>> 2013/7/26 Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>
>>> sudo -u nova ssh 192.168.3.3
>>> The authenticity of host '192.168.3.3 (192.168.3.3)' can't be
>>> established.
>>> ECDSA key fingerprint is e3:ca:f7:dd:1e:1f:e3:e3:17:f8:52:09:a4:41:e1:40.
>>> Are you sure you want to continue connecting (yes/no)? yes
>>> Failed to add the host to the list of known hosts
>>> (/var/lib/nova/.ssh/known_hosts).
>>> nova@192.168.3.3's password:
>>>
>>> But I don't know the user nova's password.
>>>
>>>
>>> 2013/7/26 Filipe Manco <filipe.manco@xxxxxxxxx>
>>>
>>>> Can you try *sudo -u nova ssh <remote-compute-node>* form the compute
>>>> node currently running the VM and check if it works?
>>>>
>>>> Filipe Manco
>>>> http://about.me/fmanco
>>>>
>>>>
>>>> 2013/7/26 Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>>>
>>>>> Hello Filipe,
>>>>>
>>>>> I made the copy from my .ssh/know_hosts to places you told me, but
>>>>> I'm still getting the same error. Any other ideas?
>>>>>
>>>>> Thank you.
>>>>>
>>>>>
>>>>> 2013/7/26 Filipe Manco <filipe.manco@xxxxxxxxx>
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> This error means that you cannot ssh into the new compute node
>>>>>> because it's public key cannot be verified. Add the public keys of the
>>>>>> compute nodes to /etc/ssh/know_hosts file (or the nova one, usually under
>>>>>> /var/lib/nova/.ssh/) and the problem should be gone.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Filipe Manco
>>>>>> http://about.me/fmanco
>>>>>>
>>>>>>
>>>>>> 2013/7/26 Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>>>>>
>>>>>>> Ok, I'll look more carefully the log and it's cool this command from
>>>>>>> libvirt, I didn't know it, I'll try it and return to you.
>>>>>>>
>>>>>>> Thank you again.
>>>>>>>
>>>>>>>
>>>>>>> 2013/7/25 Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>
>>>>>>>
>>>>>>>> **
>>>>>>>> That's confused me, "Unexpected error while running command" will
>>>>>>>> give us what command is running when this error occured, you may check the
>>>>>>>> log more carefully.
>>>>>>>> and you also can migrate the VM directly by libvirt command line
>>>>>>>> like below:
>>>>>>>> virsh migrate --live --persistent --undefinesource --verbose
>>>>>>>> --domain ${instance-uuid} --desturi qemu+tcp://${the-dest-host-ip}/system
>>>>>>>> then see the output result.
>>>>>>>>
>>>>>>>> 2013-07-26
>>>>>>>> ------------------------------
>>>>>>>> Wangpan
>>>>>>>> ------------------------------
>>>>>>>> *发件人：*Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>>>>>>> *发送时间：*2013-07-26 09:30
>>>>>>>> *主题：*Re: Re: Re: Re: Re: [Openstack] Configuring Live Migration
>>>>>>>> *收件人：*"Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>
>>>>>>>> *抄送：*"openstack@xxxxxxxxxxxxxxxxxxx"<openstack@xxxxxxxxxxxxxxxxxxx>
>>>>>>>>
>>>>>>>> When I enter the action to migrate, the only two errors showen at
>>>>>>>> this period are these two.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/7/25 Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>
>>>>>>>>
>>>>>>>>> **
>>>>>>>>> There may have more error logs at this minute 2013-07-25 17:28,
>>>>>>>>> you can get the backtrace near the error log you post.
>>>>>>>>>
>>>>>>>>> 2013-07-26
>>>>>>>>> ------------------------------
>>>>>>>>> Wangpan
>>>>>>>>> ------------------------------
>>>>>>>>> *发件人：*Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>>>>>>>> *发送时间：*2013-07-26 04:48
>>>>>>>>> *主题：*Re: Re: Re: Re: [Openstack] Configuring Live Migration
>>>>>>>>> *收件人：*"Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>,"
>>>>>>>>> openstack@xxxxxxxxxxxxxxxxxxx"<openstack@xxxxxxxxxxxxxxxxxxx>
>>>>>>>>> *抄送：*
>>>>>>>>>
>>>>>>>>> Hello guys, I'm getting problems when trying to migrate a VM.
>>>>>>>>>
>>>>>>>>> I'm getting this error on my log:
>>>>>>>>>
>>>>>>>>> 2013-07-25 17:28:31.582 ERROR nova.compute.manager
>>>>>>>>> [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4 c402785616534f2096b34ce132b7d3f2
>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>> 919f13d2-8116-4714-b2ec-d2e0f3d3c910] Unexpected error while running
>>>>>>>>> command.
>>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance
>>>>>>>>> vm_state to ERROR
>>>>>>>>> 2013-07-25 17:28:32.212 ERROR nova.openstack.common.rpc.amqp
>>>>>>>>> [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4 c402785616534f2096b34ce132b7d3f2
>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The user Wangpan was helping me, but I guess his hour is different
>>>>>>>>> from mine, can anybody five me an idea?
>>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>>
>>>>>>>>> Guilherme.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/7/25 Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>>>>>>>>
>>>>>>>>>> Hello Wangpan, I've looking at the logs and I saw one error with
>>>>>>>>>> my ssh port, and then I fixed the port, but now I'm getting this error:
>>>>>>>>>>
>>>>>>>>>> 2013-07-25 14:09:50.388 ERROR nova.compute.manager
>>>>>>>>>> [req-6930a2e2-60e7-421f-a560-26a626ecebcb c402785616534f2096b34ce132b7d3f2
>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>> d268d24b-96fc-4450-88d4-19ba83cd8892] Unexpected error while running
>>>>>>>>>> command.
>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance
>>>>>>>>>> vm_state to ERROR
>>>>>>>>>> 2013-07-25 14:09:51.005 ERROR nova.openstack.common.rpc.amqp
>>>>>>>>>> [req-6930a2e2-60e7-421f-a560-26a626ecebcb c402785616534f2096b34ce132b7d3f2
>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>>>>>>
>>>>>>>>>> What "host key verification"? Where should I put this key?
>>>>>>>>>>
>>>>>>>>>> Thank you.
>>>>>>>>>>
>>>>>>>>>> Guilherme.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/7/25 Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>>>>>>>>>
>>>>>>>>>>> Typed wrong here
>>>>>>>>>>> Em 25/07/2013 00:42, "Wangpan" <hzwangpan@xxxxxxxxxxxxxxxx>
>>>>>>>>>>> escreveu:
>>>>>>>>>>>
>>>>>>>>>>>> **
>>>>>>>>>>>> chmod 777 -R /var/lib/nova/instances
>>>>>>>>>>>> not the /var/log/nova/instances
>>>>>>>>>>>>
>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>> Wangpan
>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>> *发件人：*Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>>>>>>>>>>> *发送时间：*2013-07-25 10:39
>>>>>>>>>>>> *主题：*Re: Re: Re: [Openstack] Configuring Live Migration
>>>>>>>>>>>> *收件人：*"Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>
>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>
>>>>>>>>>>>> I've done chmod 777 -R /var/log/nova/instances
>>>>>>>>>>>>
>>>>>>>>>>>> Does it work?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/7/24 Wangpan <hzwangpan@xxxxxxxxxxxxxxxx>
>>>>>>>>>>>>
>>>>>>>>>>>>> **
>>>>>>>>>>>>> I saw this error "Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/'" in your logs.
>>>>>>>>>>>>> you may need to chown the instances
>>>>>>>>>>>>> path '/var/lib/nova/instances/'
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>> Wangpan
>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>> *发件人：*Guilherme Russi <luisguilherme.cr@xxxxxxxxx>
>>>>>>>>>>>>> *发送时间：*2013-07-25 10:08
>>>>>>>>>>>>> *主题：*Re: Re: [Openstack] Configuring Live Migration
>>>>>>>>>>>>> *收件人：*"Wangpan"<hzwangpan@xxxxxxxxxxxxxxxx>
>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>
>>>>>>>>>>>>> Compute Node 01:
>>>>>>>>>>>>>
>>>>>>>>>>>>> cat /var/log/nova/nova-*.log | grep -E ERROR
>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 10:20:29.326 1820 ERROR nova.virt.libvirt.driver
>>>>>>>>>>>>> [-] [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] During wait destroy,
>>>>>>>>>>>>> instance disappeared.
>>>>>>>>>>>>> 2013-07-24 10:20:29.511 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>> File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'\n"]
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-f10c5f14-b3ad-4c13-b20f-c53f57379163 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 10:33:36.673 1820 ERROR nova.virt.libvirt.driver
>>>>>>>>>>>>> [-] [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] During wait destroy,
>>>>>>>>>>>>> instance disappeared.
>>>>>>>>>>>>> 2013-07-24 10:33:36.837 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-f10c5f14-b3ad-4c13-b20f-c53f57379163 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>> File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'\n"]
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-5d0289b3-36d7-4046-996c-a1decb922066 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 10:40:39.402 1820 ERROR nova.virt.libvirt.driver
>>>>>>>>>>>>> [-] [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] During wait destroy,
>>>>>>>>>>>>> instance disappeared.
>>>>>>>>>>>>> 2013-07-24 10:40:39.530 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-5d0289b3-36d7-4046-996c-a1decb922066 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>> File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'\n"]
>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 11:03:19.657 ERROR nova.virt.libvirt.driver
>>>>>>>>>>>>> [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Error on '/var/lib/nova/instances' while
>>>>>>>>>>>>> checking direct I/O: '[Errno 13] Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 11:03:32.036 1685 ERROR nova.virt.libvirt.driver
>>>>>>>>>>>>> [-] [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] During wait destroy,
>>>>>>>>>>>>> instance disappeared.
>>>>>>>>>>>>> 2013-07-24 11:03:32.170 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>> in spawn\n block_device_info=block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>> in to_xml\n disk_info, rescue, block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>> in get_guest_config\n inst_type):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>> in get_guest_storage_config\n inst_type)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>> in get_guest_disk_config\n self.disk_cachemode,\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>> in disk_cachemode\n if not
>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>> in _supports_direct_io\n {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>> self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>> in _supports_direct_io\n f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-26522972-c8be-4be5-83d1-f45412718d94 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 11:22:50.791 1685 ERROR nova.virt.libvirt.driver
>>>>>>>>>>>>> [-] [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] During wait destroy,
>>>>>>>>>>>>> instance disappeared.
>>>>>>>>>>>>> 2013-07-24 11:22:50.925 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-26522972-c8be-4be5-83d1-f45412718d94 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>> File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'\n"]
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 11:27:07.567 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>> File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'\n"]
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 ERROR nova.openstack.common.rpc.amqp
>>>>>>>>>>>>> [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>>>>>>>>> 2013-07-24 11:34:02.320 1685 ERROR nova.virt.libvirt.driver
>>>>>>>>>>>>> [-] [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] During wait destroy,
>>>>>>>>>>>>> instance disappeared.
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 11:38:49.087 1685 ERROR nova.virt.libvirt.driver
>>>>>>>>>>>>> [-] [instance: 69a2c512-bea1-46de-91be-c33a07799981] During wait destroy,
>>>>>>>>>>>>> instance disappeared.
>>>>>>>>>>>>> 2013-07-24 11:38:49.220 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>> File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'\n"]
>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 13:13:09.224 ERROR nova.virt.libvirt.driver
>>>>>>>>>>>>> [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Error on '/var/lib/nova/instances' while
>>>>>>>>>>>>> checking direct I/O: '[Errno 13] Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 13:13:21.636 1694 ERROR nova.virt.libvirt.driver
>>>>>>>>>>>>> [-] [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] During wait destroy,
>>>>>>>>>>>>> instance disappeared.
>>>>>>>>>>>>> 2013-07-24 13:13:21.729 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>> in spawn\n block_device_info=block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>> in to_xml\n disk_info, rescue, block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>> in get_guest_config\n inst_type):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>> in get_guest_storage_config\n inst_type)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>> in get_guest_disk_config\n self.disk_cachemode,\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>> in disk_cachemode\n if not
>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>> in _supports_direct_io\n {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>> self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>> in _supports_direct_io\n f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lock/nova'
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lock/nova'
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lock/nova'
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 14:47:26.986 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>> 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'\n"]
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lock/nova'
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lock/nova'
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-e0716278-f864-4f70-962a-bf74559fc4c9 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 16:14:54.459 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-e0716278-f864-4f70-962a-bf74559fc4c9 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>> 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'\n"]
>>>>>>>>>>>>> 2013-07-24 16:43:47.486 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-5d9b9575-a6eb-4324-b975-78dba69c24ef c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 16:43:58.434 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-5d9b9575-a6eb-4324-b975-78dba69c24ef c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>> 'libvirt-00497f97-a0fe-4a51-b7f6-6901d0d34791'\n"]
>>>>>>>>>>>>> 2013-07-24 16:46:43.408 1694 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lock/nova'
>>>>>>>>>>>>> 2013-07-24 17:26:43.627 1694 ERROR nova.manager [-] Error
>>>>>>>>>>>>> during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission
>>>>>>>>>>>>> denied: '/var/lock/nova'
>>>>>>>>>>>>> 2013-07-24 17:47:46.003 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-370ac716-203d-43aa-a146-709297c715b3 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 8a4e0c84-07ec-40ab-9585-5433dff92b64] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 17:47:55.194 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-370ac716-203d-43aa-a146-709297c715b3 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 8a4e0c84-07ec-40ab-9585-5433dff92b64] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>> 'libvirt-8a4e0c84-07ec-40ab-9585-5433dff92b64'\n"]
>>>>>>>>>>>>> 2013-07-24 17:53:42.789 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-153949aa-d5a7-473f-b79b-38616ae77441 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 17:54:05.061 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-153949aa-d5a7-473f-b79b-38616ae77441 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>> 'libvirt-d3bfc215-7e24-40d2-b6f9-a1542a2b9d05'\n"]
>>>>>>>>>>>>> 2013-07-24 18:05:44.677 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-60486a96-54a1-4472-846b-b57209c68921 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 005f7640-a8ce-429a-ba91-85e7db1d86ac] Instance failed to spawn
>>>>>>>>>>>>> 2013-07-24 18:06:14.408 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-60486a96-54a1-4472-846b-b57209c68921 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 005f7640-a8ce-429a-ba91-85e7db1d86ac] Error: ['Traceback (most recent call
>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>> 'libvirt-005f7640-a8ce-429a-ba91-85e7db1d86ac'\n"]
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> cat /var/log/nova/nova-*.log | grep -E TRACE
>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager Traceback
>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager task(self,
>>>>>>>>>>>>> context)
>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager retval =
>>>>>>>>>>>>> f(*args, **kwargs)
>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager with
>>>>>>>>>>>>> open(id_path, 'w') as f:
>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager IOError:
>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager Traceback
>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager task(self,
>>>>>>>>>>>>> context)
>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager retval =
>>>>>>>>>>>>> f(*args, **kwargs)
>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager with
>>>>>>>>>>>>> open(id_path, 'w') as f:
>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager IOError:
>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager Traceback
>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager task(self,
>>>>>>>>>>>>> context)
>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager retval =
>>>>>>>>>>>>> f(*args, **kwargs)
>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager with
>>>>>>>>>>>>> open(id_path, 'w') as f:
>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager IOError:
>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager Traceback
>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager task(self,
>>>>>>>>>>>>> context)
>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager retval =
>>>>>>>>>>>>> f(*args, **kwargs)
>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager with
>>>>>>>>>>>>> open(id_path, 'w') as f:
>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager IOError:
>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Traceback (most recent
>>>>>>>>>>>>> call last):
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] block_device_info)
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>> in _create_image
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] os.makedirs(path)
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] mkdir(name, mode)
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] OSError: [Errno 13]
>>>>>>>>>>>>> Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'
>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Traceback (most recent
>>>>>>>>>>>>> call last):
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] block_device_info)
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>> in _create_image
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] os.makedirs(path)
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] mkdir(name, mode)
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] OSError: [Errno 13]
>>>>>>>>>>>>> Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'
>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Traceback (most recent
>>>>>>>>>>>>> call last):
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] block_device_info)
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>> in _create_image
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] os.makedirs(path)
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] mkdir(name, mode)
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] OSError: [Errno 13]
>>>>>>>>>>>>> Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'
>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager Traceback
>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager task(self,
>>>>>>>>>>>>> context)
>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager retval =
>>>>>>>>>>>>> f(*args, **kwargs)
>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager with
>>>>>>>>>>>>> open(id_path, 'w') as f:
>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager IOError:
>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Traceback (most recent
>>>>>>>>>>>>> call last):
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] block_device_info)
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>> block_device_info=block_device_info)
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>> in to_xml
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] disk_info, rescue,
>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>> in get_guest_config
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] inst_type):
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>> in get_guest_storage_config
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] inst_type)
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>> in get_guest_disk_config
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] self.disk_cachemode,
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>> in disk_cachemode
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] if not
>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>> in _supports_direct_io
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] {'path': dirpath,
>>>>>>>>>>>>> 'ex': str(e)})
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] self.gen.next()
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>> in _supports_direct_io
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] f = os.open(testfile,
>>>>>>>>>>>>> os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] OSError: [Errno 13]
>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/.directio.test'
>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Traceback (most recent
>>>>>>>>>>>>> call last):
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] block_device_info)
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>> in _create_image
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] os.makedirs(path)
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] mkdir(name, mode)
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] OSError: [Errno 13]
>>>>>>>>>>>>> Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'
>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Traceback (most recent
>>>>>>>>>>>>> call last):
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] block_device_info)
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>> in _create_image
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] os.makedirs(path)
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] mkdir(name, mode)
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] OSError: [Errno 13]
>>>>>>>>>>>>> Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'
>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp temp_level, payload)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp pass
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return function(self, context, *args,
>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp e, sys.exc_info())
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return function(self, context, *args,
>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1210, in
>>>>>>>>>>>>> run_instance
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp do_run_instance()
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 242, in inner
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1209, in
>>>>>>>>>>>>> do_run_instance
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp admin_password, is_first_time, node,
>>>>>>>>>>>>> instance)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 865, in
>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self._set_instance_error_state(context,
>>>>>>>>>>>>> instance['uuid'])
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 857, in
>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp filter_properties, bdms)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 897, in
>>>>>>>>>>>>> _reschedule_or_reraise
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self._log_original_error(exc_info,
>>>>>>>>>>>>> instance_uuid)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 892, in
>>>>>>>>>>>>> _reschedule_or_reraise
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self._shutdown_instance(context,
>>>>>>>>>>>>> instance, bdms)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1222, in
>>>>>>>>>>>>> _shutdown_instance
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp network_info =
>>>>>>>>>>>>> self._get_instance_nw_info(context, instance)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 667, in
>>>>>>>>>>>>> _get_instance_nw_info
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp instance,
>>>>>>>>>>>>> conductor_api=self.conductor_api)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 359,
>>>>>>>>>>>>> in get_instance_nw_info
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp result =
>>>>>>>>>>>>> self._get_instance_nw_info(context, instance, networks)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 367,
>>>>>>>>>>>>> in _get_instance_nw_info
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp nw_info =
>>>>>>>>>>>>> self._build_network_info_model(context, instance, networks)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 778,
>>>>>>>>>>>>> in _build_network_info_model
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp data = client.list_ports(**search_opts)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107,
>>>>>>>>>>>>> in with_params
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp ret = self.function(instance, *args,
>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 255,
>>>>>>>>>>>>> in list_ports
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp **_params)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 996,
>>>>>>>>>>>>> in list
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp for r in self._pagination(collection,
>>>>>>>>>>>>> path, **params):
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 1009,
>>>>>>>>>>>>> in _pagination
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp res = self.get(path, params=params)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 982,
>>>>>>>>>>>>> in get
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp headers=headers, params=params)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 967,
>>>>>>>>>>>>> in retry_request
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp headers=headers, params=params)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912,
>>>>>>>>>>>>> in do_request
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self._handle_fault_response(status_code,
>>>>>>>>>>>>> replybody)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893,
>>>>>>>>>>>>> in _handle_fault_response
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp exception_handler_v20(status_code,
>>>>>>>>>>>>> des_error_body)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 87,
>>>>>>>>>>>>> in exception_handler_v20
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp message=message)
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp QuantumClientException: Request Timeout
>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] Traceback (most recent
>>>>>>>>>>>>> call last):
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] block_device_info)
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981]
>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>> in _create_image
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981]
>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] os.makedirs(path)
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] mkdir(name, mode)
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] OSError: [Errno 13]
>>>>>>>>>>>>> Permission denied:
>>>>>>>>>>>>> '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'
>>>>>>>>>>>>> 2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981]
>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager Traceback
>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager task(self,
>>>>>>>>>>>>> context)
>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager
>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager retval =
>>>>>>>>>>>>> f(*args, **kwargs)
>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager with
>>>>>>>>>>>>> open(id_path, 'w') as f:
>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager IOError:
>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 TRACE nova.manager
>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager Traceback
>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager task(self,
>>>>>>>>>>>>> context)
>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager
>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager retval =
>>>>>>>>>>>>> f(*args, **kwargs)
>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager with
>>>>>>>>>>>>> open(id_path, 'w') as f:
>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager IOError:
>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 TRACE nova.manager
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Traceback (most recent
>>>>>>>>>>>>> call last):
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] block_device_info)
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]
>>>>>>>>>>>>> block_device_info=block_device_info)
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>> in to_xml
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] disk_info, rescue,
>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>> in get_guest_config
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] inst_type):
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>> in get_guest_storage_config
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] inst_type)
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>> in get_guest_disk_config
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] self.disk_cachemode,
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>> in disk_cachemode
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] if not
>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>> in _supports_direct_io
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] {'path': dirpath,
>>>>>>>>>>>>> 'ex': str(e)})
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File
>>>>>>>>>>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] self.gen.next()
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>> in _supports_direct_io
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] f = os.open(testfile,
>>>>>>>>>>>>> os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] OSError: [Errno 13]
>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/.directio.test'
>>>>>>>>>>>>> 2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager Traceback
>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager task(self,
>>>>>>>>>>>>> context)
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager
>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 210, in inner
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager
>>>>>>>>>>>>> fileutils.ensure_tree(local_lock_path)
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager
>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager
>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager OSError:
>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 TRACE nova.manager
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager Traceback
>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager task(self,
>>>>>>>>>>>>> context)
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager
>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 210, in inner
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager
>>>>>>>>>>>>> fileutils.ensure_tree(local_lock_path)
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager
>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager
>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager OSError:
>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 TRACE nova.manager
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager Traceback
>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager task(self,
>>>>>>>>>>>>> context)
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager
>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 210, in inner
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager
>>>>>>>>>>>>> fileutils.ensure_tree(local_lock_path)
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager
>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager
>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager OSError:
>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 TRACE nova.manager
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Traceback (most recent
>>>>>>>>>>>>> call last):
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] block_device_info)
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] block_device_info)
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>> in _create_domain_and_network
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] domain =
>>>>>>>>>>>>> self._create_domain(xml, instance=instance)
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>> in _create_domain
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]
>>>>>>>>>>>>> domain.createWithFlags(launch_flags)
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] result =
>>>>>>>>>>>>> proxy_call(self._autowrap, f, *args, **kwargs)
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>> proxy_call
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] rv =
>>>>>>>>>>>>> execute(f,*args,**kwargs)
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] rv =
>>>>>>>>>>>>> meth(*args,**kwargs)
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] if ret == -1: raise
>>>>>>>>>>>>> libvirtError ('virDomainCreateWithFlags() failed', dom=self)
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] libvirtError: internal
>>>>>>>>>>>>> error cannot load AppArmor profile
>>>>>>>>>>>>> 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'
>>>>>>>>>>>>> 2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager Traceback
>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager task(self,
>>>>>>>>>>>>> context)
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager
>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 210, in inner
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager
>>>>>>>>>>>>> fileutils.ensure_tree(local_lock_path)
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager
>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager
>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager OSError:
>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 TRACE nova.manager
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager Traceback
>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager task(self,
>>>>>>>>>>>>> context)
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager
>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>> 210, in inner
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager
>>>>>>>>>>>>> fileutils.ensure_tree(local_lock_path)
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager
>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager File
>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager
>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager OSError:
>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 TRACE nova.manager
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Traceback (most recent
>>>>>>>>>>>>> call last):
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] block_device_info)
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] block_device_info)
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>> in _create_domain_and_network
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] domain =
>>>>>>>>>>>>> self._create_domain(xml, instance=instance)
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>> in _create_domain
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913]
>>>>>>>>>>>>> domain.createWithFlags(launch_flags)
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] result =
>>>>>>>>>>>>> proxy_call(self._autowrap, f, *args, **kwargs)
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>> proxy_call
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] rv =
>>>>>>>>>>>>> execute(f,*args,**kwargs)
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] rv =
>>>>>>>>>>>>> meth(*args,**kwargs)
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] if ret == -1: raise
>>>>>>>>>>>>> libvirtError ('virDomainCreateWithFlags() failed', dom=self)
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] libvirtError: internal
>>>>>>>>>>>>> error cannot load AppArmor profile
>>>>>>>>>>>>> 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'
>>>>>>>>>>>>> 2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913]
>>>>>>>>>>>>> 2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Traceback (most recent
>>>>>>>>>>>>> call last):
>>>>>>>>>>>>> 2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>> 2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] block_device_info)
>>>>>>>>>>>>> 2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>> 2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager
>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791]
>>>>>>>>>>>>>
>>>>>>>>>>>> ...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Follow ups
References
-
Configuring Live Migration
From: Guilherme Russi, 2013-07-24
-
Re: Configuring Live Migration
From: Wangpan, 2013-07-25
-
Re: Configuring Live Migration
From: Guilherme Russi, 2013-07-25
-
Re: Configuring Live Migration
From: Wangpan, 2013-07-25
-
Re: Configuring Live Migration
From: Guilherme Russi, 2013-07-25
-
Re: Configuring Live Migration
From: Wangpan, 2013-07-26
-
Re: Configuring Live Migration
From: Guilherme Russi, 2013-07-26
-
Re: Configuring Live Migration
From: Wangpan, 2013-07-26
-
Re: Configuring Live Migration
From: Guilherme Russi, 2013-07-26
-
Re: Configuring Live Migration
From: Filipe Manco, 2013-07-26
-
Re: Configuring Live Migration
From: Guilherme Russi, 2013-07-26
-
Re: Configuring Live Migration
From: Filipe Manco, 2013-07-26
-
Re: Configuring Live Migration
From: Filipe Manco, 2013-07-26
-
Re: Configuring Live Migration
From: Guilherme Russi, 2013-07-26
