openstack team mailing list archive

Thread
Date
Re: Timed out waiting for RPC response: timed out

To: Mohammed Naser <mnaser@xxxxxxxxxxxx>
From: Afef MDHAFFAR <afef.mdhaffar@xxxxxxxxx>
Date: Thu, 22 Nov 2012 00:14:01 +0100
Cc: "openstack@xxxxxxxxxxxxxxxxxxx" <openstack@xxxxxxxxxxxxxxxxxxx>
In-reply-to: <CAEufa6CSVptNm8ma22CzeC5VycB0oh+umKCBwcbsK_W19XNNMg@mail.gmail.com>
It seems that the problem comes from nova-network. it returns this error
2012-11-21 23:12:19 DEBUG nova.openstack.common.rpc.amqp [-] unpacked
context: {'project_name': u'admin', 'user_id':
u'1745eb1977bc4bcaac3d44e3fbee7647', 'roles': [u'KeystoneAdmin', u'admin',
u'KeystoneServiceAdmin'], 'timestamp': u'2012-11-21T23:11:15.476696',
'auth_token': '<SANITIZED>', 'remote_address': u'10.0.1.15', 'quota_class':
None, 'is_admin': True, 'service_catalog': [{u'endpoints': [{u'adminURL': u'
http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f', u'region':
u'RegionOne', u'id': u'd4e294422e03492eafc09ad6f5766935', u'internalURL': u'
http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f', u'publicURL': u'
http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f'}],
u'endpoints_links': [], u'type': u'compute', u'name': u'nova'},
{u'endpoints': [{u'adminURL': u'http://10.0.1.15:3333', u'region':
u'RegionOne', u'id': u'9561b04884e743c1a20ae8ed41c11618', u'internalURL': u'
http://10.0.1.15:3333', u'publicURL': u'http://10.0.1.15:3333'}],
u'endpoints_links': [], u'type': u's3', u'name': u's3'}, {u'endpoints':
[{u'adminURL': u'http://10.0.1.15:9292', u'region': u'RegionOne', u'id':
u'e91b748cab2a4f82bf13b8a46fbbfb77', u'internalURL': u'http://10.0.1.15:9292',
u'publicURL': u'http://10.0.1.15:9292'}], u'endpoints_links': [], u'type':
u'image', u'name': u'glance'}, {u'endpoints': [{u'adminURL': u'
http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f', u'region':
u'RegionOne', u'id': u'cc784ab5f2664a398cce7352f71b31a7', u'internalURL': u'
http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f', u'publicURL': u'
http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f'}],
u'endpoints_links': [], u'type': u'volume', u'name': u'cinder'},
{u'endpoints': [{u'adminURL': u'http://10.0.1.15:8773/services/Admin',
u'region': u'RegionOne', u'id': u'4010dc2bbadf42a79f18af6f19e6efe7',
u'internalURL': u'http://10.0.1.15:8773/services/Cloud', u'publicURL': u'
http://10.0.1.15:8773/services/Cloud'}], u'endpoints_links': [], u'type':
u'ec2', u'name': u'ec2'}, {u'endpoints': [{u'adminURL': u'
http://10.0.1.15:35357/v2.0', u'region': u'RegionOne', u'id':
u'759f932fafea448299c9461c8a56cdf1', u'internalURL': u'
http://10.0.1.15:5000/v2.0', u'publicURL': u'http://10.0.1.15:5000/v2.0'}],
u'endpoints_links': [], u'type': u'identity', u'name': u'keystone'}],
'request_id': u'req-f5c5ae84-9175-4397-9775-5bbf14cc8038',
'instance_lock_checked': False, 'project_id':
u'bca2f35a23ab466ca6b160648461015f', 'user_name': u'admin', 'read_deleted':
u'no'} from (pid=11711) _safe_log
/opt/stack/nova/nova/openstack/common/rpc/common.py:195
2012-11-21 23:12:20 ERROR nova.openstack.common.rpc.common [-] Timed out
waiting for RPC response: timed out
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common Traceback (most
recent call last):
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common   File
"/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 552, in
ensure
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common     return
method(*args, **kwargs)
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common   File
"/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 630, in
_consume
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common     return
self.connection.drain_events(timeout=timeout)
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 110, in
drain_events
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common     return
self.transport.drain_events(self.connection, **kwargs)
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200,
in drain_events
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common     return
connection.drain_events(**kwargs)
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50,
in drain_events
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common     return
self.wait_multi(self.channels.values(), timeout=timeout)
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56,
in wait_multi
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common
chanmap.keys(), allowed_methods, timeout=timeout)
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104,
in _wait_multiple
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common     channel,
method_sig, args, content = self.read_timeout(timeout)
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86,
in read_timeout
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common     return
self.method_reader.read_method()
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common   File
"/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py",
line 215, in read_method
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common     raise m
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common timeout: timed
out
2012-11-21 23:12:20 TRACE nova.openstack.common.rpc.common
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line
80, in _spawn_n_impl
    func(*args, **kwargs)
  File "/opt/stack/nova/nova/network/rpcapi.py", line 236, in
_rpc_allocate_fixed_ip
    topic=rpc.queue_get_for(ctxt, self.topic, host))
  File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80, in
call
    return rpc.call(context, self._get_topic(topic), msg, timeout)
  File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 108,
in call
    return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
  File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 754,
in call
    rpc_amqp.get_connection_pool(conf, Connection))
  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 369, in
call
    rv = list(rv)
  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 330, in
__iter__
    self.done()
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 327, in
__iter__
    self._iterator.next()
  File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 635,
in iterconsume
    yield self.ensure(_error_callback, _consume)
  File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 555,
in ensure
    error_callback(e)
  File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 616,
in _error_callback
    raise rpc_common.Timeout()
Timeout: Timeout while waiting on RPC response.
2012-11-21 23:12:20 DEBUG nova.network.manager
[req-f5c5ae84-9175-4397-9775-5bbf14cc8038 admin admin] [instance:
c5cc24e2-751d-4f76-a093-9666bcac0a8e] network deallocation for instance
from (pid=11711) deallocate_for_instance
/opt/stack/nova/nova/network/manager.py:1093
2012-11-21 23:12:40 DEBUG nova.manager [-] Running periodic task
FlatDHCPManager.publish_service_capabilities from (pid=11711)
periodic_tasks /opt/stack/nova/nova/manager.py:171
2012-11-21 23:12:40 DEBUG nova.manager [-] Running periodic task
FlatDHCPManager._disassociate_stale_fixed_ips from (pid=11711)
periodic_tasks /opt/stack/nova/nova/manager.py:171


2012/11/22 Afef MDHAFFAR <afef.mdhaffar@xxxxxxxxx>

> thanks again for your answer. I restarted nova-network nut I still get the
> same problem :(
>
> 2012/11/22 Mohammed Naser <mnaser@xxxxxxxxxxxx>
>
>> Hi,
>>
>> Can you try restarting nova-network?  It seems that it's not properly
>> responding to the command which is why this happens.
>>
>> Thanks,
>> Mohammed
>>
>>
>> On Wed, Nov 21, 2012 at 6:02 PM, Afef MDHAFFAR <afef.mdhaffar@xxxxxxxxx>wrote:
>>
>>> thank you for the fast answer. nova-network is running.
>>> I am really going to get crazy!. I do not understand, why it does not
>>> work anymore???
>>>
>>> 2012/11/21 Mohammed Naser <mnaser@xxxxxxxxxxxx>
>>>
>>>> Hi Adef,
>>>>
>>>> Please check that your nova-network instance is running, look like it's
>>>> not (or not responding in a timely fashion)
>>>>
>>>> Regards,
>>>> Mohammed
>>>>
>>>>
>>>> On Wed, Nov 21, 2012 at 5:52 PM, Afef MDHAFFAR <afef.mdhaffar@xxxxxxxxx
>>>> > wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I installed openstack with xcp on 7 nodes, while using devsstack. It
>>>>> was working and I could launch an instance.
>>>>> Suddenly, launching instance fails and I got this error message. I
>>>>> need an urgent help, please.
>>>>> 2012-11-21 22:45:56 DEBUG nova.manager [-] Running periodic task
>>>>> ComputeManager._report_driver_status from (pid=9785) periodic_tasks
>>>>> /opt/stack/nova/nova/manager.py:171
>>>>> 2012-11-21 22:45:56 INFO nova.compute.manager [-] Updating host status
>>>>> 2012-11-21 22:45:56 DEBUG nova.virt.xenapi.host [-] Updating host
>>>>> stats from (pid=9785) update_status
>>>>> /opt/stack/nova/nova/virt/xenapi/host.py:156
>>>>> 2012-11-21 22:45:56 DEBUG nova.manager [-] Running periodic task
>>>>> ComputeManager._poll_unconfirmed_resizes from (pid=9785) periodic_tasks
>>>>> /opt/stack/nova/nova/manager.py:171
>>>>> 2012-11-21 22:46:16 ERROR nova.openstack.common.rpc.common [-] Timed
>>>>> out waiting for RPC response: timed out
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common Traceback
>>>>> (most recent call last):
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 552, in
>>>>> ensure
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     return
>>>>> method(*args, **kwargs)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 630, in
>>>>> _consume
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     return
>>>>> self.connection.drain_events(timeout=timeout)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 110, in
>>>>> drain_events
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     return
>>>>> self.transport.drain_events(self.connection, **kwargs)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200,
>>>>> in drain_events
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     return
>>>>> connection.drain_events(**kwargs)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50,
>>>>> in drain_events
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     return
>>>>> self.wait_multi(self.channels.values(), timeout=timeout)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56,
>>>>> in wait_multi
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common
>>>>> chanmap.keys(), allowed_methods, timeout=timeout)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104,
>>>>> in _wait_multiple
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common
>>>>> channel, method_sig, args, content = self.read_timeout(timeout)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86,
>>>>> in read_timeout
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     return
>>>>> self.method_reader.read_method()
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py",
>>>>> line 215, in read_method
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     raise m
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common timeout:
>>>>> timed out
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common
>>>>> 2012-11-21 22:46:16 ERROR nova.compute.manager
>>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Instance failed network setup
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Traceback (most recent call last):
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 805, in _allocate_network
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]
>>>>> requested_networks=requested_networks)
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/network/api.py", line 44, in wrapper
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     res = f(self, context, *args,
>>>>> **kwargs)
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/network/api.py", line 201, in allocate_for_instance
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     nw_info =
>>>>> self.network_rpcapi.allocate_for_instance(context, **args)
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/network/rpcapi.py", line 144, in allocate_for_instance
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     vpn=vpn,
>>>>> requested_networks=requested_networks))
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80, in call
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     return rpc.call(context,
>>>>> self._get_topic(topic), msg, timeout)
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 108, in call
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     return _get_impl().call(cfg.CONF,
>>>>> context, topic, msg, timeout)
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 754, in call
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]
>>>>> rpc_amqp.get_connection_pool(conf, Connection))
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 369, in call
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     rv = list(rv)
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 330, in __iter__
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     self.done()
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     self.gen.next()
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 327, in __iter__
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     self._iterator.next()
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 635, in
>>>>> iterconsume
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     yield
>>>>> self.ensure(_error_callback, _consume)
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 555, in
>>>>> ensure
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     error_callback(e)
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 616, in
>>>>> _error_callback
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     raise rpc_common.Timeout()
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Timeout: Timeout while waiting on RPC
>>>>> response.
>>>>> 2012-11-21 22:46:16 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]
>>>>> 2012-11-21 22:46:16 DEBUG nova.openstack.common.lockutils
>>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] Got semaphore
>>>>> "compute_resources" for method "abort"... from (pid=9785) inner
>>>>> /opt/stack/nova/nova/openstack/common/lockutils.py:185
>>>>> 2012-11-21 22:46:16 DEBUG nova.compute.claims
>>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Aborting claim: [Claim: 512 MB
>>>>> memory, 0 GB disk, 1 VCPUS] from (pid=9785) abort
>>>>> /opt/stack/nova/nova/compute/claims.py:91
>>>>> 2012-11-21 22:46:16 DEBUG nova.compute.manager
>>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Deallocating network for instance
>>>>> from (pid=9785) _deallocate_network
>>>>> /opt/stack/nova/nova/compute/manager.py:866
>>>>> 2012-11-21 22:46:16 DEBUG nova.openstack.common.rpc.amqp [-] Making
>>>>> asynchronous call on network ... from (pid=9785) multicall
>>>>> /opt/stack/nova/nova/openstack/common/rpc/amqp.py:352
>>>>> 2012-11-21 22:46:16 DEBUG nova.openstack.common.rpc.amqp [-] MSG_ID is
>>>>> eb98cef4e39143459409e551393e9980 from (pid=9785) multicall
>>>>> /opt/stack/nova/nova/openstack/common/rpc/amqp.py:355
>>>>> 2012-11-21 22:46:16 DEBUG nova.openstack.common.rpc.amqp [-] received
>>>>> {u'_context_roles': [u'KeystoneAdmin', u'admin', u'KeystoneServiceAdmin'],
>>>>> u'_msg_id': u'eb98cef4e39143459409e551393e9980', u'_context_quota_class':
>>>>> None, u'_context_request_id': u'req-b091ff10-7159-45bd-a5cf-cd33af605303',
>>>>> u'_context_service_catalog': [{u'endpoints_links': [], u'endpoints':
>>>>> [{u'adminURL': u'
>>>>> http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f',
>>>>> u'region': u'RegionOne', u'publicURL': u'
>>>>> http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f',
>>>>> u'internalURL': u'
>>>>> http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f', u'id':
>>>>> u'd4e294422e03492eafc09ad6f5766935'}], u'type': u'compute', u'name':
>>>>> u'nova'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'
>>>>> http://10.0.1.15:3333', u'region': u'RegionOne', u'publicURL': u'
>>>>> http://10.0.1.15:3333', u'internalURL': u'http://10.0.1.15:3333',
>>>>> u'id': u'9561b04884e743c1a20ae8ed41c11618'}], u'type': u's3', u'name':
>>>>> u's3'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'
>>>>> http://10.0.1.15:9292', u'region': u'RegionOne', u'publicURL': u'
>>>>> http://10.0.1.15:9292', u'internalURL': u'http://10.0.1.15:9292',
>>>>> u'id': u'e91b748cab2a4f82bf13b8a46fbbfb77'}], u'type': u'image', u'name':
>>>>> u'glance'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'
>>>>> http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f',
>>>>> u'region': u'RegionOne', u'publicURL': u'
>>>>> http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f',
>>>>> u'internalURL': u'
>>>>> http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f', u'id':
>>>>> u'cc784ab5f2664a398cce7352f71b31a7'}], u'type': u'volume', u'name':
>>>>> u'cinder'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'
>>>>> http://10.0.1.15:8773/services/Admin', u'region': u'RegionOne',
>>>>> u'publicURL': u'http://10.0.1.15:8773/services/Cloud',
>>>>> u'internalURL': u'http://10.0.1.15:8773/services/Cloud', u'id':
>>>>> u'4010dc2bbadf42a79f18af6f19e6efe7'}], u'type': u'ec2', u'name': u'ec2'},
>>>>> {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'
>>>>> http://10.0.1.15:35357/v2.0', u'region': u'RegionOne', u'publicURL':
>>>>> u'http://10.0.1.15:5000/v2.0', u'internalURL': u'
>>>>> http://10.0.1.15:5000/v2.0', u'id':
>>>>> u'759f932fafea448299c9461c8a56cdf1'}], u'type': u'identity', u'name':
>>>>> u'keystone'}], u'_context_user_name': u'admin', u'_context_auth_token':
>>>>> '<SANITIZED>', u'args': {u'instance_id': 10, u'host': None, u'project_id':
>>>>> u'bca2f35a23ab466ca6b160648461015f'}, u'_context_instance_lock_checked':
>>>>> False, u'_context_project_name': u'admin', u'_context_is_admin': True,
>>>>> u'version': u'1.0', u'_context_project_id':
>>>>> u'bca2f35a23ab466ca6b160648461015f', u'_context_timestamp':
>>>>> u'2012-11-21T22:45:11.517140', u'_context_read_deleted': u'no',
>>>>> u'_context_user_id': u'1745eb1977bc4bcaac3d44e3fbee7647', u'method':
>>>>> u'deallocate_for_instance', u'_context_remote_address': u'10.0.1.15'} from
>>>>> (pid=9786) _safe_log /opt/stack/nova/nova/openstack/common/rpc/common.py:195
>>>>> 2012-11-21 22:46:16 DEBUG nova.openstack.common.rpc.amqp [-] unpacked
>>>>> context: {'project_name': u'admin', 'user_id':
>>>>> u'1745eb1977bc4bcaac3d44e3fbee7647', 'roles': [u'KeystoneAdmin', u'admin',
>>>>> u'KeystoneServiceAdmin'], 'timestamp': u'2012-11-21T22:45:11.517140',
>>>>> 'auth_token': '<SANITIZED>', 'remote_address': u'10.0.1.15', 'quota_class':
>>>>> None, 'is_admin': True, 'service_catalog': [{u'endpoints': [{u'adminURL': u'
>>>>> http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f',
>>>>> u'region': u'RegionOne', u'id': u'd4e294422e03492eafc09ad6f5766935',
>>>>> u'internalURL': u'
>>>>> http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f',
>>>>> u'publicURL': u'
>>>>> http://10.0.1.15:8774/v2/bca2f35a23ab466ca6b160648461015f'}],
>>>>> u'endpoints_links': [], u'type': u'compute', u'name': u'nova'},
>>>>> {u'endpoints': [{u'adminURL': u'http://10.0.1.15:3333', u'region':
>>>>> u'RegionOne', u'id': u'9561b04884e743c1a20ae8ed41c11618', u'internalURL': u'
>>>>> http://10.0.1.15:3333', u'publicURL': u'http://10.0.1.15:3333'}],
>>>>> u'endpoints_links': [], u'type': u's3', u'name': u's3'}, {u'endpoints':
>>>>> [{u'adminURL': u'http://10.0.1.15:9292', u'region': u'RegionOne',
>>>>> u'id': u'e91b748cab2a4f82bf13b8a46fbbfb77', u'internalURL': u'
>>>>> http://10.0.1.15:9292', u'publicURL': u'http://10.0.1.15:9292'}],
>>>>> u'endpoints_links': [], u'type': u'image', u'name': u'glance'},
>>>>> {u'endpoints': [{u'adminURL': u'
>>>>> http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f',
>>>>> u'region': u'RegionOne', u'id': u'cc784ab5f2664a398cce7352f71b31a7',
>>>>> u'internalURL': u'
>>>>> http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f',
>>>>> u'publicURL': u'
>>>>> http://10.0.1.15:8776/v1/bca2f35a23ab466ca6b160648461015f'}],
>>>>> u'endpoints_links': [], u'type': u'volume', u'name': u'cinder'},
>>>>> {u'endpoints': [{u'adminURL': u'http://10.0.1.15:8773/services/Admin',
>>>>> u'region': u'RegionOne', u'id': u'4010dc2bbadf42a79f18af6f19e6efe7',
>>>>> u'internalURL': u'http://10.0.1.15:8773/services/Cloud',
>>>>> u'publicURL': u'http://10.0.1.15:8773/services/Cloud'}],
>>>>> u'endpoints_links': [], u'type': u'ec2', u'name': u'ec2'}, {u'endpoints':
>>>>> [{u'adminURL': u'http://10.0.1.15:35357/v2.0', u'region':
>>>>> u'RegionOne', u'id': u'759f932fafea448299c9461c8a56cdf1', u'internalURL': u'
>>>>> http://10.0.1.15:5000/v2.0', u'publicURL': u'
>>>>> http://10.0.1.15:5000/v2.0'}], u'endpoints_links': [], u'type':
>>>>> u'identity', u'name': u'keystone'}], 'request_id':
>>>>> u'req-b091ff10-7159-45bd-a5cf-cd33af605303', 'instance_lock_checked':
>>>>> False, 'project_id': u'bca2f35a23ab466ca6b160648461015f', 'user_name':
>>>>> u'admin', 'read_deleted': u'no'} from (pid=9786) _safe_log
>>>>> /opt/stack/nova/nova/openstack/common/rpc/common.py:195
>>>>> 2012-11-21 22:46:16 ERROR nova.openstack.common.rpc.common [-] Timed
>>>>> out waiting for RPC response: timed out
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common Traceback
>>>>> (most recent call last):
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 552, in
>>>>> ensure
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     return
>>>>> method(*args, **kwargs)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 630, in
>>>>> _consume
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     return
>>>>> self.connection.drain_events(timeout=timeout)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 110, in
>>>>> drain_events
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     return
>>>>> self.transport.drain_events(self.connection, **kwargs)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 200,
>>>>> in drain_events
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     return
>>>>> connection.drain_events(**kwargs)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 50,
>>>>> in drain_events
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     return
>>>>> self.wait_multi(self.channels.values(), timeout=timeout)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 56,
>>>>> in wait_multi
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common
>>>>> chanmap.keys(), allowed_methods, timeout=timeout)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 104,
>>>>> in _wait_multiple
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common
>>>>> channel, method_sig, args, content = self.read_timeout(timeout)
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 86,
>>>>> in read_timeout
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     return
>>>>> self.method_reader.read_method()
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common   File
>>>>> "/usr/local/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py",
>>>>> line 215, in read_method
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common     raise m
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common timeout:
>>>>> timed out
>>>>> 2012-11-21 22:46:16 TRACE nova.openstack.common.rpc.common
>>>>> Traceback (most recent call last):
>>>>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py",
>>>>> line 80, in _spawn_n_impl
>>>>>     func(*args, **kwargs)
>>>>>   File "/opt/stack/nova/nova/network/rpcapi.py", line 236, in
>>>>> _rpc_allocate_fixed_ip
>>>>>     topic=rpc.queue_get_for(ctxt, self.topic, host))
>>>>>   File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80,
>>>>> in call
>>>>>     return rpc.call(context, self._get_topic(topic), msg, timeout)
>>>>>   File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line
>>>>> 108, in call
>>>>>     return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
>>>>>   File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line
>>>>> 754, in call
>>>>>     rpc_amqp.get_connection_pool(conf, Connection))
>>>>>   File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 369,
>>>>> in call
>>>>>     rv = list(rv)
>>>>>   File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 330,
>>>>> in __iter__
>>>>>     self.done()
>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>     self.gen.next()
>>>>>   File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 327,
>>>>> in __iter__
>>>>>     self._iterator.next()
>>>>>   File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line
>>>>> 635, in iterconsume
>>>>>     yield self.ensure(_error_callback, _consume)
>>>>>   File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line
>>>>> 555, in ensure
>>>>>     error_callback(e)
>>>>>   File "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line
>>>>> 616, in _error_callback
>>>>>     raise rpc_common.Timeout()
>>>>> Timeout: Timeout while waiting on RPC response.
>>>>> 2012-11-21 22:46:16 DEBUG nova.network.manager
>>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] network deallocation for instance
>>>>> from (pid=9786) deallocate_for_instance
>>>>> /opt/stack/nova/nova/network/manager.py:1093
>>>>> 2012-11-21 22:46:17 ERROR nova.compute.manager
>>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Error trying to reschedule
>>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] Traceback (most recent call last):
>>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 641, in
>>>>> _reschedule_or_reraise
>>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     task_state)
>>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 659, in _reschedule
>>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]     retry =
>>>>> filter_properties.get('retry', None)
>>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248] AttributeError: 'unicode' object has
>>>>> no attribute 'get'
>>>>> 2012-11-21 22:46:17 TRACE nova.compute.manager [instance:
>>>>> 552cd71c-5e94-4c3a-b431-9244090d1248]
>>>>> 2012-11-21 22:46:17 DEBUG nova.openstack.common.lockutils
>>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] Got semaphore
>>>>> "compute_resources" for method "update_usage"... from (pid=9785) inner
>>>>> /opt/stack/nova/nova/openstack/common/lockutils.py:185
>>>>> 2012-11-21 22:46:17 DEBUG nova.openstack.common.lockutils
>>>>> [req-b091ff10-7159-45bd-a5cf-cd33af605303 admin admin] Got semaphore
>>>>> "compute_resources" for method "update_usage"... from (pid=9785) inner
>>>>> /opt/stack/nova/nova/openstack/common/lockutils.py:185
>>>>> 2012-11-21 22:46:17 ERROR nova.openstack.common.rpc.amqp [-] Exception
>>>>> during message handling
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp Traceback
>>>>> (most recent call last):
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 276, in
>>>>> _process_data
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     rval =
>>>>> self.proxy.dispatch(ctxt, version, method, **args)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 145, in
>>>>> dispatch
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     return
>>>>> getattr(proxyobj, method)(ctxt, **kwargs)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/exception.py", line 115, in wrapped
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> temp_level, payload)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/exception.py", line 90, in wrapped
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     return
>>>>> f(*args, **kw)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 176, in decorated_function
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     pass
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 162, in decorated_function
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     return
>>>>> function(self, context, *args, **kwargs)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> kwargs['instance']['uuid'], e, sys.exc_info())
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 191, in decorated_function
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     return
>>>>> function(self, context, *args, **kwargs)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 933, in run_instance
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> do_run_instance()
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/openstack/common/lockutils.py", line 228, in inner
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     retval =
>>>>> f(*args, **kwargs)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 932, in do_run_instance
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> admin_password, is_first_time, instance)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 608, in _run_instance
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> self._set_instance_error_state(context, instance['uuid'])
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 596, in _run_instance
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> is_first_time, request_spec, filter_properties)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 577, in _run_instance
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> requested_networks)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/compute/manager.py", line 805, in _allocate_network
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> requested_networks=requested_networks)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/network/api.py", line 44, in wrapper
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     res =
>>>>> f(self, context, *args, **kwargs)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/network/api.py", line 201, in allocate_for_instance
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     nw_info =
>>>>> self.network_rpcapi.allocate_for_instance(context, **args)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/network/rpcapi.py", line 144, in allocate_for_instance
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     vpn=vpn,
>>>>> requested_networks=requested_networks))
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 80, in call
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     return
>>>>> rpc.call(context, self._get_topic(topic), msg, timeout)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 108, in call
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     return
>>>>> _get_impl().call(cfg.CONF, context, topic, msg, timeout)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 754, in call
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> rpc_amqp.get_connection_pool(conf, Connection))
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 369, in call
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     rv =
>>>>> list(rv)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 330, in __iter__
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> self.done()
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 327, in __iter__
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> self._iterator.next()
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 635, in
>>>>> iterconsume
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     yield
>>>>> self.ensure(_error_callback, _consume)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 555, in
>>>>> ensure
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> error_callback(e)
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp   File
>>>>> "/opt/stack/nova/nova/openstack/common/rpc/impl_kombu.py", line 616, in
>>>>> _error_callback
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp     raise
>>>>> rpc_common.Timeout()
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp Timeout:
>>>>> Timeout while waiting on RPC response.
>>>>> 2012-11-21 22:46:17 TRACE nova.openstack.common.rpc.amqp
>>>>> 2012-11-21 22:46:42 DEBUG nova.manager [-] Running periodic task
>>>>> SchedulerManager._expire_reservations from (pid=9790) periodic_tasks
>>>>> /opt/stack/nova/nova/manager.py:171
>>>>> 2012-11-21 22:46:43 DEBUG nova.manager [-] Running periodic task
>>>>> FlatDHCPManager.publish_service_capabilities from (pid=9786) periodic_tasks
>>>>> /opt/stack/nova/nova/manager.py:171
>>>>> 2012-11-21 22:46:43 DEBUG nova.manager [-] Running periodic task
>>>>> FlatDHCPManager._disassociate_stale_fixed_ips from (pid=9786)
>>>>> periodic_tasks /opt/stack/nova/nova/manager.py:171
>>>>> 2012-11-21 22:46:56 DEBUG nova.manager [-] Running periodic task
>>>>> ComputeManager.publish_service_capabilities from (pid=9785) periodic_tasks
>>>>> /opt/stack/nova/nova/manager.py:171
>>>>> 2012-11-21 22:46:56 DEBUG nova.manager [-] Notifying Schedulers of
>>>>> capabilities ... from (pid=9785) publish_service_capabilities
>>>>> /opt/stack/nova/nova/manager.py:256
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Mohammed Naser — vexxhost
>>>> -----------------------------------------------------
>>>> D. 514-316-8872
>>>> D. 800-910-1726 ext. 200
>>>> E. mnaser@xxxxxxxxxxxx
>>>> W. http://vexxhost.com
>>>>
>>>>
>>>
>>>
>>> --
>>> Afef MDHAFFAR
>>> http://www.redcad.org/members/mdhaffar/
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Mohammed Naser — vexxhost
>> -----------------------------------------------------
>> D. 514-316-8872
>> D. 800-910-1726 ext. 200
>> E. mnaser@xxxxxxxxxxxx
>> W. http://vexxhost.com
>>
>>
>
>
> --
> Afef MDHAFFAR
> http://www.redcad.org/members/mdhaffar/
>
>
>
>


-- 
Afef MDHAFFAR
http://www.redcad.org/members/mdhaffar/
References

Timed out waiting for RPC response: timed out
From: Afef MDHAFFAR, 2012-11-21
Re: Timed out waiting for RPC response: timed out
From: Mohammed Naser, 2012-11-21
Re: Timed out waiting for RPC response: timed out
From: Afef MDHAFFAR, 2012-11-21
Re: Timed out waiting for RPC response: timed out
From: Mohammed Naser, 2012-11-21
Re: Timed out waiting for RPC response: timed out
From: Afef MDHAFFAR, 2012-11-21