openstack team mailing list archive

[Unmesh Gurjar <unmesh.gurjar@xxxxxxxxx>]

Javier,

The stack trace you provided indicates that the allocate network call from
Compute (to nova-network) timed out. I would suggest checking the
nova-network logs, that might give some insight.
Also, IMO, the create instance request is being served by Compute means the
RabbitMQ is working. You might want to ensure that nova-network has also
connected to the same AMPQ server (as Compute is).

Regards,
Unmesh G.

On Tue, Feb 26, 2013 at 2:16 PM, Javier Alvarez <javier.alvarez@xxxxxx>wrote:

>  Hello JuanFra,
>
> How can I check that? I have rabbitmq-server running on the controller
> node and I have set rabbit_host property in the .conf files. Also, if I
> restart nova-compute service, I get the following message:
>
> 2013-02-26 09:43:52 INFO nova.rpc.common [-] Connected to AMQP server on
> 172.16.8.1:5672
>
> Is there any other test I could do to verify everything is properly
> working?
>
> Thanks,
>
> Javi
>
>
> On 25/02/13 20:09, JuanFra Rodriguez Cardoso wrote:
>
> Hi:
>
>  Have you ensured your messaging queue is working correctly?
>
>  Regards,
> JuanFra
>
>
> 2013/2/25 Javier Alvarez <javier.alvarez@xxxxxx>
>
>> Hello,
>>
>> I'm trying to setup OpenStack Essex on Debian and I'm getting an error
>> when creating new VMs. The problem arises when trying to allocate the
>> network of the VM, but I'm not sure why it's happening, I'm using
>> nova-network, and that's the log output:
>>
>> 2013-02-25 18:20:28 AUDIT nova.compute.manager
>> [req-4d211944-bd5c-4e88-a8df-c4a9167676e2 36a7e60b4d134307b61e06949d33735e
>> 9558f53959a04cc992c3f8b6d91bfb9f] [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] Starting instance...
>> 2013-02-25 18:20:40 INFO nova.virt.libvirt.connection [-] Compute_service
>> record updated for bscgrid21
>> 2013-02-25 18:20:40 INFO nova.compute.manager [-] Updating host status
>> 2013-02-25 18:21:28 ERROR nova.rpc.common
>> [req-4d211944-bd5c-4e88-a8df-c4a9167676e2 36a7e60b4d134307b61e06949d33735e
>> 9558f53959a04cc992c3f8b6d91bfb9f] Timed out waiting for RPC response: timed
>> out
>> 2013-02-25 18:21:28 TRACE nova.rpc.common Traceback (most recent call
>> last):
>> 2013-02-25 18:21:28 TRACE nova.rpc.common   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in
>> ensure
>> 2013-02-25 18:21:28 TRACE nova.rpc.common     return method(*args,
>> **kwargs)
>> 2013-02-25 18:21:28 TRACE nova.rpc.common   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in
>> _consume
>> 2013-02-25 18:21:28 TRACE nova.rpc.common     return
>> self.connection.drain_events(timeout=timeout)
>> 2013-02-25 18:21:28 TRACE nova.rpc.common   File
>> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 167, in
>> drain_events
>> 2013-02-25 18:21:28 TRACE nova.rpc.common     return
>> self.transport.drain_events(self.connection, **kwargs)
>> 2013-02-25 18:21:28 TRACE nova.rpc.common   File
>> "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 262, in
>> drain_events
>> 2013-02-25 18:21:28 TRACE nova.rpc.common     return
>> connection.drain_events(**kwargs)
>> 2013-02-25 18:21:28 TRACE nova.rpc.common   File
>> "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 94, in
>> drain_events
>> 2013-02-25 18:21:28 TRACE nova.rpc.common     return
>> self.wait_multi(self.channels.values(), timeout=timeout)
>> 2013-02-25 18:21:28 TRACE nova.rpc.common   File
>> "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 100, in
>> wait_multi
>> 2013-02-25 18:21:28 TRACE nova.rpc.common     chanmap.keys(),
>> allowed_methods, timeout=timeout)
>> 2013-02-25 18:21:28 TRACE nova.rpc.common   File
>> "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 159, in
>> _wait_multiple
>> 2013-02-25 18:21:28 TRACE nova.rpc.common     channel, method_sig, args,
>> content = read_timeout(timeout)
>> 2013-02-25 18:21:28 TRACE nova.rpc.common   File
>> "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line 132, in
>> read_timeout
>> 2013-02-25 18:21:28 TRACE nova.rpc.common     return
>> self.method_reader.read_method()
>> 2013-02-25 18:21:28 TRACE nova.rpc.common   File
>> "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py",
>> line 221, in read_method
>> 2013-02-25 18:21:28 TRACE nova.rpc.common     raise m
>> 2013-02-25 18:21:28 TRACE nova.rpc.common timeout: timed out
>> 2013-02-25 18:21:28 TRACE nova.rpc.common
>> 2013-02-25 18:21:28 ERROR nova.compute.manager
>> [req-4d211944-bd5c-4e88-a8df-c4a9167676e2 36a7e60b4d134307b61e06949d33735e
>> 9558f53959a04cc992c3f8b6d91bfb9f] [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] Instance failed network setup
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] Traceback (most recent call last):
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 571, in
>> _allocate_network
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] requested_networks=requested_networks)
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 178, in
>> allocate_for_instance
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]     'args': args})
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 68, in call
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]     return _get_impl().call(context,
>> topic, msg, timeout)
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 674, in call
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]     return rpc_amqp.call(context,
>> topic, msg, timeout, Connection.pool)
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 343, in call
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]     rv = list(rv)
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 304, in __iter__
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]     self.done()
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]     self.gen.next()
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 301, in __iter__
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]     self._iterator.next()
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 572, in
>> iterconsume
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]     yield
>> self.ensure(_error_callback, _consume)
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 503, in
>> ensure
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]     error_callback(e)
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 553, in
>> _error_callback
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]     raise rpc_common.Timeout()
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] Timeout: Timeout while waiting on RPC
>> response.
>> 2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
>> 8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]
>> 2013-02-25 18:21:28 ERROR nova.rpc.amqp
>> [req-4d211944-bd5c-4e88-a8df-c4a9167676e2 36a7e60b4d134307b61e06949d33735e
>> 9558f53959a04cc992c3f8b6d91bfb9f] Exception during message handling
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp Traceback (most recent call last):
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 253, in
>> _process_data
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     rval =
>> node_func(context=ctxt, **node_args)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     return f(*args, **kw)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 183, in
>> decorated_function
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     sys.exc_info())
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     self.gen.next()
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in
>> decorated_function
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     return function(self,
>> context, instance_uuid, *args, **kwargs)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 663, in
>> run_instance
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     do_run_instance()
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 945, in inner
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     retval = f(*args, **kwargs)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 662, in
>> do_run_instance
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp self._run_instance(context,
>> instance_uuid, **kwargs)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 463, in
>> _run_instance
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp
>> self._set_instance_error_state(context, instance_uuid)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     self.gen.next()
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 436, in
>> _run_instance
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     requested_networks)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 571, in
>> _allocate_network
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp
>> requested_networks=requested_networks)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 178, in
>> allocate_for_instance
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     'args': args})
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 68, in call
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     return
>> _get_impl().call(context, topic, msg, timeout)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 674, in call
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     return rpc_amqp.call(context,
>> topic, msg, timeout, Connection.pool)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 343, in call
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     rv = list(rv)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 304, in __iter__
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     self.done()
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     self.gen.next()
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 301, in __iter__
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     self._iterator.next()
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 572, in
>> iterconsume
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     yield
>> self.ensure(_error_callback, _consume)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 503, in
>> ensure
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     error_callback(e)
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 553, in
>> _error_callback
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp     raise rpc_common.Timeout()
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp Timeout: Timeout while waiting on
>> RPC response.
>> 2013-02-25 18:21:28 TRACE nova.rpc.amqp
>>
>> Thanks in advance,
>>
>> Javi
>>
>> --
>> Javier Álvarez Cid-Fuentes
>> Grid Computing and Clusters Group
>> Barcelona Supercomputing Center (BSC-CNS)
>> Tel. (+34) 93 413 72 46
>>
>>
>> WARNING / LEGAL TEXT: This message is intended only for the use of the
>> individual or entity to which it is addressed and may contain
>> information which is privileged, confidential, proprietary, or exempt
>> from disclosure under applicable law. If you are not the intended
>> recipient or the person responsible for delivering the message to the
>> intended recipient, you are strictly prohibited from disclosing,
>> distributing, copying, or in any way using this message. If you have
>> received this communication in error, please notify the sender and
>> destroy and delete any copies you may have received.
>>
>> http://www.bsc.es/disclaimer
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
>
> --
> Javier Álvarez Cid-Fuentes
> Grid Computing and Clusters Group
> Barcelona Supercomputing Center (BSC-CNS)
> Tel. (+34) 93 413 72 46
>
>
>
> WARNING / LEGAL TEXT: This message is intended only for the use of the
> individual or entity to which it is addressed and may contain information
> which is privileged, confidential, proprietary, or exempt from disclosure
> under applicable law. If you are not the intended recipient or the person
> responsible for delivering the message to the intended recipient, you are
> strictly prohibited from disclosing, distributing, copying, or in any way
> using this message. If you have received this communication in error,
> please notify the sender and destroy and delete any copies you may have
> received.
>
> http://www.bsc.es/disclaimer <http://www.bsc.es/disclaimer.htm>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>