openstack team mailing list archive

Thread
Date

Re: VM creation failure

To: Unmesh Gurjar <unmesh.gurjar@xxxxxxxxx>
From: Javier Alvarez <javier.alvarez@xxxxxx>
Date: Tue, 26 Feb 2013 12:19:58 +0100
Cc: openstack@xxxxxxxxxxxxxxxxxxx
In-reply-to: <CANtbCc9=HwQN+YjpHSxZY4YKkWX2jDca0sPj_=njW6XvPVqPOg@mail.gmail.com>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130106 Thunderbird/17.0.2

Hello,

Apparently the problem was that, following a tutorial, I created anetwork in multi-host mode but in my case nova-network is only runningin the controller node.


Regards,

Javi

On 26/02/13 11:44, Unmesh Gurjar wrote:

Javier,

The stack trace you provided indicates that the allocate network callfrom Compute (to nova-network) timed out. I would suggest checking thenova-network logs, that might give some insight.Also, IMO, the create instance request is being served by Computemeans the RabbitMQ is working. You might want to ensure thatnova-network has also connected to the same AMPQ server (as Compute is).


Regards,
Unmesh G.

On Tue, Feb 26, 2013 at 2:16 PM, Javier Alvarez <javier.alvarez@xxxxxx<mailto:javier.alvarez@xxxxxx>> wrote:


    Hello JuanFra,

    How can I check that? I have rabbitmq-server running on the
    controller node and I have set rabbit_host property in the .conf
    files. Also, if I restart nova-compute service, I get the
    following message:

    2013-02-26 09:43:52 INFO nova.rpc.common [-] Connected to AMQP
    server on 172.16.8.1:5672 <http://172.16.8.1:5672>

    Is there any other test I could do to verify everything is
    properly working?

    Thanks,

    Javi


    On 25/02/13 20:09, JuanFra Rodriguez Cardoso wrote:

    Hi:

    Have you ensured your messaging queue is working correctly?

    Regards,
    JuanFra


    2013/2/25 Javier Alvarez <javier.alvarez@xxxxxx
    <mailto:javier.alvarez@xxxxxx>>

        Hello,

        I'm trying to setup OpenStack Essex on Debian and I'm getting
        an error when creating new VMs. The problem arises when
        trying to allocate the network of the VM, but I'm not sure
        why it's happening, I'm using nova-network, and that's the
        log output:

        2013-02-25 18:20:28 AUDIT nova.compute.manager
        [req-4d211944-bd5c-4e88-a8df-c4a9167676e2
        36a7e60b4d134307b61e06949d33735e
        9558f53959a04cc992c3f8b6d91bfb9f] [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] Starting instance...
        2013-02-25 18:20:40 INFO nova.virt.libvirt.connection [-]
        Compute_service record updated for bscgrid21
        2013-02-25 18:20:40 INFO nova.compute.manager [-] Updating
        host status
        2013-02-25 18:21:28 ERROR nova.rpc.common
        [req-4d211944-bd5c-4e88-a8df-c4a9167676e2
        36a7e60b4d134307b61e06949d33735e
        9558f53959a04cc992c3f8b6d91bfb9f] Timed out waiting for RPC
        response: timed out
        2013-02-25 18:21:28 TRACE nova.rpc.common Traceback (most
        recent call last):
        2013-02-25 18:21:28 TRACE nova.rpc.common File
        "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py",
        line 490, in ensure
        2013-02-25 18:21:28 TRACE nova.rpc.common return
        method(*args, **kwargs)
        2013-02-25 18:21:28 TRACE nova.rpc.common File
        "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py",
        line 567, in _consume
        2013-02-25 18:21:28 TRACE nova.rpc.common return
        self.connection.drain_events(timeout=timeout)
        2013-02-25 18:21:28 TRACE nova.rpc.common File
        "/usr/lib/python2.7/dist-packages/kombu/connection.py", line
        167, in drain_events
        2013-02-25 18:21:28 TRACE nova.rpc.common return
        self.transport.drain_events(self.connection, **kwargs)
        2013-02-25 18:21:28 TRACE nova.rpc.common File
        "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line
        262, in drain_events
        2013-02-25 18:21:28 TRACE nova.rpc.common return
        connection.drain_events(**kwargs)
        2013-02-25 18:21:28 TRACE nova.rpc.common File
        "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line
        94, in drain_events
        2013-02-25 18:21:28 TRACE nova.rpc.common return
        self.wait_multi(self.channels.values(), timeout=timeout)
        2013-02-25 18:21:28 TRACE nova.rpc.common File
        "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line
        100, in wait_multi
        2013-02-25 18:21:28 TRACE nova.rpc.common chanmap.keys(),
        allowed_methods, timeout=timeout)
        2013-02-25 18:21:28 TRACE nova.rpc.common File
        "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line
        159, in _wait_multiple
        2013-02-25 18:21:28 TRACE nova.rpc.common channel,
        method_sig, args, content = read_timeout(timeout)
        2013-02-25 18:21:28 TRACE nova.rpc.common File
        "/usr/lib/python2.7/dist-packages/kombu/transport/amqplib.py", line
        132, in read_timeout
        2013-02-25 18:21:28 TRACE nova.rpc.common return
        self.method_reader.read_method()
        2013-02-25 18:21:28 TRACE nova.rpc.common File
        "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py",
        line 221, in read_method
        2013-02-25 18:21:28 TRACE nova.rpc.common raise m
        2013-02-25 18:21:28 TRACE nova.rpc.common timeout: timed out
        2013-02-25 18:21:28 TRACE nova.rpc.common
        2013-02-25 18:21:28 ERROR nova.compute.manager
        [req-4d211944-bd5c-4e88-a8df-c4a9167676e2
        36a7e60b4d134307b61e06949d33735e
        9558f53959a04cc992c3f8b6d91bfb9f] [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] Instance failed network
        setup
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] Traceback (most recent
        call last):
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
        "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
        line 571, in _allocate_network
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]
        requested_networks=requested_networks)
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
        "/usr/lib/python2.7/dist-packages/nova/network/api.py", line
        178, in allocate_for_instance
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] 'args': args})
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line
        68, in call
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] return
        _get_impl().call(context, topic, msg, timeout)
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py",
        line 674, in call
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] return
        rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line
        343, in call
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]     rv = list(rv)
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line
        304, in __iter__
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] self.done()
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
        "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] self.gen.next()
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line
        301, in __iter__
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] self._iterator.next()
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py",
        line 572, in iterconsume
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] yield
        self.ensure(_error_callback, _consume)
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py",
        line 503, in ensure
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] error_callback(e)
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py",
        line 553, in _error_callback
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] raise rpc_common.Timeout()
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60] Timeout: Timeout while
        waiting on RPC response.
        2013-02-25 18:21:28 TRACE nova.compute.manager [instance:
        8c5b2d4a-c715-4f4d-8d74-a6bdccbe1b60]
        2013-02-25 18:21:28 ERROR nova.rpc.amqp
        [req-4d211944-bd5c-4e88-a8df-c4a9167676e2
        36a7e60b4d134307b61e06949d33735e
        9558f53959a04cc992c3f8b6d91bfb9f] Exception during message
        handling
        2013-02-25 18:21:28 TRACE nova.rpc.amqp Traceback (most
        recent call last):
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line
        253, in _process_data
        2013-02-25 18:21:28 TRACE nova.rpc.amqp rval =
        node_func(context=ctxt, **node_args)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/exception.py", line
        114, in wrapped
        2013-02-25 18:21:28 TRACE nova.rpc.amqp return f(*args, **kw)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
        line 183, in decorated_function
        2013-02-25 18:21:28 TRACE nova.rpc.amqp sys.exc_info())
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
        2013-02-25 18:21:28 TRACE nova.rpc.amqp self.gen.next()
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
        line 177, in decorated_function
        2013-02-25 18:21:28 TRACE nova.rpc.amqp return function(self,
        context, instance_uuid, *args, **kwargs)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
        line 663, in run_instance
        2013-02-25 18:21:28 TRACE nova.rpc.amqp do_run_instance()
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/utils.py", line 945,
        in inner
        2013-02-25 18:21:28 TRACE nova.rpc.amqp retval = f(*args,
        **kwargs)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
        line 662, in do_run_instance
        2013-02-25 18:21:28 TRACE nova.rpc.amqp
        self._run_instance(context, instance_uuid, **kwargs)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
        line 463, in _run_instance
        2013-02-25 18:21:28 TRACE nova.rpc.amqp
        self._set_instance_error_state(context, instance_uuid)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
        2013-02-25 18:21:28 TRACE nova.rpc.amqp self.gen.next()
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
        line 436, in _run_instance
        2013-02-25 18:21:28 TRACE nova.rpc.amqp requested_networks)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",
        line 571, in _allocate_network
        2013-02-25 18:21:28 TRACE nova.rpc.amqp
        requested_networks=requested_networks)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/network/api.py", line
        178, in allocate_for_instance
        2013-02-25 18:21:28 TRACE nova.rpc.amqp 'args': args})
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line
        68, in call
        2013-02-25 18:21:28 TRACE nova.rpc.amqp return
        _get_impl().call(context, topic, msg, timeout)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py",
        line 674, in call
        2013-02-25 18:21:28 TRACE nova.rpc.amqp return
        rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line
        343, in call
        2013-02-25 18:21:28 TRACE nova.rpc.amqp     rv = list(rv)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line
        304, in __iter__
        2013-02-25 18:21:28 TRACE nova.rpc.amqp self.done()
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
        2013-02-25 18:21:28 TRACE nova.rpc.amqp self.gen.next()
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line
        301, in __iter__
        2013-02-25 18:21:28 TRACE nova.rpc.amqp self._iterator.next()
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py",
        line 572, in iterconsume
        2013-02-25 18:21:28 TRACE nova.rpc.amqp yield
        self.ensure(_error_callback, _consume)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py",
        line 503, in ensure
        2013-02-25 18:21:28 TRACE nova.rpc.amqp error_callback(e)
        2013-02-25 18:21:28 TRACE nova.rpc.amqp   File
        "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py",
        line 553, in _error_callback
        2013-02-25 18:21:28 TRACE nova.rpc.amqp raise
        rpc_common.Timeout()
        2013-02-25 18:21:28 TRACE nova.rpc.amqp Timeout: Timeout
        while waiting on RPC response.
        2013-02-25 18:21:28 TRACE nova.rpc.amqp

        Thanks in advance,

        Javi

--Javier Álvarez Cid-Fuentes

        Grid Computing and Clusters Group
        Barcelona Supercomputing Center (BSC-CNS)
        Tel. (+34) 93 413 72 46


        WARNING / LEGAL TEXT: This message is intended only for the
        use of the
        individual or entity to which it is addressed and may contain
        information which is privileged, confidential, proprietary,
        or exempt
        from disclosure under applicable law. If you are not the intended
        recipient or the person responsible for delivering the
        message to the
        intended recipient, you are strictly prohibited from disclosing,
        distributing, copying, or in any way using this message. If
        you have
        received this communication in error, please notify the
        sender and
        destroy and delete any copies you may have received.

        http://www.bsc.es/disclaimer

        _______________________________________________
        Mailing list: https://launchpad.net/~openstack
        <https://launchpad.net/%7Eopenstack>
        Post to     : openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
        Unsubscribe : https://launchpad.net/~openstack
        <https://launchpad.net/%7Eopenstack>
        More help   : https://help.launchpad.net/ListHelp

--Javier Álvarez Cid-Fuentes

    Grid Computing and Clusters Group
    Barcelona Supercomputing Center (BSC-CNS)
    Tel. (+34) 93 413 72 46



    WARNING / LEGAL TEXT: This message is intended only for the use of
    the individual or entity to which it is addressed and may contain
    information which is privileged, confidential, proprietary, or
    exempt from disclosure under applicable law. If you are not the
    intended recipient or the person responsible for delivering the
    message to the intended recipient, you are strictly prohibited
    from disclosing, distributing, copying, or in any way using this
    message. If you have received this communication in error, please
    notify the sender and destroy and delete any copies you may have
    received.

    http://www.bsc.es/disclaimer <http://www.bsc.es/disclaimer.htm>

    _______________________________________________
    Mailing list: https://launchpad.net/~openstack
    <https://launchpad.net/%7Eopenstack>
    Post to     : openstack@xxxxxxxxxxxxxxxxxxx
    <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
    Unsubscribe : https://launchpad.net/~openstack
    <https://launchpad.net/%7Eopenstack>
    More help   : https://help.launchpad.net/ListHelp



--
Javier Álvarez Cid-Fuentes
Grid Computing and Clusters Group
Barcelona Supercomputing Center (BSC-CNS)
Tel. (+34) 93 413 72 46



WARNING / LEGAL TEXT: This message is intended only for the use of the
individual or entity to which it is addressed and may contain
information which is privileged, confidential, proprietary, or exempt
from disclosure under applicable law. If you are not the intended
recipient or the person responsible for delivering the message to the
intended recipient, you are strictly prohibited from disclosing,
distributing, copying, or in any way using this message. If you have
received this communication in error, please notify the sender and
destroy and delete any copies you may have received.

http://www.bsc.es/disclaimer

References

VM creation failure
From: Javier Alvarez, 2013-02-25
Re: VM creation failure
From: JuanFra Rodriguez Cardoso, 2013-02-25
Re: VM creation failure
From: Javier Alvarez, 2013-02-26
Re: VM creation failure
From: Unmesh Gurjar, 2013-02-26