openstack team mailing list archive

Thread
Date
launching instance failed since of nova-network quantum

To: openstack@xxxxxxxxxxxxxxxxxxx
From: livemoon <mwjpiero@xxxxxxxxx>
Date: Wed, 2 May 2012 13:43:27 +0800
Hi, all

I use quantum and OVS in ubuntu 12.04 ,when I lanuch an instance, it return
networking error.

This is nova-network log:
2012-05-02 13:26:38 DEBUG nova.utils
[req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45
5567b78b1a3f4bc68421fe74e044f4e3] Attempting to grab semaphore
"quantum-enable-dhcp" for method "enable_dhcp"... from (pid=31609) inner
/usr/lib/python2.7/dist-packages/nova/utils.py:927
2012-05-02 13:26:38 DEBUG nova.utils
[req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45
5567b78b1a3f4bc68421fe74e044f4e3] Got semaphore "quantum-enable-dhcp" for
method "enable_dhcp"... from (pid=31609) inner
/usr/lib/python2.7/dist-packages/nova/utils.py:931
2012-05-02 13:26:38 INFO nova.network.quantum.manager
[req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45
5567b78b1a3f4bc68421fe74e044f4e3] Using DHCP for network: test-1
2012-05-02 13:26:38 DEBUG nova.network.quantum.quantum_connection
[req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45
5567b78b1a3f4bc68421fe74e044f4e3] Quantum Client Request: GET
/v1.1/tenants/5567b78b1a3f4bc68421fe74e044f4e3/networks/b7eb78f2-b4c2-4106-99c0-0b99f58b2e1e/ports.json?attachment=gw-b7eb78f2-b4
from (pid=31609) do_request
/usr/lib/python2.7/dist-packages/nova/network/quantum/client.py:181
2012-05-02 13:26:38 DEBUG nova.network.quantum.quantum_connection
[req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45
5567b78b1a3f4bc68421fe74e044f4e3] Quantum Client Reply (code = 200) :
 {"ports": []} from (pid=31609) do_request
/usr/lib/python2.7/dist-packages/nova/network/quantum/client.py:192
2012-05-02 13:26:38 DEBUG nova.utils
[req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45
5567b78b1a3f4bc68421fe74e044f4e3] Running cmd (subprocess): ip link show
dev gw-b7eb78f2-b4 from (pid=31609) execute
/usr/lib/python2.7/dist-packages/nova/utils.py:219
2012-05-02 13:26:38 DEBUG nova.utils
[req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45
5567b78b1a3f4bc68421fe74e044f4e3] Result was 1 from (pid=31609) execute
/usr/lib/python2.7/dist-packages/nova/utils.py:235
2012-05-02 13:26:38 DEBUG nova.utils
[req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45
5567b78b1a3f4bc68421fe74e044f4e3] Running cmd (subprocess): sudo
nova-rootwrap ovs-vsctl -- --may-exist add-port br-int gw-b7eb78f2-b4 --
set Interface gw-b7eb78f2-b4 type=internal -- set Interface gw-b7eb78f2-b4
external-ids:iface-id=gw-b7eb78f2-b4 -- set Interface gw-b7eb78f2-b4
external-ids:iface-status=active -- set Interface gw-b7eb78f2-b4
external-ids:attached-mac=fa:16:3e:6f:b7:38 from (pid=31609) execute
/usr/lib/python2.7/dist-packages/nova/utils.py:219
2012-05-02 13:27:16 DEBUG nova.manager
[req-91974e5f-cea9-44a3-a4e3-ef5955aee599 None None] Running periodic task
QuantumManager._publish_service_capabilities from (pid=31609)
periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152
2012-05-02 13:27:16 DEBUG nova.manager
[req-91974e5f-cea9-44a3-a4e3-ef5955aee599 None None] Running periodic task
QuantumManager._disassociate_stale_fixed_ips from (pid=31609)
periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152
2012-05-02 13:27:23 DEBUG nova.rpc.amqp [-] received {u'_context_roles':
[u'admin'], u'_msg_id': u'7584dd267c8f47caa02927c88912383a',
u'_context_read_deleted': u'no', u'_context_request_id':
u'req-9e8a3e43-f55c-4a1e-8565-892708a85338', u'args': {u'instance_id': 2,
u'instance_uuid': u'dc18235c-977f-4fd2-8828-b98155ce2307', u'host':
u'cloud', u'project_id': u'5567b78b1a3f4bc68421fe74e044f4e3',
u'rxtx_factor': 1.0}, u'_context_auth_token': '<SANITIZED>',
u'_context_is_admin': True, u'_context_project_id': None,
u'_context_timestamp': u'2012-05-02T05:27:22.702310', u'_context_user_id':
None, u'method': u'get_instance_nw_info', u'_context_remote_address': None}
from (pid=31609) _safe_log
/usr/lib/python2.7/dist-packages/nova/rpc/common.py:160
2012-05-02 13:27:23 DEBUG nova.rpc.amqp
[req-9e8a3e43-f55c-4a1e-8565-892708a85338 None None] unpacked context:
{'user_id': None, 'roles': [u'admin'], 'timestamp':
'2012-05-02T05:27:22.702310', 'auth_token': '<SANITIZED>',
'remote_address': None, 'is_admin': True, 'request_id':
u'req-9e8a3e43-f55c-4a1e-8565-892708a85338', 'project_id': None,
'read_deleted': u'no'} from (pid=31609) _safe_log
/usr/lib/python2.7/dist-packages/nova/rpc/common.py:160

This is nova-compute log:

2012-05-02 13:27:23 DEBUG nova.virt.libvirt.connection
[req-9e8a3e43-f55c-4a1e-8565-892708a85338 None None] Connecting to libvirt:
qemu:///system from (pid=31711) _get_connection
/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py:292
2012-05-02 13:27:23 DEBUG nova.virt.libvirt.connection
[req-9e8a3e43-f55c-4a1e-8565-892708a85338 None None] Updating host stats
from (pid=31711) update_status
/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py:2467
2012-05-02 13:27:23 DEBUG nova.manager
[req-9e8a3e43-f55c-4a1e-8565-892708a85338 None None] Running periodic task
ComputeManager._poll_unconfirmed_resizes from (pid=31711) periodic_tasks
/usr/lib/python2.7/dist-packages/nova/manager.py:152
2012-05-02 13:27:38 ERROR nova.rpc.common
[req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45
5567b78b1a3f4bc68421fe74e044f4e3] Timed out waiting for RPC response: timed
out
2012-05-02 13:27:38 TRACE nova.rpc.common Traceback (most recent call last):
2012-05-02 13:27:38 TRACE nova.rpc.common   File
"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in
ensure
2012-05-02 13:27:38 TRACE nova.rpc.common     return method(*args, **kwargs)
2012-05-02 13:27:38 TRACE nova.rpc.common   File
"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in
_consume
2012-05-02 13:27:38 TRACE nova.rpc.common     return
self.connection.drain_events(timeout=timeout)
2012-05-02 13:27:38 TRACE nova.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in
drain_events
2012-05-02 13:27:38 TRACE nova.rpc.common     return
self.transport.drain_events(self.connection, **kwargs)
2012-05-02 13:27:38 TRACE nova.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238,
in drain_events
2012-05-02 13:27:38 TRACE nova.rpc.common     return
connection.drain_events(**kwargs)
2012-05-02 13:27:38 TRACE nova.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57,
in drain_events
2012-05-02 13:27:38 TRACE nova.rpc.common     return
self.wait_multi(self.channels.values(), timeout=timeout)
2012-05-02 13:27:38 TRACE nova.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63,
in wait_multi
2012-05-02 13:27:38 TRACE nova.rpc.common     chanmap.keys(),
allowed_methods, timeout=timeout)
2012-05-02 13:27:38 TRACE nova.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120,
in _wait_multiple
2012-05-02 13:27:38 TRACE nova.rpc.common     channel, method_sig, args,
content = read_timeout(timeout)
2012-05-02 13:27:38 TRACE nova.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94,
in read_timeout
2012-05-02 13:27:38 TRACE nova.rpc.common     return
self.method_reader.read_method()
2012-05-02 13:27:38 TRACE nova.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py",
line 221, in read_method
2012-05-02 13:27:38 TRACE nova.rpc.common     raise m
2012-05-02 13:27:38 TRACE nova.rpc.common timeout: timed out
2012-05-02 13:27:38 TRACE nova.rpc.common
2012-05-02 13:27:38 ERROR nova.compute.manager
[req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45
5567b78b1a3f4bc68421fe74e044f4e3] [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01] Instance failed network setup
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01] Traceback (most recent call last):
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 559, in
_allocate_network
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]
requested_networks=requested_networks)
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]   File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 170, in
allocate_for_instance
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]     'args': args})
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]   File
"/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 68, in call
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]     return _get_impl().call(context,
topic, msg, timeout)
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]   File
"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 674, in call
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]     return rpc_amqp.call(context,
topic, msg, timeout, Connection.pool)
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]   File
"/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 338, in call
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]     rv = list(rv)
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]   File
"/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 299, in __iter__
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]     self._iterator.next()
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]   File
"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 572, in
iterconsume
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]     yield
self.ensure(_error_callback, _consume)
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]   File
"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 503, in
ensure
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]     error_callback(e)
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]   File
"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 553, in
_error_callback
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]     raise rpc_common.Timeout()
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01] Timeout: Timeout while waiting on RPC
response.
2012-05-02 13:27:38 TRACE nova.compute.manager [instance:
8fc1536d-3681-4d13-9211-dc88c4dc2b01]
2012-05-02 13:27:38 ERROR nova.rpc.amqp
[req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45
5567b78b1a3f4bc68421fe74e044f4e3] Exception during message handling
2012-05-02 13:27:38 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in
_process_data
2012-05-02 13:27:38 TRACE nova.rpc.amqp     rval = node_func(context=ctxt,
**node_args)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-05-02 13:27:38 TRACE nova.rpc.amqp     return f(*args, **kw)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in
decorated_function
2012-05-02 13:27:38 TRACE nova.rpc.amqp     sys.exc_info())
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-05-02 13:27:38 TRACE nova.rpc.amqp     self.gen.next()
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 171, in
decorated_function
2012-05-02 13:27:38 TRACE nova.rpc.amqp     return function(self, context,
instance_uuid, *args, **kwargs)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 651, in
run_instance
2012-05-02 13:27:38 TRACE nova.rpc.amqp     do_run_instance()
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 945, in inner
2012-05-02 13:27:38 TRACE nova.rpc.amqp     retval = f(*args, **kwargs)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 650, in
do_run_instance
2012-05-02 13:27:38 TRACE nova.rpc.amqp     self._run_instance(context,
instance_uuid, **kwargs)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 451, in
_run_instance
2012-05-02 13:27:38 TRACE nova.rpc.amqp
self._set_instance_error_state(context, instance_uuid)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-05-02 13:27:38 TRACE nova.rpc.amqp     self.gen.next()
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 424, in
_run_instance
2012-05-02 13:27:38 TRACE nova.rpc.amqp     requested_networks)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 559, in
_allocate_network
2012-05-02 13:27:38 TRACE nova.rpc.amqp
requested_networks=requested_networks)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/network/api.py", line 170, in
allocate_for_instance
2012-05-02 13:27:38 TRACE nova.rpc.amqp     'args': args})
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 68, in call
2012-05-02 13:27:38 TRACE nova.rpc.amqp     return
_get_impl().call(context, topic, msg, timeout)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 674, in call
2012-05-02 13:27:38 TRACE nova.rpc.amqp     return rpc_amqp.call(context,
topic, msg, timeout, Connection.pool)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 338, in call
2012-05-02 13:27:38 TRACE nova.rpc.amqp     rv = list(rv)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 299, in __iter__
2012-05-02 13:27:38 TRACE nova.rpc.amqp     self._iterator.next()
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 572, in
iterconsume
2012-05-02 13:27:38 TRACE nova.rpc.amqp     yield
self.ensure(_error_callback, _consume)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 503, in
ensure
2012-05-02 13:27:38 TRACE nova.rpc.amqp     error_callback(e)
2012-05-02 13:27:38 TRACE nova.rpc.amqp   File
"/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 553, in
_error_callback
2012-05-02 13:27:38 TRACE nova.rpc.amqp     raise rpc_common.Timeout()
2012-05-02 13:27:38 TRACE nova.rpc.amqp Timeout: Timeout while waiting on
RPC response.
2012-05-02 13:27:38 TRACE nova.rpc.amqp

my nova.conf is

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--force_dhcp_release
--iscsi_helper=tgtadm
--libvirt_use_virtio_for_bridges
--connection_type=libvirt
--root_helper=sudo nova-rootwrap
--verbose
--ec2_private_dns_show_ip

--network_manager=nova.network.quantum.manager.QuantumManager
--linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver
--libvirt_ovs_bridge=br-int
--libvirt_vif_type=ethernet
--libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
--quantum_use_dhcp=True
--quantum_connection_host=localhost
--quantum_connection_port=9696

--scheduler_driver=nova.scheduler.simple.SimpleScheduler
#--fixed_range=10.0.0.0/16
--s3_host=172.16.0.159
--osapi_compute_extension=nova.api.openstack.compute.contrib.standard_extensions
--my_ip=172.16.0.159
--public_interface=eth0
#--vlan_interface=
#--flat_network_bridge=br100
#--flat_interface=eth1
--sql_connection=mysql://novadbadmin:123456@localhost/nova
--instance_name_template=instance-%08x
--novncproxy_base_url=http://172.16.0.159:6080/vnc_auto.html
--xvpvncproxy_base_url=http://172.16.0.159:6081/console
--vncserver_listen=172.16.0.159
--vncserver_proxyclient_address=172.16.0.159
--api_paste_config=/etc/nova/api-paste.ini
--image_service=nova.image.glance.GlanceImageService
--ec2_dmz_host=172.16.0.159
--rabbit_host=172.16.0.159
--rabbit_password=guest
--glance_api_servers=172.16.0.159:9292
--force_dhcp_release
--send_arp_for_ha
--libvirt_use_virtio_for_bridges
--firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
--auth_strategy=keystone

my nova-manage network list is:
id      IPv4                    IPv6            start address   DNS1
     DNS2            VlanID          project         uuid
6       10.0.1.0/24             None            10.0.1.2        8.8.4.4
    None            None            None
 56136df2-4644-4d68-bec1-d9285cfa9186
7       10.0.2.0/24             None            10.0.2.2        8.8.4.4
    None            None            5567b78b1a3f4bc68421fe74e044f4e3
b7eb78f2-b4c2-4106-99c0-0b99f58b2e1e

ovs network is:
root@cloud:~# ovs-vsctl show
3f474726-21fc-40a4-b7b8-f438fcfe2c3e
    Bridge br-int
        Port "eth1"
            Interface "eth1"
        Port br-int
            Interface br-int
                type: internal


-- 
非淡薄无以明志，非宁静无以致远
Follow ups

Re: launching instance failed since of nova-network quantum
From: Vaze, Mandar, 2012-05-02
Re: launching instance failed since of nova-network quantum
From: Hua BJ Zhang, 2012-05-02
Re: launching instance failed since of nova-network quantum
From: Somik Behera, 2012-05-02