openstack team mailing list archive
-
openstack team
-
Mailing list archive
-
Message #18689
VM instance creation fails due to nova-network encountering an RPC timeout.
-
To:
"openstack@xxxxxxxxxxxxxxxxxxx" <openstack@xxxxxxxxxxxxxxxxxxx>
-
From:
Ahmed Al-Mehdi <ahmed@xxxxxxxxxx>
-
Date:
Sun, 18 Nov 2012 18:40:40 -0600
-
Accept-language:
en-US
-
Acceptlanguage:
en-US
-
Thread-index:
Ac3F7oDEgsg0OF6XRIGUD4p79ZvKWw==
-
Thread-topic:
VM instance creation fails due to nova-network encountering an RPC timeout.
-
User-agent:
Microsoft-MacOutlook/14.2.5.121010
Hello,
I have a two node OpenStack setup, one control-node and one compute node, with the services up and running on each node.
After a minute of issuing the "nova-boot" command on the control-node, I see the following error message in nova-network.log file
ERROR nova.openstack.common.rpc.common [-] Timed out waiting for RPC response: timed out
I feel like I am running into some networking configuration issue. Can someone who is familiar with nova-network please help.
control-node (hostname: bodega, eth0 : 10.176.20.158, eth1 : internal, VM network)
compute-node (hostname: sonoma, eth0: 10.176.20.4, eth1: internal, VM network)
root@bodega:~# nova-manage service list
Binary Host Zone Status State Updated_At
nova-network bodega nova enabled :-) 2012-11-19 00:27:17
nova-scheduler bodega nova enabled :-) 2012-11-19 00:27:22
nova-compute sonoma nova enabled :-) 2012-11-19 00:27:21
nova-cert bodega nova enabled :-) 2012-11-19 00:27:13
nova-consoleauth bodega nova enabled :-) 2012-11-19 00:27:12
root@bodega:~# date; nova boot --flavor 2 --image 19beac6b-5a46-4712-a5b5-99c6e99daa67 --key_name mykey --security_group default cirros
Sun Nov 18 15:50:26 PST 2012
+-------------------------------------+----------------------------------------------------------+
| Property | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-SRV-ATTR:host | sonoma |
| OS-EXT-SRV-ATTR:hypervisor_hostname | sonoma.eng-rwc.coraid.com |
| OS-EXT-SRV-ATTR:instance_name | instance-00000005 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | Zk6VS8U7Kdoe |
| config_drive | |
| created | 2012-11-18T23:50:28Z |
| flavor | m1.small |
| hostId | af229f25f97dbd0b697da3d76d8f6fc09cf36a36a181b43ecf065843 |
| id | 4e80964e-5bd1-4df4-a517-223c79d55517 |
| image | cirros-0.3.0-x86_64 |
| key_name | mykey |
| metadata | {} |
| name | cirros |
| progress | 0 |
| security_groups | [{u'name': u'default'}] |
| status | BUILD |
| tenant_id | ce1e819636744dc680fa5515f6475e87 |
| updated | 2012-11-18T23:50:28Z |
| user_id | ce016bb05df949ebbafcc7c165359d7c |
+-------------------------------------+----------------------------------------------------------+
root@bodega:~# nova list
+--------------------------------------+--------+--------+----------+
| ID | Name | Status | Networks |
+--------------------------------------+--------+--------+----------+
| 4e80964e-5bd1-4df4-a517-223c79d55517 | cirros | BUILD | |
+--------------------------------------+--------+--------+----------+
root@bodega:~# sleep 5; date; nova list
Sun Nov 18 15:51:30 PST 2012
+--------------------------------------+--------+--------+----------+
| ID | Name | Status | Networks |
+--------------------------------------+--------+--------+----------+
| 4e80964e-5bd1-4df4-a517-223c79d55517 | cirros | ERROR | |
+--------------------------------------+--------+--------+----------+
nova-network.log
2012-11-18 15:51:29 ERROR nova.openstack.common.rpc.common [-] Timed out waiting for RPC response: timed out
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 513, in ensure
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common return method(*args, **kwargs)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 590, in _consume
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common return connection.drain_events(**kwargs)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common return self.method_reader.read_method()
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common raise m
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common timeout: timed out
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common
2012-11-18 15:51:29 DEBUG nova.utils [req-f2b4df1f-6c29-48dc-991b-93fb5eb29d08 ce016bb05df949ebbafcc7c165359d7c ce1e819636744dc680fa5515f6475e87] Got semaphore "get_dhcp" for method "_get_dhcp_ip"...
from (pid=1375) inner /usr/lib/python2.7/dist-packages/nova/utils.py:713
2012-11-18 15:51:29 DEBUG nova.utils [req-f2b4df1f-6c29-48dc-991b-93fb5eb29d08 ce016bb05df949ebbafcc7c165359d7c ce1e819636744dc680fa5515f6475e87] Got semaphore "get_dhcp" for method "_get_dhcp_ip"...
from (pid=1375) inner /usr/lib/python2.7/dist-packages/nova/utils.py:713
2012-11-18 15:51:58 DEBUG nova.openstack.common.rpc.amqp [-] received {u'_context_roles': [], u'_msg_id': u'c1537de718aa4678b51cdb7c9628e502', u'_context_quota_class': None, u'_context_request_id': u'
req-b54a9c72-edc8-4013-8095-1af3a078c6ac', u'_context_service_catalog': None, u'_context_user_name': None, u'_context_auth_token': '<SANITIZED>', u'args': {u'instance_id': 5, u'instance_uuid': u'4e809
64e-5bd1-4df4-a517-223c79d55517', u'host': u'sonoma', u'project_id': u'ce1e819636744dc680fa5515f6475e87', u'rxtx_factor': 1.0}, u'_context_instance_lock_checked': False, u'_context_project_name': None
, u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-11-18T23:51:52.661432', u'_context_read_deleted': u'no', u'_context_user_id': None, u'method': u'get_instance_
nw_info', u'_context_remote_address': None} from (pid=1375) _safe_log /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
2012-11-18 15:51:58 DEBUG nova.openstack.common.rpc.amqp [-] unpacked context: {'project_name': None, 'user_id': None, 'roles': [], 'timestamp': u'2012-11-18T23:51:52.661432', 'auth_token': '<SANITIZE
D>', 'remote_address': None, 'quota_class': None, 'is_admin': True, 'service_catalog': None, 'request_id': u'req-b54a9c72-edc8-4013-8095-1af3a078c6ac', 'instance_lock_checked': False, 'project_id': No
ne, 'user_name': None, 'read_deleted': u'no'} from (pid=1375) _safe_log /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
2012-11-18 15:51:58 DEBUG nova.utils [req-b54a9c72-edc8-4013-8095-1af3a078c6ac None None] Got semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=1375) inner /usr/lib/python2.7/dist-packages/n
ova/utils.py:713
2012-11-18 15:51:58 DEBUG nova.utils [req-b54a9c72-edc8-4013-8095-1af3a078c6ac None None] Got semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=1375) inner /usr/lib/python2.7/dist-packages/n
ova/utils.py:713
Thank you,
Ahmed.
