← Back to team overview

openstack team mailing list archive

VM instance creation fails due to nova-network encountering an RPC timeout.

 

Hello,

I have a two node OpenStack setup, one control-node and one compute node, with the services up and running on each node.

After a minute of issuing the "nova-boot" command on the control-node, I see the following error message in nova-network.log file
    ERROR nova.openstack.common.rpc.common [-] Timed out waiting for RPC response: timed out

I feel like I am running into some networking configuration issue.  Can someone who is familiar with nova-network please help.


control-node (hostname: bodega,  eth0 : 10.176.20.158,  eth1 : internal, VM network)
compute-node (hostname: sonoma,  eth0: 10.176.20.4,  eth1: internal, VM network)

root@bodega:~# nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-network     bodega                               nova             enabled    :-)   2012-11-19 00:27:17
nova-scheduler   bodega                               nova             enabled    :-)   2012-11-19 00:27:22
nova-compute     sonoma                               nova             enabled    :-)   2012-11-19 00:27:21
nova-cert         bodega                               nova             enabled    :-)   2012-11-19 00:27:13
nova-consoleauth bodega                               nova             enabled    :-)   2012-11-19 00:27:12


root@bodega:~# date; nova boot --flavor 2 --image 19beac6b-5a46-4712-a5b5-99c6e99daa67  --key_name mykey  --security_group  default cirros
Sun Nov 18 15:50:26 PST 2012
+-------------------------------------+----------------------------------------------------------+
| Property                            | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                   |
| OS-EXT-SRV-ATTR:host                | sonoma                                                   |
| OS-EXT-SRV-ATTR:hypervisor_hostname | sonoma.eng-rwc.coraid.com                                |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000005                                        |
| OS-EXT-STS:power_state              | 0                                                        |
| OS-EXT-STS:task_state               | scheduling                                               |
| OS-EXT-STS:vm_state                 | building                                                 |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| adminPass                           | Zk6VS8U7Kdoe                                             |
| config_drive                        |                                                          |
| created                             | 2012-11-18T23:50:28Z                                     |
| flavor                              | m1.small                                                 |
| hostId                              | af229f25f97dbd0b697da3d76d8f6fc09cf36a36a181b43ecf065843 |
| id                                  | 4e80964e-5bd1-4df4-a517-223c79d55517                     |
| image                               | cirros-0.3.0-x86_64                                      |
| key_name                            | mykey                                                    |
| metadata                            | {}                                                       |
| name                                | cirros                                                   |
| progress                            | 0                                                        |
| security_groups                     | [{u'name': u'default'}]                                  |
| status                              | BUILD                                                    |
| tenant_id                           | ce1e819636744dc680fa5515f6475e87                         |
| updated                             | 2012-11-18T23:50:28Z                                     |
| user_id                             | ce016bb05df949ebbafcc7c165359d7c                         |
+-------------------------------------+----------------------------------------------------------+
root@bodega:~# nova list
+--------------------------------------+--------+--------+----------+
| ID                                   | Name   | Status | Networks |
+--------------------------------------+--------+--------+----------+
| 4e80964e-5bd1-4df4-a517-223c79d55517 | cirros | BUILD  |          |
+--------------------------------------+--------+--------+----------+
root@bodega:~# sleep 5; date; nova list
Sun Nov 18 15:51:30 PST 2012
+--------------------------------------+--------+--------+----------+
| ID                                   | Name   | Status | Networks |
+--------------------------------------+--------+--------+----------+
| 4e80964e-5bd1-4df4-a517-223c79d55517 | cirros | ERROR  |          |
+--------------------------------------+--------+--------+----------+


nova-network.log

2012-11-18 15:51:29 ERROR nova.openstack.common.rpc.common [-] Timed out waiting for RPC response: timed out
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 513, in ensure
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     return method(*args, **kwargs)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 590, in _consume
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     return self.connection.drain_events(timeout=timeout)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     return self.transport.drain_events(self.connection, **kwargs)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     return connection.drain_events(**kwargs)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     return self.wait_multi(self.channels.values(), timeout=timeout)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     chanmap.keys(), allowed_methods, timeout=timeout)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     channel, method_sig, args, content = read_timeout(timeout)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     return self.method_reader.read_method()
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     raise m
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common timeout: timed out
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common
2012-11-18 15:51:29 DEBUG nova.utils [req-f2b4df1f-6c29-48dc-991b-93fb5eb29d08 ce016bb05df949ebbafcc7c165359d7c ce1e819636744dc680fa5515f6475e87] Got semaphore "get_dhcp" for method "_get_dhcp_ip"...
from (pid=1375) inner /usr/lib/python2.7/dist-packages/nova/utils.py:713
2012-11-18 15:51:29 DEBUG nova.utils [req-f2b4df1f-6c29-48dc-991b-93fb5eb29d08 ce016bb05df949ebbafcc7c165359d7c ce1e819636744dc680fa5515f6475e87] Got semaphore "get_dhcp" for method "_get_dhcp_ip"...
from (pid=1375) inner /usr/lib/python2.7/dist-packages/nova/utils.py:713
2012-11-18 15:51:58 DEBUG nova.openstack.common.rpc.amqp [-] received {u'_context_roles': [], u'_msg_id': u'c1537de718aa4678b51cdb7c9628e502', u'_context_quota_class': None, u'_context_request_id': u'
req-b54a9c72-edc8-4013-8095-1af3a078c6ac', u'_context_service_catalog': None, u'_context_user_name': None, u'_context_auth_token': '<SANITIZED>', u'args': {u'instance_id': 5, u'instance_uuid': u'4e809
64e-5bd1-4df4-a517-223c79d55517', u'host': u'sonoma', u'project_id': u'ce1e819636744dc680fa5515f6475e87', u'rxtx_factor': 1.0}, u'_context_instance_lock_checked': False, u'_context_project_name': None
, u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-11-18T23:51:52.661432', u'_context_read_deleted': u'no', u'_context_user_id': None, u'method': u'get_instance_
nw_info', u'_context_remote_address': None} from (pid=1375) _safe_log /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
2012-11-18 15:51:58 DEBUG nova.openstack.common.rpc.amqp [-] unpacked context: {'project_name': None, 'user_id': None, 'roles': [], 'timestamp': u'2012-11-18T23:51:52.661432', 'auth_token': '<SANITIZE
D>', 'remote_address': None, 'quota_class': None, 'is_admin': True, 'service_catalog': None, 'request_id': u'req-b54a9c72-edc8-4013-8095-1af3a078c6ac', 'instance_lock_checked': False, 'project_id': No
ne, 'user_name': None, 'read_deleted': u'no'} from (pid=1375) _safe_log /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
2012-11-18 15:51:58 DEBUG nova.utils [req-b54a9c72-edc8-4013-8095-1af3a078c6ac None None] Got semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=1375) inner /usr/lib/python2.7/dist-packages/n
ova/utils.py:713
2012-11-18 15:51:58 DEBUG nova.utils [req-b54a9c72-edc8-4013-8095-1af3a078c6ac None None] Got semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=1375) inner /usr/lib/python2.7/dist-packages/n
ova/utils.py:713

Thank you,
Ahmed.