openstack team mailing list archive

[Ahmed Al-Mehdi <ahmed@xxxxxxxxxx>]

Hi Vish,

Thank you very much gain.  I never set multi_host=True in my nova.conf file.  But I think I know how it could have got set.  In the "OpenStack Install and Deploy Manual", section "Creating the Network for Compute VMs" (  http://docs.openstack.org/folsom/openstack-compute/install/apt/content/compute-create-network.html ), I used the following command from the section to create the network:

        nova-manage network create private --multi_host=T --fixed_range_v4=192.168.100.0/24 --bridge_interface=br100 --num_networks=1 --network_size=256


Is that a typo in the document?

I would really prefer to run "multi_host=0" mode, so I don't run into other issues.  I am not familiar with, so I feel I might end up doing more damage than good trying to muck with the db.  What would you suggest, run nova-network on sonoma, and that will pretty much solve my issues, or should I just re-do the setup from scratch, which would not be too bad for me as it will take me an hour or two.

Regards,
Ahmed.


From: Vishvananda Ishaya <vishvananda@xxxxxxxxx<mailto:vishvananda@xxxxxxxxx>>
Date: Tuesday, November 20, 2012 4:59 PM
To: Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>>
Cc: "openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>" <openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
Subject: Re: [Openstack] Floating IP vs. Fixed IP in nova-network

Your hosts get ips from the fixed range in multi_host mode. It looks like your network is multi_host=True. this is the only reason  sonoma would have been assigned an ip. So you can either run nova-network on sonoma or set multi_host=0 on your network in the database. If you want to switch back to multi_host=0 you will likely have to clean out the db tables, so you might do best dropping all records from the networks and fixed_ips tables and recreating the network.

Vish

On Nov 20, 2012, at 4:52 PM, Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>> wrote:

Hi Vish,

Thank you very much for your help, I really appreciate it.

My setup has two nodes:

controller-node (hostname: bodega;  nova-network running;  no nova-compute running)
eth0:10.176.20.158
eth1:No IP assigned (VM network)

compute-node (hostname: sonoma;  nova-compute running only).
eth0:10.176.20.4
eth1:No IP assigned (VM network)

My network configuration is in single-host mode.  Both the host's hostname and their IPs has not changed.

I believe my setup is affected with issue (a), "have a network or fixed ip with an old hostname assigned".

root@bodega:/etc/nova# mysql -u root -pmysqlsecret   nova -e 'select * from fixed_ips where host="sonoma"'
+---------------------+---------------------+------------+---------+----+---------------+------------+-----------+--------+----------+----------------------+--------+---------------+
| created_at          | updated_at          | deleted_at | deleted | id | address       | network_id | allocated | leased | reserved | virtual_interface_id | host   | instance_uuid |
+---------------------+---------------------+------------+---------+----+---------------+------------+-----------+--------+----------+----------------------+--------+---------------+
| 2012-11-13 18:49:37 | 2012-11-16 21:45:32 | NULL       |       0 |  3 | 192.168.100.2 |          1 |         0 |      0 |        0 |                 NULL | sonoma | NULL          |
+---------------------+---------------------+------------+---------+----+---------------+------------+-----------+--------+----------+----------------------+--------+---------------+
root@bodega:/etc/nova#
root@bodega:/etc/nova# mysql -u root -pmysqlsecret   nova -e 'select * from networks where host="sonoma"'
root@bodega:/etc/nova#  (NO OUTPUT)

I am a bit confused, why is "192.168.100.2" assigned to sonoma?  Isn't that IP range reserved for VMs?

Should sonoma have the IP address "10.176.20.4"?  How can I clear the issue, so I don't get the RPC message timeout.


Regards,
Ahmed.


From: Vishvananda Ishaya <vishvananda@xxxxxxxxx<mailto:vishvananda@xxxxxxxxx>>
Date: Tuesday, November 20, 2012 4:08 PM
To: Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>>
Cc: "openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>" <openstack@xxxxxxxxxxxxxxxxxxx<mailto:openstack@xxxxxxxxxxxxxxxxxxx>>
Subject: Re: [Openstack] Floating IP vs. Fixed IP in nova-network


On Nov 20, 2012, at 4:02 PM, Ahmed Al-Mehdi <ahmed@xxxxxxxxxx<mailto:ahmed@xxxxxxxxxx>> wrote:

Hi Vish,

I do not have auto_assign_floating_ip set.  So, I can safely assume nova-network is assigning fixed-IP, right?

you always get a fixed ip

Sorry to impose, but do you have a few minutes to help me understand by I am getting a RPC message timeout issue which is prohibiting me from launching a VM.  I looked through the logs extensively, but I can't figure out, who the RPC msg is destine for, and why no response.

do you have a machine with the hostname sonoma? Perhaps you did at one point and the hostname has changed?

I suspect either:

a) you have a network or fixed ip with an old hostname assigned:
mysql nova -e 'select * from fixed_ips where host="sonoma"'
mysql nova -e 'select * from networks where host="sonoma"'

or (more likely)
b) you are running multi_host mode and you have nova-compute running on the host 'sonoma' and you don't have nova-network running on the host like you should.

Vish


2012-11-18 15:50:29 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous call on network.sonoma ... from (pid=1375) multicall /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp
.py:351
2012-11-18 15:50:29 DEBUG nova.openstack.common.rpc.amqp [-] MSG_ID is d73be9ea76b3412493d0752abb9d5a02 from (pid=1375) multicall /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:
354
2012-11-18 15:50:52 DEBUG nova.openstack.common.rpc.amqp [-] received {u'_context_roles': [], u'_msg_id': u'b2bc0715982846cd916a8ff61b2513af', u'_context_quota_class': None, u'_context_request_id':
 u'req-22e6e99a-c582-449c-8d61-d4ee57f1ac57', u'_context_service_catalog': None, u'_context_user_name': None, u'_context_auth_token': '<SANITIZED>', u'args': {u'instance_id': 5, u'instance_uuid': u
'4e80964e-5bd1-4df4-a517-223c79d55517', u'host': u'sonoma', u'project_id': u'ce1e819636744dc680fa5515f6475e87', u'rxtx_factor': 1.0}, u'_context_instance_lock_checked': False, u'_context_project_na
me': None, u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-11-18T23:50:47.233052', u'_context_read_deleted': u'no', u'_context_user_id': None, u'method': u'g
et_instance_nw_info', u'_context_remote_address': None} from (pid=1375) _safe_log /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
2012-11-18 15:50:52 DEBUG nova.openstack.common.rpc.amqp [-] unpacked context: {'project_name': None, 'user_id': None, 'roles': [], 'timestamp': u'2012-11-18T23:50:47.233052', 'auth_token': '<SANIT
IZED>', 'remote_address': None, 'quota_class': None, 'is_admin': True, 'service_catalog': None, 'request_id': u'req-22e6e99a-c582-449c-8d61-d4ee57f1ac57', 'instance_lock_checked': False, 'project_i
d': None, 'user_name': None, 'read_deleted': u'no'} from (pid=1375) _safe_log /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py:195
2012-11-18 15:50:52 DEBUG nova.utils [req-22e6e99a-c582-449c-8d61-d4ee57f1ac57 None None] Got semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=1375) inner /usr/lib/python2.7/dist-package
s/nova/utils.py:713
2012-11-18 15:50:52 DEBUG nova.utils [req-22e6e99a-c582-449c-8d61-d4ee57f1ac57 None None] Got semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=1375) inner /usr/lib/python2.7/dist-package
s/nova/utils.py:713
2012-11-18 15:51:09 DEBUG nova.manager [-] Running periodic task FlatDHCPManager._publish_service_capabilities from (pid=1375) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:172
2012-11-18 15:51:09 DEBUG nova.manager [-] Running periodic task FlatDHCPManager._disassociate_stale_fixed_ips from (pid=1375) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:172
2012-11-18 15:51:29 ERROR nova.openstack.common.rpc.common [-] Timed out waiting for RPC response: timed out
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common Traceback (most recent call last):
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 513, in ensure
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     return method(*args, **kwargs)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 590, in _consume
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     return self.connection.drain_events(timeout=timeout)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     return self.transport.drain_events(self.connection, **kwargs)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     return connection.drain_events(**kwargs)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     return self.wait_multi(self.channels.values(), timeout=timeout)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     chanmap.keys(), allowed_methods, timeout=timeout)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     channel, method_sig, args, content = read_timeout(timeout)
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     return self.method_reader.read_method()
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common   File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common     raise m
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common timeout: timed out
2012-11-18 15:51:29 TRACE nova.openstack.common.rpc.common

Thank you,
Ahmed.