openstack team mailing list archive
-
openstack team
-
Mailing list archive
-
Message #23202
Re: Multinode setup? [SOLVED]
For completeness sake here's what I ended up doing to resolve my ongoing issues with second Cinder node:
we're dealing with RHEL6.x here with Folsom from EPEL.
For simplification of things both iptables and SELinux were dropped into permissive mode on both nodes (which I will be bringing back up right away)
Multi-node Cinder
=================
We will use 2 nodes:
* master
* openstack-keystone
* openstack-cinder-api
* openstack-cinder-scheduler
* openstack-cinder-volume
* qpidd
* tgtd
* ...
* slave
* openstack-cinder-volume
* tgtd
* Networking
* 10.10.10.0/24 - Admin network
* 10.10.10.1 - master
* 10.10.10.2 - slave
Note
----
Master node also hosts keystone and other core OpenStack services.
Configuration
=============
Make sure you're configuring your OpenStack infrastructure without the use of localhost and 127.0.0.1 references.
Also make sure your tgt is configured appropriately (openstack-setup scripts will do it for you). For RHEL <6.4 ::
if [ "$DISTRO_VER" -eq '6' ] ; then
if [ "$DISTRO_REL" -le '3' ] ; then
## as per comment in /etc/tgt/conf.d/cinder.conf
## while not flexible or smart - it should do the trick
grep -F 'include /etc/cinder/volumes/*' /etc/tgt/targets.conf || \
sed -i '0 i ##OPENSTACK-SETUP: \
include /etc/cinder/volumes/* \
' /etc/tgt/targets.conf
fi
Master
------
/etc/cinder/cinder.conf
~~~~~~~~~~~~~~~~~~~~~~~
First, the contents::
[DEFAULT]
logdir = /var/log/cinder
state_path = /var/lib/cinder
lock_path = /var/lib/cinder/tmp
volumes_dir = /etc/cinder/volumes
pybasedir=/usr/lib/python2.6/site-packages
# Directory where cinder binaries are installed (string value)
bindir=/usr/bin
my_ip = 10.10.10.1
iscsi_helper = tgtadm
iscsi_ip_prefix= 10.10.10.1
iscsi_ip_address= 10.10.10.1
sql_connection = mysql://cinder:cinder@localhost/cinder
rpc_backend = cinder.openstack.common.rpc.impl_qpid
rootwrap_config = /etc/cinder/rootwrap.conf
auth_strategy = keystone
[keystone_authtoken]
admin_tenant_name = service
admin_user = cinder
admin_password = cinder
auth_host = 10.10.10.1
auth_port = 35357
auth_protocol = http
signing_dirname = /tmp/keystone-signing-cinder
/etc/cinder/api-paste.ini
~~~~~~~~~~~~~~~~~~~~~~~~~
We'll skip the boring part and look at filter:authtoken::
....
[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = 10.10.10.1
service_port = 5000
Slave
-----
/etc/cinder/cinder.conf
~~~~~~~~~~~~~~~~~~~~~~~
Content is slightly different from Master::
[DEFAULT]
## See more options at:
## https://github.com/openstack/cinder/blob/master/etc/cinder/cinder.conf.sample
logdir = /var/log/cinder
state_path = /var/lib/cinder
lock_path = /var/lib/cinder/tmp
volumes_dir = /etc/cinder/volumes
pybasedir=/usr/lib/python2.6/site-packages
# Directory where cinder binaries are installed (string value)
bindir=/usr/bin
my_ip = 10.10.10.2
iscsi_helper = tgtadm
iscsi_ip_prefix= 10.10.10.2
iscsi_ip_address= 10.10.10.2
sql_connection = mysql://cinder:cinder@10.10.10.1/cinder
rpc_backend = cinder.openstack.common.rpc.impl_qpid
# Qpid broker hostname (string value)
qpid_hostname=10.10.10.1
rootwrap_config = /etc/cinder/rootwrap.conf
auth_strategy = keystone
[keystone_authtoken]
admin_tenant_name = service
admin_user = cinder
admin_password = cinder
auth_host = 10.10.10.1
auth_port = 35357
auth_protocol = http
signing_dirname = /tmp/keystone-signing-cinder
Note that we have defined qpid_hostname and our sql_connection now uses proper IP
/etc/cinder/api-paste.ini
~~~~~~~~~~~~~~~~~~~~~~~~~
this one is pretty much a carbon-copy of Master::
....
[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = 10.10.10.1
service_port = 5000
Looks like two things have helped:
1. Using non-localhost URLs for endpoints and other configuration
2. adding "my_ip" and "qpid_hostname" to the config in addition to existing "iscsi_ip*" stuff.
References