openstack team mailing list archive

[Samuel Winchenbach <swinchen@xxxxxxxxx>]

Any idea why calling keystone from "test1" always gets routed to "test2"
and calling it from "test2" requests always get routed to "test1"?   It
wouldn't be a big deal but if I manually stop keystone on one of the hosts
then requests from the other fail.

Thanks,
Sam


On Mon, Feb 18, 2013 at 2:52 AM, Sebastien HAN <han.sebastien@xxxxxxxxx>wrote:

> Hi,
>
> Good to hear that you finally managed to get it working. Usually the
> postrouting rule is more for clients that needs to be routed.
>
> Cheers!
>
> On 16 févr. 2013, at 03:06, Samuel Winchenbach <swinchen@xxxxxxxxx> wrote:
>
> Well I got it to work.  I was being stupid, and forgot to change over the
> endpoints in keystone.
>
> One thing I find interesting is that if I call "keystone user-list" from
> test1 it _always_ sends the request to test2 and vice versa.
>
> Also I did not need to add the POSTROUTING rule... I am not sure why.
>
>
> On Fri, Feb 15, 2013 at 3:44 PM, Samuel Winchenbach <swinchen@xxxxxxxxx>wrote:
>
>> Hrmmm it isn't going so well:
>>
>> root@test1# ip a s dev eth0
>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
>> qlen 1000
>>     link/ether 00:25:90:10:00:78 brd ff:ff:ff:ff:ff:ff
>>     inet 10.21.0.1/16 brd 10.21.255.255 scope global eth0
>>     inet 10.21.1.1/16 brd 10.21.255.255 scope global secondary eth0
>>     inet 10.21.21.1/16 scope global secondary eth0
>>     inet6 fe80::225:90ff:fe10:78/64 scope link
>>        valid_lft forever preferred_lft forever
>>
>>
>> root@test1# ipvsadm -L -n
>> IP Virtual Server version 1.2.1 (size=4096)
>> Prot LocalAddress:Port Scheduler Flags
>>   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
>> TCP  10.21.21.1:5000 wlc persistent 600
>>   -> 10.21.0.1:5000               Masq    100    0          1
>>   -> 10.21.0.2:5000               Masq    100    0          0
>> TCP  10.21.21.1:35357 wlc persistent 600
>>   -> 10.21.0.1:35357              Masq    100    0          0
>>   -> 10.21.0.2:35357              Masq    100    0          0
>>
>> root@test1# iptables -L -v -tnat
>> Chain PREROUTING (policy ACCEPT 283 packets, 24902 bytes)
>>  pkts bytes target     prot opt in     out     source
>> destination
>>
>> Chain INPUT (policy ACCEPT 253 packets, 15256 bytes)
>>  pkts bytes target     prot opt in     out     source
>> destination
>>
>> Chain OUTPUT (policy ACCEPT 509 packets, 37182 bytes)
>>  pkts bytes target     prot opt in     out     source
>> destination
>>
>> Chain POSTROUTING (policy ACCEPT 196 packets, 12010 bytes)
>>  pkts bytes target     prot opt in     out     source
>> destination
>>   277 16700 MASQUERADE  all  --  any    eth0    anywhere
>> anywhere
>>
>> root@test1:~# export OS_AUTH_URL="http://10.21.21.1:5000/v2.0/";
>>  root@test1:~# keystone user-list
>> No handlers could be found for logger "keystoneclient.client"
>> Unable to communicate with identity service: [Errno 113] No route to
>> host. (HTTP 400)
>>
>>
>> I still have some debugging to do with tcpdump, but I thought I would
>> post my initial results.
>>
>>
>> On Fri, Feb 15, 2013 at 2:56 PM, Sébastien Han <han.sebastien@xxxxxxxxx>wrote:
>>
>>> Well if you follow my article, you will get LVS-NAT running. It's fairly
>>> easy, no funky stuff. Yes you will probably need the postrouting rule, as
>>> usual :). Let me know how it goes ;)
>>>
>>> --
>>> Regards,
>>> Sébastien Han.
>>>
>>>
>>> On Fri, Feb 15, 2013 at 8:51 PM, Samuel Winchenbach <swinchen@xxxxxxxxx>wrote:
>>>
>>>> I
>>>> didn't give NAT a shot because it didn't seem as well documented.
>>>>
>>>> I will give NAT a shot.  Will I need to enable to iptables and add a
>>>> rule to the nat table?   None of the documentation mentioned that but every
>>>> time I have ever done NAT I had to setup a rule like... iptables -t nat -A
>>>> POSTROUTING -o eth0 -j MASQUERADE
>>>>
>>>> Thanks for helping me with this.
>>>>
>>>>
>>>> On Fri, Feb 15, 2013 at 2:07 PM, Sébastien Han <han.sebastien@xxxxxxxxx
>>>> > wrote:
>>>>
>>>>> Ok but why direct routing instead of NAT? If the public IPs are _only_
>>>>> on LVS there is no point to use LVS-DR.
>>>>>
>>>>> LVS has the public IPs and redirects to the private IPs, this _must_
>>>>> work.
>>>>>
>>>>> Did you try NAT? Or at least can you give it a shot?
>>>>> --
>>>>> Regards,
>>>>> Sébastien Han.
>>>>>
>>>>>
>>>>> On Fri, Feb 15, 2013 at 3:55 PM, Samuel Winchenbach <
>>>>> swinchen@xxxxxxxxx> wrote:
>>>>> > Sure...  I have undone these settings but I saved a copy:
>>>>> >
>>>>> > two hosts:
>>>>> > test1 eth0: 10.21.0.1/16 eth1: 130.x.x.x/24
>>>>> > test2 eth0: 10.21.0.2/16 eth1: 130.x.x.x/24
>>>>> >
>>>>> > VIP: 10.21.21.1  (just for testing, later I would add a 130.x.x.x/24
>>>>> VIP for
>>>>> > public APIs
>>>>> >
>>>>> > k
>>>>> > eystone is bound to 10.21.0.1 on test1 and 10.21.0.2 on test2
>>>>> >
>>>>> >
>>>>> >
>>>>> > in /etc/sysctl.conf:
>>>>> >    net.ipv4.conf.all.arp_ignore = 1
>>>>> >    net.ipv4.conf.eth0.arp_ignore = 1
>>>>> >    net.ipv4.conf.all.arp_announce = 2
>>>>> >    net.ipv4.conf.eth0.arp_announce = 2
>>>>> >
>>>>> > root# sysctl -p
>>>>> >
>>>>> > in /etc/sysctl.conf:
>>>>> >
>>>>> > checktimeout=
>>>>> > 3
>>>>> >
>>>>> >
>>>>> > checkinterval=
>>>>> > 5
>>>>> >
>>>>> >
>>>>> > autoreload=
>>>>> > yes
>>>>> >
>>>>> >
>>>>> > logfile="/var/log/ldirectord.log"
>>>>> >
>>>>> > quiescent=no
>>>>> >
>>>>> > virtual=10.21.21.1:5000
>>>>> >
>>>>> > real=10.2
>>>>> > 1
>>>>> > .0.1:5000 gate
>>>>> >
>>>>> > real=10.2
>>>>> > 1
>>>>> > .0.2:5000 gate
>>>>> >
>>>>> > scheduler=
>>>>> > w
>>>>> > rr
>>>>> >       protocol=tcp
>>>>> >       checktype=connect
>>>>> >       checkport=5000
>>>>> >
>>>>> > virtual=10.21.21.1:
>>>>> > 35357
>>>>> >
>>>>> > real=10.2
>>>>> > 1
>>>>> > .0.1:
>>>>> > 35357
>>>>> > gate
>>>>> >
>>>>> > real=10.2
>>>>> > 1
>>>>> > .0.2:
>>>>> > 35357
>>>>> > gate
>>>>> >
>>>>> > scheduler=
>>>>> > w
>>>>> > rr
>>>>> >       protocol=tcp
>>>>> >       checktype=connect
>>>>> >       checkport=35357
>>>>> >
>>>>> >
>>>>> > crm shell:
>>>>> >
>>>>> >
>>>>> > primitive
>>>>> > p_openstack_
>>>>> > ip ocf:heartbeat:IPaddr2 \
>>>>> >
>>>>> >
>>>>> > op monitor interval="60" timeout="20" \
>>>>> >
>>>>> >
>>>>> > params ip=
>>>>> > "10.21.21.1
>>>>> > "
>>>>> > cidr_netmask="
>>>>> > 16
>>>>> > "
>>>>> > lvs_support="true"
>>>>> >
>>>>> > p
>>>>> > rimitive
>>>>> > p_openstack_ip_lo
>>>>> >  ocf:heartbeat:IPaddr2 \
>>>>> >
>>>>> >
>>>>> > op monitor interval="60" timeout="20" \
>>>>> >
>>>>> >
>>>>> > params ip="
>>>>> > 10.21.21.1
>>>>> > " nic="lo"
>>>>> > cidr_netmask="32"
>>>>> >
>>>>> > primitive
>>>>> > p_openstack_
>>>>> > lvs ocf:heartbeat:ldirectord \
>>>>> >
>>>>> >
>>>>> > op monitor interval="20" timeout="10"
>>>>> >
>>>>> > group
>>>>> > g_openstack_
>>>>> > ip
>>>>> > _
>>>>> > lvs
>>>>> > p_openstack_
>>>>> > ip
>>>>> > p_openstack_
>>>>> > lvs
>>>>> >
>>>>> > clone
>>>>> > c_openstack_ip_lo
>>>>> >
>>>>> > p_openstack_ip_lo
>>>>> > meta interleave="true"
>>>>> >
>>>>> > colocation
>>>>> > co_openstack_lo_never_lvs
>>>>> > -inf: c
>>>>> > _openstack_ip_lo
>>>>> >
>>>>> > g_openstack_ip_lvs
>>>>> >
>>>>> > Thanks for taking a look at this.
>>>>> >
>>>>> > Sam
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Fri, Feb 15, 2013 at 3:54 AM, Sébastien Han <
>>>>> han.sebastien@xxxxxxxxx>
>>>>> > wrote:
>>>>> >>
>>>>> >> Hum I don't see the problem, it's possible to load-balance VIPs
>>>>> with LVS,
>>>>> >> there are just IPs... Can I see your conf?
>>>>> >>
>>>>> >> --
>>>>> >> Regards,
>>>>> >> Sébastien Han.
>>>>> >>
>>>>> >>
>>>>> >> On Thu, Feb 14, 2013 at 8:34 PM, Samuel Winchenbach <
>>>>> swinchen@xxxxxxxxx>
>>>>> >> wrote:
>>>>> >>>
>>>>> >>> W
>>>>> >>> ell, I think I will have to go with one ip per service and forget
>>>>> about
>>>>> >>> load balancing.  It seems as though with LVS routing requests
>>>>> internally
>>>>> >>> through the VIP is difficult (impossible?) at least with LVS-DR.
>>>>>  It seems
>>>>> >>> like a shame not to be able to distribute the work among the
>>>>> controller
>>>>> >>> nodes.
>>>>> >>>
>>>>> >>>
>>>>> >>> On Thu, Feb 14, 2013 at 9:50 AM, Samuel Winchenbach <
>>>>> swinchen@xxxxxxxxx>
>>>>> >>> wrote:
>>>>> >>>>
>>>>> >>>> Hi Sébastien,
>>>>> >>>>
>>>>> >>>> I have two hosts with public interfaces with a number (~8)
>>>>> compute nodes
>>>>> >>>> behind them.   I am trying to set the two public nodes in for HA
>>>>> and load
>>>>> >>>> balancing,  I plan to run all the openstack services on these two
>>>>> nodes in
>>>>> >>>> Active/Active where possible.   I currently have MySQL and
>>>>> RabbitMQ setup in
>>>>> >>>> pacemaker with a drbd backend.
>>>>> >>>>
>>>>> >>>> That is a quick summary.   If there is anything else I can answer
>>>>> about
>>>>> >>>> my setup please let me know.
>>>>> >>>>
>>>>> >>>> Thanks,
>>>>> >>>> Sam
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Thu, Feb 14, 2013 at 9:26 AM, Sébastien Han <
>>>>> han.sebastien@xxxxxxxxx>
>>>>> >>>> wrote:
>>>>> >>>>>
>>>>> >>>>> Well I don't know your setup, if you use LB for API service or
>>>>> if you
>>>>> >>>>> use an active/passive pacemaker but at the end it's not that
>>>>> much IPs I
>>>>> >>>>> guess. I dare to say that Keepalived sounds outdated to me...
>>>>> >>>>>
>>>>> >>>>> If you use pacemaker and want to have the same IP for all the
>>>>> resources
>>>>> >>>>> simply create a resource group with all the openstack service
>>>>> inside it
>>>>> >>>>> (it's ugly but if it's what you want :)). Give me more info
>>>>> about your setup
>>>>> >>>>> and we can go further in the discussion :).
>>>>> >>>>>
>>>>> >>>>> --
>>>>> >>>>> Regards,
>>>>> >>>>> Sébastien Han.
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> On Thu, Feb 14, 2013 at 3:15 PM, Samuel Winchenbach
>>>>> >>>>> <swinchen@xxxxxxxxx> wrote:
>>>>> >>>>>>
>>>>> >>>>>> T
>>>>> >>>>>> he only real problem is that it would consume a lot of IP
>>>>> addresses
>>>>> >>>>>> when exposing the public interfaces.   I _think_ I may have the
>>>>> solution in
>>>>> >>>>>> your blog actually:
>>>>> >>>>>>
>>>>> http://www.sebastien-han.fr/blog/2012/10/19/highly-available-lvs/
>>>>> >>>>>> and
>>>>> >>>>>> http://clusterlabs.org/wiki/Using_ldirectord
>>>>> >>>>>>
>>>>> >>>>>> I am trying to weigh the pros and cons of this method vs
>>>>> >>>>>> keepalived/haproxy and just biting the bullet and using one IP
>>>>> per service.
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> On Thu, Feb 14, 2013 at 4:17 AM, Sébastien Han
>>>>> >>>>>> <han.sebastien@xxxxxxxxx> wrote:
>>>>> >>>>>>>
>>>>> >>>>>>> What's the problem to have one IP on service pool basis?
>>>>> >>>>>>>
>>>>> >>>>>>> --
>>>>> >>>>>>> Regards,
>>>>> >>>>>>> Sébastien Han.
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> On Wed, Feb 13, 2013 at 8:45 PM, Samuel Winchenbach
>>>>> >>>>>>> <swinchen@xxxxxxxxx> wrote:
>>>>> >>>>>>>>
>>>>> >>>>>>>> What if the VIP is created on a different host than keystone
>>>>> is
>>>>> >>>>>>>> started on?   It seems like you either need to set
>>>>> net.ipv4.ip_nonlocal_bind
>>>>> >>>>>>>> = 1 or create a colocation in pacemaker (which would either
>>>>> require all
>>>>> >>>>>>>> services to be on the same host, or have an ip-per-service).
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>> On Wed, Feb 13, 2013 at 2:28 PM, Razique Mahroua
>>>>> >>>>>>>> <razique.mahroua@xxxxxxxxx> wrote:
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> There we go
>>>>> >>>>>>>>> https://review.openstack.org/#/c/21581/
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Razique Mahroua - Nuage & Co
>>>>> >>>>>>>>> razique.mahroua@xxxxxxxxx
>>>>> >>>>>>>>> Tel : +33 9 72 37 94 15
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Le 13 févr. 2013 à 20:15, Razique Mahroua
>>>>> >>>>>>>>> <razique.mahroua@xxxxxxxxx> a écrit :
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> I'm currently updating that part of the documentation -
>>>>> indeed it
>>>>> >>>>>>>>> states that two IPs are used, but in fact, you end up with
>>>>> only one VIP for
>>>>> >>>>>>>>> the API service.
>>>>> >>>>>>>>> I'll send the patch tonight
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Razique Mahroua - Nuage & Co
>>>>> >>>>>>>>> razique.mahroua@xxxxxxxxx
>>>>> >>>>>>>>> Tel : +33 9 72 37 94 15
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> <NUAGECO-LOGO-Fblan_petit.jpg>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Le 13 févr. 2013 à 20:05, Samuel Winchenbach <
>>>>> swinchen@xxxxxxxxx> a
>>>>> >>>>>>>>> écrit :
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> In that documentation it looks like each openstack service
>>>>> gets it
>>>>> >>>>>>>>> own IP (keystone is being assigned 192.168.42.103 and glance
>>>>> is getting
>>>>> >>>>>>>>> 192.168.42.104).
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> I might be missing something too because in the section
>>>>> titled
>>>>> >>>>>>>>> "Configure the VIP" it create a primitive called "p_api-ip"
>>>>> (or p_ip_api if
>>>>> >>>>>>>>> you read the text above it) and then in "Adding Keystone
>>>>> resource to
>>>>> >>>>>>>>> Pacemaker" it creates a group with "p_ip_keystone"???
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Stranger yet, "Configuring OpenStack Services to use High
>>>>> Available
>>>>> >>>>>>>>> Glance API" says:  "For Nova, for example, if your Glance
>>>>> API service IP
>>>>> >>>>>>>>> address is 192.168.42.104 as in the configuration explained
>>>>> here, you would
>>>>> >>>>>>>>> use the following line in your nova.conf file :
>>>>> glance_api_servers =
>>>>> >>>>>>>>> 192.168.42.103"  But, in the step before it set:
>>>>>  "registry_host =
>>>>> >>>>>>>>> 192.168.42.104"?
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> So I am not sure which ip you would connect to here...
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Sam
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> On Wed, Feb 13, 2013 at 1:29 PM, JuanFra Rodriguez Cardoso
>>>>> >>>>>>>>> <juanfra.rodriguez.cardoso@xxxxxxxxx> wrote:
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Hi Samuel:
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Yes, it's possible with pacemaker. Look at
>>>>> >>>>>>>>>>
>>>>> http://docs.openstack.org/trunk/openstack-ha/content/ch-intro.html.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Regards,
>>>>> >>>>>>>>>> JuanFra
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> 2013/2/13 Samuel Winchenbach <swinchen@xxxxxxxxx>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Hi All,
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> I currently have a HA OpenStack cluster running where the
>>>>> >>>>>>>>>>> OpenStack services are kept alive with a combination of
>>>>> haproxy and
>>>>> >>>>>>>>>>> keepalived.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Is it possible to configure pacemaker so that all the
>>>>> OpenStack
>>>>> >>>>>>>>>>> services  are served by the same IP?  With keepalived I
>>>>> have a virtual ip
>>>>> >>>>>>>>>>> that can move from server to server and haproxy sends the
>>>>> request to a
>>>>> >>>>>>>>>>> machine that has a "live" service.   This allows one
>>>>> (public) ip to handle
>>>>> >>>>>>>>>>> all incoming requests.  I believe it is the combination of
>>>>> VRRP/IPVS that
>>>>> >>>>>>>>>>> allows this.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Is it possible to do something similar with pacemaker?  I
>>>>> really
>>>>> >>>>>>>>>>> don't want to have an IP for each service, and I don't
>>>>> want to make it a
>>>>> >>>>>>>>>>> requirement that all OpenStack services must be running on
>>>>> the same server.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Thanks... I hope this question is clear, I feel like I
>>>>> sort of
>>>>> >>>>>>>>>>> butchered the wording a bit.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Sam
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> _______________________________________________
>>>>> >>>>>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> >>>>>>>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>> >>>>>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> >>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> _______________________________________________
>>>>> >>>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> >>>>>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>> >>>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> >>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>> _______________________________________________
>>>>> >>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>> >>>>>>>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>>>>> >>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>> >>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>> >>>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>
>>>>> >>>>>
>>>>> >>>>
>>>>> >>>
>>>>> >>
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>