← Back to team overview

openstack team mailing list archive

Re: Howto Nova setup with HA?

 

On 02/27/2012 08:06 AM, i3D.net - Tristan van Bokkem wrote:
MySQL DB
* multi-master configuration with pacemaker providing an floating IP. <-
seems to work.

The term "multi-master" in the MySQL world typically means something very specific: you are running multiple master nodes, all in an active setup using something like MMM, MySQL Cluster or Galera Cluster. I don't think this is what you are referring to above. Use of pacemaker is typically an active/passive setup, where Pacemaker can be used to redirect requests from a failed active master to the passive master node that is in standby mode (basically, DRBD is used to replicate the InnoDB and binary logs from the active master to the passive node).

So, I would avoid using the term "multi-master" unless you are specifically talking about an active-active setup, but maybe that's just me being picky ;)

Glance
* Run on multiple servers <-- how do they get synced?

For the Glance registry, you can set up a passive glance-registry node that has connection information to your main HA-setup MySQL database instance.

You can do a number of things for the image storage side of things (the Glance API servers, not the Glance registry). You could set Glance to use a distributed storage backend (like Swift or RADOS) and let that distributed backend provide high availability.

Alternately, for smaller setups, you could do like we are doing in the TryStack project and use the filesystem storage backend in Glance and have Pacemaker monitor a passive Glance API node with a simple rsync script that runs on a periodic interval that syncs the image data.

Compute servers
* Each run their own copy of nova-api (only instances running on the
node use this)

No. Compute nodes run nova-compute, not nova-api.

* nova-network (multi-host configuration) with private network <-- Needs
its own discussion I guess.

Keystone
* How about keystone, same as glance I guess?

Similar to glance-registry.

Cheers,
-jay

Best regards,

Tristan van Bokkem
Datacenter Operations

Contact:
E-mail Personal: tristanvanbokkem@xxxxxxx <mailto:tristanvanbokkem@xxxxxxx>
E-mail Support: info@xxxxxxx <mailto:info@xxxxxxx>
E-mail NOC: noc@xxxxxxx <mailto:noc@xxxxxxx>
Website: http://www.i3d.net Office:
Interactive 3D B.V.
Meent 93b
3011 JG Rotterdam
The Netherlands

Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft.
datacenter in Rotterdam, The Netherlands. High density hosting –
multiple fiber carriers in-house – Level3 PoP.

Interactive 3D (i3D.net) is a company registered in The Netherlands at
Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL
8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content
protection and security. We are ranked in the Deloitte Technology Fast
50 as one of the fastest growing technology companies.

    ------------------------------------------------------------------------
    *From:* Dan Wendlandt [mailto:dan@xxxxxxxxxx]
    *To:* Ghe Rivero [mailto:ghe.rivero@xxxxxxxxxxxx]
    *Cc:* netstack@xxxxxxxxxxxxxxxxxxx, darren.birkett@xxxxxxxxx,
    openstack@xxxxxxxxxxxxxxxxxxx
    *Sent:* Tue, 21 Feb 2012 18:13:55 +0100
    *Subject:* Re: [Openstack] Howto Nova setup with HA?



    2012/2/21 Ghe Rivero <ghe.rivero@xxxxxxxxxxxx
    <mailto:ghe.rivero@xxxxxxxxxxxx>>

        BigSwitch and Nicira have created different blueprints in
        quantum to add their plugins for essex-4 about ten days ago,
        Cisco already have it included almost from the beginning. No
        news of them for the moment, but i really hope that both of them
        make it in time!
        Ghe Rivero

        https://blueprints.launchpad.net/quantum/+spec/quantum-nvp-plugin
        https://blueprints.launchpad.net/quantum/+spec/quantum-bigswitch-plugin


    To be clear, there are several more existing and planned plugins for
    Quantum, even beyond the three you mention. Each plugin will make
    its own trade-offs in terms of HA for control plane and data plane
    components. I'm planning on organizing a talk at the Folsom
    summit/conference where people have a chance to talk about their
    plugins. No need for a flame war here :)

    One other point of confusion that I'd like to address is that there
    is NO requirement that a plugin be merged into the main Quantum repo
    for it to be a "valid" plugin. I know of several plugins that are
    used despite not being merged in (in fact, at the summit, we're
    going to discuss if ANY plugins should remain in the main repo...
    this is likely a larger decision that should be made consistently
    across all openstack projects).

    Dan



        2012/2/21 Diego Parrilla Santamaría
        <diego.parrilla.santamaria@xxxxxxxxx
        <mailto:diego.parrilla.santamaria@xxxxxxxxx>>

            This is the kind of flame war I love... ;-)

            if you think the HA challenges in Openstack Nova are at
            MySQL and Rabbit level, you are not putting the focus on the
            right components. The biggest HA (and scalability)
            challenges in Nova are in the networking side. There are a
            lot of companies tackling the networking challenges of Nova
            (BigSwitch, Nicira, Cisco, etc...), and I would like to see
            how they evolve in the future and how they will address this
            problem. Right now, only promises.

            Fight!
            --
            Diego Parrilla
            <http://www.stackops.com/>*CEO*
            **www.stackops.com* <http://www.stackops.com/> |
            *diego.parrilla@xxxxxxxxxxxx
            <mailto:diego.parrilla@xxxxxxxxxxxx>****| +34 649 94 43 29
            <tel:%2B34%20649%2094%2043%2029> | skype:diegoparrilla*
            * <http://www.stackops.com/>
            **



            On Tue, Feb 21, 2012 at 1:04 PM, i3D.net - Tristan van
            Bokkem <tristanvanbokkem@xxxxxx
            <mailto:tristanvanbokkem@xxxxxx>> wrote:

                __
                Hi Darren,

                No, I am not looking at a performance increasment at
                this moment. Just trying to make the different
                components Nova is build on redundant. Of course a
                master/master setup of mysql / rabbitmq with a floating
                IP will probably work but it bugs me that Nova is not
                really build with flexibility of it components in mind.

                I understand it is impossible to cover every scenario
                and you can make things redundant into the infinitive
                but why doesn't have Nova any of this build in? Why do I
                need to setup a Virtual IP/floating IP where Nova simple
                could accept multiple locations and failover if the
                first location fails. Or even round-robin to distribute
                the load. All I am doing now is yet add another
                component that can possibly fail as well.


                Best regards,

                Tristan van Bokkem
                Datacenter Operations

                Contact:
                E-mail Personal: tristanvanbokkem@xxxxxxx
                <mailto:tristanvanbokkem@xxxxxxx>
                E-mail Support: info@xxxxxxx <mailto:info@xxxxxxx>
                E-mail NOC: noc@xxxxxxx <mailto:noc@xxxxxxx>
                Website: http://www.i3d.net Office:
                Interactive 3D B.V.
                Meent 93b
                3011 JG Rotterdam
                The Netherlands

                Visit www.smartdc.net <http://www.smartdc.net> – SmartDC
                is our in-house 36,000 sq. ft. datacenter in Rotterdam,
                The Netherlands. High density hosting – multiple fiber
                carriers in-house – Level3 PoP.

                Interactive 3D (i3D.net) is a company registered in The
                Netherlands at Meent 93b, Rotterdam. Registration #:
                14074337 - VAT # NL 8202.63.886 <tel:8202.63.886>.B01.
                Interactive 3D (i3D.net) is CDSA certified on content
                protection and security. We are ranked in the Deloitte
                Technology Fast 50 as one of the fastest growing
                technology companies.

                    ------------------------------------------------------------------------
                    *From:* Darren Birkett
                    [mailto:darren.birkett@xxxxxxxxx
                    <mailto:darren.birkett@xxxxxxxxx>]
                    *To:* i3D.net - Tristan van Bokkem
                    [mailto:tristanvanbokkem@xxxxxx
                    <mailto:tristanvanbokkem@xxxxxx>]
                    *Cc:* Florian Haas [mailto:florian@xxxxxxxxxxx
                    <mailto:florian@xxxxxxxxxxx>],
                    openstack@xxxxxxxxxxxxxxxxxxx
                    <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
                    *Sent:* Tue, 21 Feb 2012 12:51:11 +0100

                    *Subject:* Re: [Openstack] Howto Nova setup with HA?

                    Surely it depends what you want out of it? If just
                    pure high
                    availability (ie ability to continue functioning
                    when primary node
                    goes down), then doesn't a master/master with
                    floating IP fit the
                    bill?

                    It sounds like what you're after is better
                    performance/throughput
                    rather than HA - the ability to write to multiple
                    masters at the same
                    time. Even here if nova *was* able to hit multiple
                    databases, a
                    master/master setup isn't going to suit, as each
                    write to one master
                    still needs to be replicated to the other so
                    throughput is going to be
                    the same. You'd then need to look to something like
                    a native mysql
                    cluster to get actual increased throughput.

                    Darren

                    On 21 February 2012 11:34, i3D.net - Tristan van Bokkem

                    <tristanvanbokkem@xxxxxx
                    <mailto:tristanvanbokkem@xxxxxx>> wrote:
                     > Ho Florian,
                     >
                     > I respect your opinion of course and I am not
                    here to start a fight. Point
                     > is, how I understand it Nova really isn't very
                    flexible on High Availability
                     > terms. I have now a master/master mysql setup
                    with a Virtual IP working just
                     > to work around the shortcomings Nova imo
                    introduces. This same story goes
                     > for RabbitMQ, I need to setup RabbitMQ in HA
                    (which is fine) but I need to
                     > setup a Virtual IP for it as well just so Nova
                    can correctly connect to it.
                     > Without the VIP the hole point of setting it up
                    in a cluster is useless as
                     > you still will point nova to one single host.
                     >
                     > It shouldn't be that hard to make Nova listen to
                    multiple mysql addresses
                     > and rabbitmq hosts in a cluster?
                     >
                     >
                     > Best regards,
                     >
                     > Tristan van Bokkem
                     > Datacenter Operations
                     >
                     > Contact:
                     > E-mail Personal: tristanvanbokkem@xxxxxxx
                    <mailto:tristanvanbokkem@xxxxxxx>
                     > E-mail Support: info@xxxxxxx <mailto:info@xxxxxxx>
                     > E-mail NOC: noc@xxxxxxx <mailto:noc@xxxxxxx>
                     > Website: http://www.i3d.net Office:
                     > Interactive 3D B.V.
                     > Meent 93b
                     > 3011 JG Rotterdam
                     > The Netherlands
                     >
                     > Visit www.smartdc.net <http://www.smartdc.net> –
                    SmartDC is our in-house 36,000 sq. ft. datacenter in
                     > Rotterdam, The Netherlands. High density hosting
                    – multiple fiber carriers
                     > in-house – Level3 PoP.
                     >
                     > Interactive 3D (i3D.net) is a company registered
                    in The Netherlands at Meent
                     > 93b, Rotterdam. Registration #: 14074337 - VAT #
                    NL 8202.63.886 <tel:8202.63.886>.B01.
                     > Interactive 3D (i3D.net) is CDSA certified on
                    content protection and
                     > security. We are ranked in the Deloitte
                    Technology Fast 50 as one of the
                     > fastest growing technology companies.
                     >
                     > ________________________________
                     > From: Florian Haas [mailto:florian@xxxxxxxxxxx
                    <mailto:florian@xxxxxxxxxxx>]
                     > To: openstack@xxxxxxxxxxxxxxxxxxx
                    <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
                     > Sent: Fri, 17 Feb 2012 13:32:51 +0100
                     >
                     > Subject: Re: [Openstack] Howto Nova setup with HA?
                     >
                     > On Thu, Feb 16, 2012 at 3:28 PM, i3D.net -
                    Tristan van Bokkem
                     > <tristanvanbokkem@xxxxxx
                    <mailto:tristanvanbokkem@xxxxxx>> wrote:
                     >> Hi Tom,
                     >>
                     >> Thank you very much for sharing your knowledge,
                    this will help not only
                     >> me!
                     >> As I am still working on the MySQL part, you
                    mentioned multi-master which
                     >> I
                     >> have running indeed, but I suppose there still
                    will need to be some kind
                     >> of
                     >> entering point within the cluster. How did you
                    managed this?
                     >
                     > Virtual cluster IP addresses and highly available
                    MySQL (either DRBD
                     > or MySQL replication based). Pacemaker is very
                    well capable of
                     > managing this.
                     >
                     > Of course, since you also quoted
                     >
                    http://openlife.cc/blogs/2011/july/ultimate-mysql-high-availability-solution
                     > parts of which I wholeheartedly disagree with (as
                    with all of Henrik's
                     > Pacemaker bashing, but he's of course entitled to
                    his own opinion),
                     > then you might run away screaming from anything
                    that barely mentions
                     > Pacemaker in passing. But in reality[1],
                    Pacemaker is an extremely
                     > capable HA stack that would be well suited for this.
                     >
                     > Cheers,
                     > Florian
                     >
                     > [1] To be polite, I should say "in my humble
                    opinion," but my opinion
                     > on this really isn't humble at all, so I prefer
                    being honest over
                     > being polite. :)
                     >
                     > --
                     > Need help with High Availability?
                     > http://www.hastexo.com/now
                     >
                     > _______________________________________________
                     > Mailing list: https://launchpad.net/~openstack
                    <https://launchpad.net/%7Eopenstack>
                     > Post to : openstack@xxxxxxxxxxxxxxxxxxx
                    <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
                     > Unsubscribe : https://launchpad.net/~openstack
                    <https://launchpad.net/%7Eopenstack>
                     > More help : https://help.launchpad.net/ListHelp
                     >
                     >
                     > _______________________________________________
                     > Mailing list: https://launchpad.net/~openstack
                    <https://launchpad.net/%7Eopenstack>
                     > Post to : openstack@xxxxxxxxxxxxxxxxxxx
                    <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
                     > Unsubscribe : https://launchpad.net/~openstack
                    <https://launchpad.net/%7Eopenstack>
                     > More help : https://help.launchpad.net/ListHelp
                     >


                _______________________________________________
                Mailing list: https://launchpad.net/~openstack
                <https://launchpad.net/%7Eopenstack>
                Post to : openstack@xxxxxxxxxxxxxxxxxxx
                <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
                Unsubscribe : https://launchpad.net/~openstack
                <https://launchpad.net/%7Eopenstack>
                More help : https://help.launchpad.net/ListHelp



            _______________________________________________
            Mailing list: https://launchpad.net/~openstack
            <https://launchpad.net/%7Eopenstack>
            Post to : openstack@xxxxxxxxxxxxxxxxxxx
            <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
            Unsubscribe : https://launchpad.net/~openstack
            <https://launchpad.net/%7Eopenstack>
            More help : https://help.launchpad.net/ListHelp




        --
        Ghe Rivero
        *OpenStack & Distribution Engineer
        ***www.stackops.com* <http://www.stackops.com/> |
        *ghe.rivero@xxxxxxxxxxxx
        <mailto:diego.parrilla@xxxxxxxxxxxx>****| +34 625 63 45 23 |
        skype:ghe.rivero*
        * <http://www.stackops.com/>
        *

        ****

        *

        ******************** ADVERTENCIA LEGAL ********************
        Le informamos, como destinatario de este mensaje, que el correo
        electrónico y las comunicaciones por medio de Internet no
        permiten asegurar ni garantizar la confidencialidad de los
        mensajes transmitidos, así como tampoco su integridad o su
        correcta recepción, por lo que STACKOPS TECHNOLOGIES S.L. no
        asume responsabilidad alguna por tales circunstancias. Si no
        consintiese en la utilización del correo electrónico o de las
        comunicaciones vía Internet le rogamos nos lo comunique y ponga
        en nuestro conocimiento de manera inmediata. Este mensaje va
        dirigido, de manera exclusiva, a su destinatario y contiene
        información confidencial y sujeta al secreto profesional, cuya
        divulgación no está permitida por la ley. En caso de haber
        recibido este mensaje por error, le rogamos que, de forma
        inmediata, nos lo comunique mediante correo electrónico remitido
        a nuestra atención y proceda a su eliminación, así como a la de
        cualquier documento adjunto al mismo. Asimismo, le comunicamos
        que la distribución, copia o utilización de este mensaje, o de
        cualquier documento adjunto al mismo, cualquiera que fuera su
        finalidad, están prohibidas por la ley.

        ***************** PRIVILEGED AND CONFIDENTIAL ****************
        We hereby inform you, as addressee of this message, that e-mail
        and Internet do not guarantee the confidentiality, nor the
        completeness or proper reception of the messages sent and, thus,
        STACKOPS TECHNOLOGIES S.L. does not assume any liability for
        those circumstances. Should you not agree to the use of e-mail
        or to communications via Internet, you are kindly requested to
        notify us immediately. This message is intended exclusively for
        the person to whom it is addressed and contains privileged and
        confidential information protected from disclosure by law. If
        you are not the addressee indicated in this message, you should
        immediately delete it and any attachments and notify the sender
        by reply e-mail. In such case, you are hereby notified that any
        dissemination, distribution, copying or use of this message or
        any attachments, for any purpose, is strictly prohibited by law.



        _______________________________________________
        Mailing list: https://launchpad.net/~openstack
        <https://launchpad.net/%7Eopenstack>
        Post to : openstack@xxxxxxxxxxxxxxxxxxx
        <mailto:openstack@xxxxxxxxxxxxxxxxxxx>
        Unsubscribe : https://launchpad.net/~openstack
        <https://launchpad.net/%7Eopenstack>
        More help : https://help.launchpad.net/ListHelp




    --
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Dan Wendlandt
    Nicira Networks: www.nicira.com <http://www.nicira.com>
    twitter: danwendlandt
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~



_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to     : openstack@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Follow ups

References