openstack team mailing list archive

[Eric Day <eday@xxxxxxxxxxxx>]

Haha, 'nerdgasms'. :)

I think your last line sums it up, there are going to be many use
cases. There will be applications that have long-lived messages
(maybe a cron job pulling them out once in a while), and high
throughput applications that need short-lived messages (web apps that
need distributed work completed ASAP). There will be short-lived
connections from clients and polling workers, but also long-lived
connections for long-polling workers.

Message size will of course vary and the max will be configurable,
but it will probably be on the smaller side. I think a reasonable
max message size would be around 64k, and anything bigger you should
stick it in swift and pass a URL.

Persistence will also vary due to it being pluggable. Some apps may
submit messages that need guaranteed delivery so will need a commit
to disk before responding, where others simply toss it in and hope
for the best (in-memory queues).

We need to rely on our knowledge of how we have built and used queues
in the past and hopefully roll that into a useful service. There isn't
going to be list of specific metrics we can meaningfully measure things
against up front due to the variety of uses, these will evolve with
the project and it's usage.

The main question right now is where to land on the spectrum of service
efficiency vs ease of development (C/C++ on one end and Python on
the other). It seems we're landing in the middle with Erlang. :)

-Eric

On Fri, Feb 18, 2011 at 10:29:56PM +0000, Curtis Carter wrote:
> What are the types of loads we expect the system to handle? Without knowing that this is all guesswork and nerdgasms.
> 
> The amount of connections, length and frequency of connections,  number of messages, average size of message, max size of message, total size of messages, average time expected in queue, max time expected in queue, and the exact requirements for persistence will all need to be taken into consideration. (and I'm sure more, I'm not a queue expert)
> 
> We should definitely learn from past projects and mistakes. ie Twitter with Starling
> As far as I know their backend is still Kestrel (Using Scala with actors)
> 
> If we are trying to build a queue system that can handle any type of usage there will be tradeoffs.
> 
> 
> On Feb 18, 2011, at 1:19 PM, Schwartz, Philip Marc (LNG-BCT) wrote:
> 
> > One option might be to build this directly in python using a high speed http frame work like Facebook's Tornado. I have created a similar proprietary queue system here internally using Tornado with a REST based access system to it. The framework is pretty light weight and performs exceptionally well. We have seen in testing so far 130k ops per second with the queue spread on 4 nodes running in 8 threads on each node with a nginx load balancer in front of it. The avg was just under 35k ops per second when run in 8 threads on a single node.
> > 
> > My solution here is not transactional, but can easily be updated to it because of the ease of coding with the framework.
> > 
> > So don't count out python up front for performance as this system has proven to beat many queue systems I tested written in c++.
> > 
> > Thank You,
> > Philip Schwartz
> > Software Engineering
> > LexisNexis RS
> > O - 561 999 4472
> > C - 954 290 4024
> > 
> > -----Original Message-----
> > From: openstack-bounces+philip.schwartz=lexisnexis.com@xxxxxxxxxxxxxxxxxxx [mailto:openstack-bounces+philip.schwartz=lexisnexis.com@xxxxxxxxxxxxxxxxxxx] On Behalf Of Eric Day
> > Sent: Friday, February 18, 2011 1:54 PM
> > To: ksankar@xxxxxxxxxxxxxx
> > Cc: openstack@xxxxxxxxxxxxxxxxxxx
> > Subject: Re: [Openstack] Queue Service, next steps
> > 
> > I checked out the CDMI spec, and while it covers the basics, it doesn't
> > seem to include features for visibility, concurrency control (multiple
> > workers grabbing messages via atomic operations), or long-polling. We
> > can still support it by having a CMDI protocol module if the protocol
> > gains more traction, but at this stage I wouldn't want to restrict
> > ourselves to just this protocol (and the features it describes)
> > this early on in a project.
> > 
> > Do you know what queue services use the CDMI spec?
> > 
> > As for Erlang, I'm currently looking into WebMachine and a couple
> > other HTTP/REST frameworks for relative performance comparison. The
> > Erlang core has quite a bit of HTTP protocol parsing built into it to
> > (written in C, passed in via socket reads), so for something simple
> > and lightweight we can use that. I'll be writing up more on this
> > later as I investigate.
> > 
> > -Eric
> > 
> > On Fri, Feb 18, 2011 at 09:23:28AM -0700, ksankar@xxxxxxxxxxxxxx wrote:
> >>     * On the topic of a REST API for the queue, I think implementing the
> >>       CDMI queue APIs might be a good choice.
> >>     * The spec is at
> >>       http://www.snia.org/tech_activities/publicreview/CDMI_Spec_v1.01f_DRAFT.pdf.
> >>       There are a few other things, but focus on Section 11 P.104, Queue
> >>       Object Resources.
> >>     * BTW, <cdmi-queue> is also an approved media-type
> >>       [http://www.iana.org/assignments/media-types/application/index.html]
> >>     * Would be happy to implement the CDMI REST interface, if all think this
> >>       is a good idea
> >>     * And on the topic of Erlang, it is viable. I have done some work with
> >>       it. If we also use WebMachine, REST interface would be easier.
> >>   Cheers
> >>   <k/>
> >> 
> >>     -------- Original Message --------
> >>     Subject: Re: [Openstack] Queue Service, next steps
> >>     From: Daniel Salinas <imsplitbit@xxxxxxxxx>
> >>     Date: Fri, February 18, 2011 8:20 am
> >>     To: "openstack@xxxxxxxxxxxxxxxxxxx" <openstack@xxxxxxxxxxxxxxxxxxx>
> >> 
> >>     I would totally be interested in an Erlang bootstrap. In SAT. Lets see
> >>     if
> >>     we can get a group together.
> >> 
> >>     On 2/18/11 4:03 AM, "Curtis Carter" <curtis.carter@xxxxxxxxxxxxx>;
> >>     wrote:
> >> 
> >>> I'm in if it's done in Erlang.
> >>> I am willing to give talks or help anyone wanting to learn Erlang in
> >>     the
> >>> Austin/San Antonio area as well.
> >>> 
> >>> On Feb 18, 2011, at 12:11 AM, Eric Day wrote:
> >>> 
> >>>> Duck farm? :)
> >>>> 
> >>>> Are you two concerned about building a developer community around a
> >>>> project in Erlang? I'm all for going that route if other folks are
> >>>> comfortable with it.
> >>>> 
> >>>> I also have some concern around the speed of Erlang. It's great,
> >>>> especially if you know what primitives can be expensive and how to
> >>>> tame beam, but it will always be slower (sometimes significantly)
> >>>> that C/C++ for some tasks.
> >>>> 
> >>>> -Eric
> >>>> 
> >>>> On Thu, Feb 17, 2011 at 03:26:09PM -0800, Joshua McKenty wrote:
> >>>>> +1 for Erlang, as long as we have a duck farm.
> >>>>> 
> >>>>> On Thu, Feb 17, 2011 at 2:25 PM, Devin Carlen
> >>>>> <devin.carlen@xxxxxxxxx>;
> >>>>> wrote:
> >>>>> 
> >>>>> I'll put in a +1 for Erlang as an OpenStack supported platform.
> >>>>> We'd be
> >>>>> able to write a stable queue with a much more concise code base,
> >>>>> and
> >>>>> this is project would be a great fit for Erlang.
> >>>>> Devin
> >>>>> On Feb 17, 2011, at 2:21 PM, Eric Day wrote:
> >>>>> 
> >>>>>> Thanks to everyone who gave great feedback on the first queue
> >>     service
> >>>>>> thread. I've updated the wiki page to include the suggestions.
> >>>>>> 
> >>>>>> http://wiki.openstack.org/QueueService
> >>>>>> 
> >>>>>> With a decent vision of what we want to build, the next step is
> >>>>>> figuring out how. In a previous thread it was suggested that the
> >>>>>> preferred languages for OpenStack projects are Python, C, and
> >>>>>> C++. Since there is an emphasis on speed and efficiency for the
> >>>>>> queue service, I suggest we use C++. I expect this service to be
> >>>>>> CPU bound and would benefit being able to leverage multiple cores
> >>>>>> efficiently (within the same process), so I don't think Python
> >>>>>> is a good fit. I think C++ is a better fit than C due to the need
> >>>>>> for modular interfaces. While this can obviously be done in C, C++
> >>>>>> APIs are more concise and much less error prone. The OO style will
> >>>>>> also make it easier for Python developers who also want to learn
> >>     and
> >>>>>> assist with C++ projects.
> >>>>>> 
> >>>>>> Erlang is not on the preferred lists, but I would also put it out
> >>>>>> there as an option. While it may be a great fit for a project like
> >>>>>> this, I worry it won't attract the developer resources since Erlang
> >>>>>> isn't really a first-class language yet.
> >>>>>> 
> >>>>>> If we decide to take the C++ path, I propose using a modular
> >>>>>> application framework I've been working on over the past year
> >>     (mostly
> >>>>>> in my spare time). It provides a simple module programming
> >>     interface
> >>>>>> with dependency tracking (kind of like Linux kernel modules). It
> >>>>>> already provides a multi-threaded event module (currently based on
> >>>>>> libevent, but this is pluggable) with simple networking
> >>     abstractions
> >>>>>> built on top of it. We should be able to dive in an start writing
> >>     the
> >>>>>> HTTP protocol module and queue processing modules. You can check
> >>     out
> >>>>>> the current project at:
> >>>>>> 
> >>>>>> https://launchpad.net/scalestack
> >>>>>> http://scalestack.org/
> >>>>>> 
> >>>>>> The intention of using a framework like this is so we can easily
> >>     reuse
> >>>>>> the other modules (auth, HTTP, logging, ...) for other OpenStack
> >>>>>> services in the future. Much like we use Eventlet, WSGI, etc. (and
> >>>>>> eventually openstack-common) for Python, we could prefer using the
> >>>>>> modules in Scale Stack for lower level projects.
> >>>>>> 
> >>>>>> Thoughts?
> >>>>>> -Eric
> >>>>>> 
> >>>>>> _______________________________________________
> >>>>>> Mailing list: https://launchpad.net/~openstack
> >>>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
> >>>>>> Unsubscribe : https://launchpad.net/~openstack
> >>>>>> More help : https://help.launchpad.net/ListHelp
> >>>>> 
> >>>>> _______________________________________________
> >>>>> Mailing list: https://launchpad.net/~openstack
> >>>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
> >>>>> Unsubscribe : https://launchpad.net/~openstack
> >>>>> More help : https://help.launchpad.net/ListHelp
> >>>> 
> >>>> _______________________________________________
> >>>> Mailing list: https://launchpad.net/~openstack
> >>>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
> >>>> Unsubscribe : https://launchpad.net/~openstack
> >>>> More help : https://help.launchpad.net/ListHelp
> >>> 
> >>> 
> >>> 
> >>> Confidentiality Notice: This e-mail message (including any attached or
> >>> embedded documents) is intended for the exclusive and confidential use
> >>     of
> >>> the
> >>> individual or entity to which this message is addressed, and unless
> >>> otherwise
> >>> expressly indicated, is confidential and privileged information of
> >>> Rackspace.
> >>> Any dissemination, distribution or copying of the enclosed material is
> >>> prohibited.
> >>> If you receive this transmission in error, please notify us immediately
> >>> by e-mail
> >>> at abuse@xxxxxxxxxxxxx, and delete the original message.
> >>> Your cooperation is appreciated.
> >>> 
> >>> 
> >>> _______________________________________________
> >>> Mailing list: https://launchpad.net/~openstack
> >>> Post to : openstack@xxxxxxxxxxxxxxxxxxx
> >>> Unsubscribe : https://launchpad.net/~openstack
> >>> More help : https://help.launchpad.net/ListHelp
> >> 
> >>     _______________________________________________
> >>     Mailing list: https://launchpad.net/~openstack
> >>     Post to : openstack@xxxxxxxxxxxxxxxxxxx
> >>     Unsubscribe : https://launchpad.net/~openstack
> >>     More help : https://help.launchpad.net/ListHelp
> > 
> >> _______________________________________________
> >> Mailing list: https://launchpad.net/~openstack
> >> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> >> Unsubscribe : https://launchpad.net/~openstack
> >> More help   : https://help.launchpad.net/ListHelp
> > 
> > 
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> > 
> > This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law.  If you are not the intended recipient, you should delete this message.  Any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited.
> > 
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
> Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of the
> individual or entity to which this message is addressed, and unless otherwise
> expressly indicated, is confidential and privileged information of Rackspace.
> Any dissemination, distribution or copying of the enclosed material is prohibited.
> If you receive this transmission in error, please notify us immediately by e-mail
> at abuse@xxxxxxxxxxxxx, and delete the original message.
> Your cooperation is appreciated.