openstack team mailing list archive

[Monsyne Dragon <mdragon@xxxxxxxxxxxxx>]

On Oct 27, 2011, at 10:20 PM, Bryan Taylor wrote:

> Just to be clear we are talking about APIs fit for customer consumption here, not internal integrations where both ends are under our control.
> 
> On 10/27/2011 11:38 AM, George Reese wrote:
>>> I disagree. The web was designed specifically to solve the distributed scaling problem and it's based on HTTP polling. It scales pretty well. The argument against polling not scaling inevitably neglects using caching properly.
>> The web was not designed to deal with a bunch of clients needing to
>> know about infrastructure changes the instant they happen.
> Neither physics nor math were designed for that either. The CAP theorem simply doesn't allow a distributed system with an uptime guarantee to communicate changes "the instant they happen". Once you realize the best your clients can hope for is eventual consistency, the sooner you'll realize that polling is just fine.

In theory, your theory describes the situation . In practice it's different. :-> 
Yes, clients cant get truly instant notifications. What they can get is ASAP notifications. Which is what they really need/want in some situations.    

Now, this could be implemented by polling some sort of "changes since: X"  resource, as in the Fielding article, or it could be done via a push system.  Which one is appropriate depends on the relative importance of a number of things like acceptable latency of notifications since the event, frequency of notifications, requited reliability, etc to the client, and the relative cost of retrieving certain information to the serving system. 

In the case of Nova, we have use cases for both methods.    

> BTW, here's Roy Fielding's article on this subject of poll vs push.
> http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-dragons
>> And API data should not be cached. The Rackspace API used to do that,
>> and it created a mess.
> I'm not sure what you are referring to, but this is a classic strawman. Somebody implemented a "mess" using caching, so caching is bad!? You didn't say what the mess was, so there's no way to even evaluate your statement.
>>> Push doesn't scaled because it requires the server to know about every client and track conversational state with them.
>> No, it doesn't. You push changes as they occur to a message queue. A
>> separate system tracks subscribers and sends them out. There is no
>> conversational state if done right.
> 
> A "separate" system? That's why you think it's simple -- you push the hard part outside of your box and claim victory. It's not a separate system, it's all one big cloud. If there are N interested clients the process you described requires O(N) resources. Moving it to another tier means it's somebody else's O(N) resources. You are illustrating Fielding's point in the article above: "People need to understand that general-purpose PubSub is not a solution to scalability problems — it simply moves the problem somewhere else, and usually to a place that is inversely supported by the economics. "
> 
> How exactly does this separate system know where to "send them out" to? Each client has to tell it and you have to store it and look it up on a per outbound message basis. And keep it accurate. Customers just love keeping you informed of where they want to send their messages. Do you know what happens when they forget to tell you they moved and they don't get the message? They blame you and ask for a credit memo. And do you know what happens when you tell them no. They go to your competitor. If there is no conversational state, then you aren't waiting for an acknowledgement from the other side for each message and you can't prove that it was delivered or even try again.

It is a separate system, if you can make use of systems someone else has already designed & tested for you.  The easiest work is that someone else has already done for you.  (For example, we don't worry about making use of the worlds largest/oldest push notification system (email) where needed. Postfix has already been designed for us. ) Hubs (and other pubsub/queueing) systems already exist.  And the reliability issue is a red herring.  push/messaging systems are often of less than guaranteed reliability, which is fine if communicated. There are many use cases where that is "good enough".  

In any case the order of  resources needed is closer to O(N * Fp) for a polling system, where N is the number of clients, and Fp is the polling frequency  (i.e. the inverse of the minimum acceptable latency).  The order for a push system is roughly O(N * Fe) where N is the number of clients, and Fe is the frequency of events you need to notify clients about.  

Which is appropriate depends on if Fe < Fp.  If your clients need notification at low latency of infrequent events, they will pound your API servers thru the floor with their polling when  a push system will be hardly doing any work.  On the other hand, if the client needs a huge bunch of information, but doesn't care so much about latency, then polling is better.  [*]

Nova has clients and information with both types of characters.   We can already provide for polling via the API's, we should provide for both methods. 

[*] this is assuming the cost on the system side of  providing the data is the same.  This is not always true, as sometimes in distributed systems, answering the question "Hey, has anything in X changed recently" can be much more expensive than the system telling someone "Hey, X just changed it's Z", since in the first case, the part of the system being asked needs to find which other part(s) have that information. 


--
	Monsyne M. Dragon
	OpenStack/Nova 
	cell 210-441-0965
	work x 5014190