launchpad-dev team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

On Thu, 2009-12-17 at 15:24 -0600, Edwin Grubbs wrote:
> On Wed, Nov 18, 2009 at 9:19 PM, Robert Collins

> > Don't use or depend on HTTP pipelining.
> >
> > Its a security and correctness nightmare.
> >
> > -Rob
> 
> Since you didn't define which security and correctness nightmares that
> HTTP pipelining introduces, I tried to google for an answer. 

Security:
 - http smuggling is the main attack vector

Correctness:
 - many proxies have pipelining turned off, if we ever do do http, we'll
run into that.
 - the semantics of pipelining combined with request delimiting and
interoperation with 1.0 clients (including many proxies) are very
complex. Bugs such as dropped responses, indefinite buffering, response
sequencing issues are not uncommon.

> All I
> could find were criticisms that pipeling would make it hard to detect
> DOS attacks and that its performance improvements are limited because
> the spec requires that the responses are sent in the same order as the
> requests, which can prevent quick results from being sent before slow
> results are finished.

Error cases are poorly defined, dropped connections more so. Its not
that its impossible to get right, its just very hard, and you will
depend on every client doing it right. Add to that that large objects
saturate the pipeline - its not like e.g. atm where large objects are
fragmented.

Current HTTP recommendations are now for more connections per client and
no pipelining  (see the HTTPbis work).

>  Rolling our own solution would allow us to
> circumvent pipelining's ordering requirement, but I am wondering if we
> will run into all of its other drawbacks such as networking equipment
> servers handling more and more long running connections.

if you're talking long-poll, it has its own issues, but as the primary
overhead is simply a tcp socket (given an appropriate networking
library), its becoming a very common idiom. I don't know how prepared
zope is for it, with its thread-per-request model.

> I stumbled upon an interesting proof of concept of this idea. The
> DUI.Stream javascript library actually parses each chunk of data as it
> comes in by polling the request.responseText attribute every 15ms
> after the request enters the LOADING state. It expects the response to
> be in multipart/mixed mime format.
> 
> http://about.digg.com/blog/duistream-and-mxhr

That sounds like 'trickle feed' rather than 'long-poll', which is a
different thing again. Trickle feed often behaves poorly with virus
scanners, (both host and intermediary based) which buffer all data in
responses until they are sure its not an attack.

My advice, to get the most out of http and have the least
other-equipment-and-software issues is to:
 - send one http request per 'give me stuff' request
 - have a request API that lets you express 'lots of stuff' in a single
request.See for instance the amazon ec2 api.
 - do not expect that you will get less than all of a response (for the
trickle feed buffering issue and also the issue that responses can be
interrupted: until its all there you may not get it all)
 - make use of plenty of connections for concurrency (browsers are
upping their defaults as we speak, nothing special is needed to make
this happen)

-Rob

Attachment: signature.asc
Description: This is a digitally signed message part