launchpad-dev team mailing list archive

[Gavin Panella <gavin.panella@xxxxxxxxxxxxx>]

Hi,

I've been exploring a hunch that negotiation and renegotiation of SSL
sessions might be causing Launchpad some issues.

Most of Launchpad's domain names have two addresses:

  $ host bugs.edge.launchpad.net
  bugs.edge.launchpad.net has address 91.189.89.226
  bugs.edge.launchpad.net has address 91.189.89.227

If your browser sets up an SSL session with one of them, it is not
shared with the other. This is the output of the attached ssltest
script:

  91.189.89.226
    Session-ID: 579A6CA6AE1C379CE61027E9AE3BB91A20191AF09B7B8D106151A87C78C5500B
  91.189.89.227
    Session-ID: A3BE2E977CC3C787A709FCFB00E831631CA0444AFA7396B4C65B846E1E3E5C8F
  91.189.89.226
    Session-ID: 579A6CA6AE1C379CE61027E9AE3BB91A20191AF09B7B8D106151A87C78C5500B

When it connects to the 227 host it attempts to use the session
created in the previous step. You can see that it succeeds in reusing
the saved session when it again connects to the 226 host.

This means - I think - that a web browser must negotiate a session
with both hosts. That's a fair bit of start-up penalty.

Using resource tracking in Chromium's developer tools, I can see that
the start-up time (i.e. the time from clicking a link to the first
byte of the HTTP response) on each page hops between about 2s and 4s
(For information, I'm less than 200km from the data-centre and very
close to my local exchange).

However, adding the following to /etc/hosts:

  91.189.89.226 bugs.edge.launchpad.net

keeps the start-up time consistently at around 2s. There could be
other reasons for this, but I'm going with my session renegotiation
theory for now.

Next, does the web browser maintain a session per address or per host
name? If it's per *name* this could mean that sessions are short lived
and constantly being renegotiated.

Some manual tests with Wireshark, just requesting /@@/launchpad.png
over and over, gave the following:

{{{
Filter: ip.addr == 91.189.89.226 || ip.addr == 91.189.89.227

Looking at Client Hello and Server Hello messages:

->226 Session ID: 20AB0905EE73C64A9623900A2B48D7EAB92D2CBDD60BB8DA...
226-> Session ID: 20AB0905EE73C64A9623900A2B48D7EAB92D2CBDD60BB8DA...

->227 Session ID: F8352810450CA4087E365C733038FAC0D2A41783B566E85C...
227-> Session ID: 288FD8B423F613C03485608F8753BB1013D1A6563849E992...

->226 Session ID: 20AB0905EE73C64A9623900A2B48D7EAB92D2CBDD60BB8DA...
226-> Session ID: 20AB0905EE73C64A9623900A2B48D7EAB92D2CBDD60BB8DA...

->227 Session ID: 288FD8B423F613C03485608F8753BB1013D1A6563849E992...
227-> Session ID: 288FD8B423F613C03485608F8753BB1013D1A6563849E992...

->226 Session ID: 20AB0905EE73C64A9623900A2B48D7EAB92D2CBDD60BB8DA...
226-> Session ID: 7F74AA5764428A4C8005CBC0EE9E2B77D481103EA3EBC668...

->227 Session ID: 288FD8B423F613C03485608F8753BB1013D1A6563849E992...
227-> Session ID: 288FD8B423F613C03485608F8753BB1013D1A6563849E992...

->226 Session ID: 7F74AA5764428A4C8005CBC0EE9E2B77D481103EA3EBC668...
226-> Session ID: 7F74AA5764428A4C8005CBC0EE9E2B77D481103EA3EBC668...

->227 Session ID: 288FD8B423F613C03485608F8753BB1013D1A6563849E992...
227-> Session ID: 84B4245FE9AD3A4A0BA46944FA3830DA740D1F8E3747D4BD...
}}}

Each pair of lines is a request and a response to the specified
address, 226 or 227.

As you can see, the browser - Chromium again - maintains a session per
*address*, which is good news. Other browsers may behave differently.

Also, most of the time the server responds with the same session ID;
the session is reused. More good news.

What you can't see is that, when the server responds with a different
session ID, it had not been very long - less than 1 minute at a guess,
but I didn't measure it - since the session was last used. In other
words, sessions are being discarded by the server more rapidly than by
the client.

I am by no means an expert on SSL - this is all done from cobbled
together knowledge - so it would be nice if someone else could confirm
what I've found, or, equally, show it to be incorrect.

If what I am seeing is valid, then I can see two - I assume relatively
easy - things we can do to help:

- Only publish one address for each vhost.

- In Apache, increase SSLSessionCacheTimeout to, say, 10 minutes.

Gavin.

Attachment: ssltest
Description: Binary data