launchpad-reviewers team mailing list archive

[Andrew Bennetts <andrew.bennetts@xxxxxxxxxxxxx>]

The proposal to merge lp:~spiv/launchpad/haproxy-for-twisted-services into lp:launchpad has been updated.

Description changed to:

This enhances the codehosting SSH daemon to:

 * not exit immediately upon SIGTERM (or SIGINT), instead stop listening on the SSH port immediately but postpone exiting until all existing connections have finished;
 * provide an HTTP service that returns status code 200 (OK) normally, and 503 (Service Unavailable) after SIGTERM/INT.

The idea we can use multiple instances of this SSH daemon + a frontend like HAProxy to provide graceful server restarts the same sort of way we restart the lpnet services:
 * SIGTERM some fraction of the codehosting instances
 * Those instances will stop listening on their port
 * The frontend will observe (via the HTTP status service) that those services are no longer to be used for new connections
 * when all outstanding connections on those instances have finished they will stop, and then can be restarted (with new configs or code or whatever)
 * repeat this process for the other instances

The code itself is fairly small: it adds some infrastructure in lp.services.twistedsupport.gracefulshutdown, then uses them in sftp.tac.  As a followup I'll apply these helpers to other tac files too (librarian, poppy, maybe others?).

The significant part of the HTTP service is the the status code (200 vs. 503), but it also emits some basic information about the number of connections and their addresses for the benefit of humans looking at the service.

There's a first guess at likely production config changes at <lp:~spiv/lp-production-configs/codehost-codehost>: basically we need to allocate ports (and OOPS prefixes) for each instance, just like we do for the lpnet* instances.  We'll also need to arrange for HAProxy or whatever frontend to be deployed too.

Some of this code is fairly generic, and perhaps should be contributed to upstream (Twisted).

We should obviously take care to make sure this works nicely on qastaging before putting this on production!



For more details, see:
https://code.launchpad.net/~spiv/launchpad/haproxy-for-twisted-services/+merge/46996
-- 
https://code.launchpad.net/~spiv/launchpad/haproxy-for-twisted-services/+merge/46996
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~spiv/launchpad/haproxy-for-twisted-services into lp:launchpad.