launchpad-dev team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

On Wed, Nov 17, 2010 at 4:54 AM, Robert Collins
<robert.collins@xxxxxxxxxxxxx> wrote:
> So, when I started this role, I put performance and continuous
> deployment - release features when they are done - as my kanban cards.
> These things are where I've been putting all my ongoing-project focus
> (vs day to day stuff like helping with design and code discussions).
>
> It seems to me that our continuous deployment project is nearly
> complete, and its time to start thinking about what I should take on
> to replace it.

I'm very glad that so many folk replied to this thread.

I'll more details and exposition below, but the short story is - its
my opinion that fixing our data mapping story is the highest leverage
item that requires TA input to achieve pervasive implementation and
deployment. It follows that its the highest priority item to put in as
a primary project, and thus will be in my Kanban very soon.

My basic plan is this:
 - between now and the epic, bootstrap a skeleton providing the
interface, test and metrics we need.
 - at the epic recruit a task force of 4-5 interested developers to be
seconded [exact details to follow - I don't know the mechanics. It
might be something like 'for 6 months, a new task force to be
constituted if that isn't long enough']
 - flesh out the skeleton and propogate it across the code base - no
module left behind.

Now, a breakdown of why:

> The candidates I have already are:
>  - fixing a bunch of baseline issues - like: overuse of interfaces,
> scripts not establishing participations, sqlbase migration. Time
> consuming, tricky things that need some attention to detail, but which
> when done will significantly simplify much of our code base.

The benefits of these things would be removal of a lot of corner cases
in plumbing. That would make it easier to create extensions (like
feature flags, the new timeline, and so forth). That would help us
create better infrastructure level improvements, but it wouldn't solve
many immediate problems in Launchpad itself.

>  - test suite parallelisation: parallelisation initially, then
> parallelisation in our CI/landing robot

This would, as many folk have noted, dramatically improve our cycle
time and thus is definitely well worth doing. That *hard stuff* is
done now (I have two branches with minor errors remaining - if anyone
wants to land those that would be awesome - or I will plug away as
time permits). So while its important, I don't think its a 6 month
full time task - its going to be very choppy to get all the kinks out.

>  - data mapping layer: Build and deploy across the whole code base a
> dedicated persistence layer.
>   This would be a choke point through which all database access goes,
> would include the group based idiom I've been rabbiting on about, and
> to be successful would define an iron-clad contract which we can
> confidently replace with a testable test double. I'd be looking to
> form a working-group for this - a group of interested folk seconded
> from their regular team to work with me on developing and deploying
> this project wide.

We have > 50 timeout bugs, many of which have many hundred, or
thousands of discrete queries being made. This task speaks directly to
those bugs and to our ongoing ability to efficiently write fast
database layer code. It has the ability to dramatically improve test
suite performance by splitting our logic into two clear layers where
the data access layer can be switched out for the test suite : and I'd
make doing that a success condition for this project. The benefits
then are:
 - faster page loads
 - faster test suite
 - cleaner code throughout the logic layer

>  - faster/easier stock page creation: Suggested by Gary, this would
> aim to massively reduce the overhead in the non-ajax form development,
> so that we can focus on ajax form development. For instance, perhaps
> we'd get rid of generated forms and use node.js to render pages server
> side when working with a javascriptless browser.

This concept needs more baking I think - the whole question - do we do
AJAX only, AJAX and template forms, both AJAX and custom forms - that
all needs more examination and testing - testing of search engine
effectiveness on AJAX only sites, examination of a better testing
story for js - a precondition I think of going heavy-ajax, and so
forth.

-Rob