launchpad-dev team mailing list archive

[Martin Pool <mbp@xxxxxxxxxxxxx>]

On 17 May 2011 22:42, Robert Collins <robertc@xxxxxxxxxxxxxxxxx> wrote:

You're probably right to split the internal from the external API.
Much as I'd like to see the external one improved, you are right that
the internal one is more important right now.

But I can't quite get away from thinking about how they ought to relate. :)

Tangentially to my post, I saw a demo of Orchestra, and despite that
it is only alpha-level it looks quite impressive, and could be a good
way to spin up interconnected vms.

> > * The lp api is really good in some ways (mostly, coverage) but
> > limited in others, and people speak of lazr.restful and co as being
> > hard to change or to represent particular things.  So it seems like
> > this project could either snag on that or be a powerful motivator to
> > improve it.
>
> Indeed. I think we have different constraints and requirements for
> internal apis vs external. We don't need long term support (we control
> the clients), we don't want totally different dialects when things
> change (we want different calls to be isolated so we can be upgrading
> individual points not everything in lock-step), we simply can't use
> launchpadlib (as I describe in the wiki page), and we need to make
> creating and destroying services as lightweight as possible, we want
> concurrent requests, and requests to different services; we need our
> own business logic (so want data objects not mapped objects). In the
> Template/API service I suggest we could split the public API out to
> the front end servers; what we use internally is then totally
> unrelated (but it may still be RESTful).

I agree they are different, though many of those points apply to both
the internal webservices approach and external clients:
 * the things that would make launchpadlib inadequate for Launchpad's
own front end also make it inadequate for writing interesting external
clients (with concurrency and talking to multiple endpoints being
particular cases);
 * people want to get higher level results than quasi-db-rows per round trip
 * I'm not sure quite what "make creating and destroying services as
lightweight as possible" means but it certainly sounds like I want it
for external clients too ;-)
 * some clients may well be web apps
 * merely having an unstable API does not seem like a strong reason
not to publish it: people would rather have an unstable way to access
something than nothing at all (or more likely, screenscraping.)

A lot of my questions, and your responses, are about this being a
different problem from the existing external API, and that makes
sense.  I suppose my question then would be: could it be possible to
provide an external authenticated version of the internal transport,
or some subsection of it, or if not, why?

As a story: I'd like it to be possible to prototype a new feature in
Launchpad by running the front end for it on an external server. If it
requires some back end changes I'd like to put up a branch for
Launchpad that adds this, and then to keep doing the frontend
development externally.  If it turns out well, the app could
eventually become indistinguishable from Launchpad proper.  I wonder
how this could work.

> Transactions are slow. GPG is slow. Consistency checks are slow.
> Initialisation of data on disk is slow.
>
> -> fakes from day one
>
> This is something I wish we had done in bzr much earlier - it would
> have informed our design for things like tree transform (which doesn't
> run on a transport) by making us measure the cost of the abstraction
> vs the cost of using the full implementation in all our other tests.
> (Tree transform is nice - this isn't a slight, merely an example I
> knew of offhand).

I don't draw that conclusion from our experience with bzr: good
abstractions and good abstraction-aware tests are great, but fakes
should be used when there is a good reason (no sooner or later).  Tree
transform is actually an interesting example, because there were three
possibilities:

0 - what actually happened: it's directly bound to local disk: this is
asserted to be the fastest approach, but also inflexible, and we don't
actually *know* whether it's any faster than going through an
abstraction
1- go through the Transport abstraction, to local disk: then later on
you can play tricks with simulating a FAT or NFS disk or just use a
fast in-memory transport; I hope to get bzr to this level
2- go through a Transport abstraction to a MemoryTransport fake

I think the important thing is to start out at stage 1; you can always
go to stage 2 later.  But it may turn out that you soon throw away or
change the underlying abstraction, and then having a fake of it was
waste.  Running against a fake has dangers too: it may not match;
there can be bugs that occur only when integrated; the fake can skew
from the other implementations; and I think some of these ideas can
perversely lead to tests that know too much about the code under test.

(But this is really a side point; if people add fakes for everything
on the first day I will be very curious whether it really turns out
well.)

> >  * So a specific step towards this could be using loggerhead's json
> > api to display some branch content inline in the main application?  By
> > talking to it from the client, or by having the webapp templates proxy
> > it?
>
> Right. If we want webapp clients to view it we either need xhr
> permission glue, or to map loggerhead into the lp namespace using
> apache's url routing facilities.
> If we do it ourselves in the template layer, we don't need either of
> those things but we will be dependent on its reponsiveness to meet our
> page render times.
>
> We may want to do both depending on the situation. I'd certainly start
> by sending clients to it (e.g. map it into the main namespace via
> apache).

The Bazaar team is coming to the Launchpad Thunderdome (whatever that
is) and I'd like us to do something aligned with the goals of the
sprint and also connecting lp and bzr.  Perhaps adding better apis to
loggerhead and using them from lp would be a good specific topic.

loggerhead seems like an example where the same apis could be useful
internally and externally.  But perhaps not everything is in both
categories.

> Having got a handle on the major performance
> issues that were driving our users batty (so that now they are asking
> about features more than speed) I think our next major single-minded
> theme needs to be internal development efficiency, which this proposal
> is all about (even though it seems to be just architecture :)).

<hearty applause>

Martin