launchpad-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On Thu, May 19, 2011 at 5:46 AM, Martin Pool <mbp@xxxxxxxxxxxxx> wrote:
> On 17 May 2011 22:42, Robert Collins <robertc@xxxxxxxxxxxxxxxxx> wrote:
>
> You're probably right to split the internal from the external API.
> Much as I'd like to see the external one improved, you are right that
> the internal one is more important right now.
>
> But I can't quite get away from thinking about how they ought to relate. :)

Sure. I think we may have /some/ apis we choose to expose externally,
but the default is internal. And this isn't a question of openness:
the control protocol for the blob-store for Launchpad is only relevant
to LP servers & scripts. The blob store client protocol is useful for
anyone getting data from / putting it into Launchpad. And if we
productise it, the blob store control protocol may well be generally
useful. But there is a /significant/ gap between the before and after
of that productisation. I don't think we should commit to productising
by default.

We may want to productise as much as 50% of our services, so its a
good question. But those that we don't won't fit in the 'launchpad
API' stack because lazr.restful exposes /one/ service - and we're
going to have N separate services. We /might/ use lazr.restful for
them, but its not a no-brainer, and it is a separate question.

...

> I agree they are different, though many of those points apply to both
> the internal webservices approach and external clients:
>  * the things that would make launchpadlib inadequate for Launchpad's
> own front end also make it inadequate for writing interesting external
> clients (with concurrency and talking to multiple endpoints being
> particular cases);

Thats true, and something that makes me sad. I don't think fixing that
as a prerequisite to changing LP to a services model makes sense: its
more investment in something useful for exactly one stack - and we
know we have heterogeneous stacks to deal with.

>  * people want to get higher level results than quasi-db-rows per round trip

Sure, but the slices they want are not the slices the internal service
needs, by and large. (Alternatively, show that they are :P)

>  * I'm not sure quite what "make creating and destroying services as
> lightweight as possible" means but it certainly sounds like I want it
> for external clients too ;-)

I don't think you do. For external clients you want stable API's.
Internally we want to evolve rapidly.

>  * some clients may well be web apps
>  * merely having an unstable API does not seem like a strong reason
> not to publish it: people would rather have an unstable way to access
> something than nothing at all (or more likely, screenscraping.)

We have an unstable API for the launchpadlib API already, but it has a
high cost to our users when we do break something they use. We really
don't want external clients talking to internal components. We don't
want the overhead of OAuth on internal components, we don't want to
have to do internal-id to url translation on all API calls, etc.

So please please please:
 - external API is external API
 - internal API is internal API

*if* and when we want to take something internal, we can explicitly promote it.

> A lot of my questions, and your responses, are about this being a
> different problem from the existing external API, and that makes
> sense.  I suppose my question then would be: could it be possible to
> provide an external authenticated version of the internal transport,
> or some subsection of it, or if not, why?

Of course its possible, but it's a separate project, with its own
risks, benefits, security considerations etc. I don't want us to have
to pay those overheads internally by default.

> As a story: I'd like it to be possible to prototype a new feature in
> Launchpad by running the front end for it on an external server. If it
> requires some back end changes I'd like to put up a branch for
> Launchpad that adds this, and then to keep doing the frontend
> development externally.  If it turns out well, the app could
> eventually become indistinguishable from Launchpad proper.  I wonder
> how this could work.

If its possible to do this, great. I don't see it as something to aim
for. The resources for bandwidth, time, security, reliability are
extremely hard to provision outside of the datacentre. I want to make
bringing up a prototype connected to staging much cheaper, for
instance.

> The Bazaar team is coming to the Launchpad Thunderdome (whatever that
> is) and I'd like us to do something aligned with the goals of the
> sprint and also connecting lp and bzr.  Perhaps adding better apis to
> loggerhead and using them from lp would be a good specific topic.
>
> loggerhead seems like an example where the same apis could be useful
> internally and externally.  But perhaps not everything is in both
> categories.

Loggerhead is already an external service, so as an example its
biased. A better example of an internal service is the 'buildd slave'
service. Thats one that there is /no way/ we'd be comfortable exposing
without some -major- overhauling. A reasonable example of an internal
service we might want to expose is the group membership/directory
service.

-Rob