launchpad-reviewers team mailing list archive

[Colin Watson <mp+420028@xxxxxxxxxxxxxxxxxx>]

I discussed this with Alex on Friday.

For CI builds in the `soss` distribution, the behaviour we want is to send a reasonable baseline list of Artifactory repositories to builds by default, and to have a way to supplement that for individual builds, probably with a new job configuration option in `.launchpad.yaml`.  The list of repositories we should send by default is all of the ones in https://canonical.jfrog.io/ui/admin/repositories/virtual matching `soss-*-stable`, excluding `soss-stable` which was an experiment in combining all of them in a single virtual repository but which unfortunately doesn't seem to work.  This is analogous to the Ubuntu primary archive, except that there's more than one of them.

As mentioned before, CI builds that aren't in the `soss` distribution shouldn't send Artifactory repositories at all.

We can determine whether a CI build belongs to a particular distribution by checking `IDistributionSourcePackage.providedBy(build.git_repository.target)` and then looking at `build.git_repository.target.distribution` (this is kind of awkward and it might be nice if `IGitRepository` had a `pillar` attribute to encapsulate it all and handle the various cases, but you don't have to do that now).  Aside from that, our data model isn't going to do a good job at supporting this at the moment, because distributions don't really have a robust database link to Artifactory information, and we don't have a way to record which Artifactory repositories are "primary".

However, we have imminent deadlines to meet: so for the time being, let's do something basic but functional.  I recommend adding the following config keys in the `[artifactory]` section in `lib/lp/services/config/schema-lazr.conf`:

 * `distributions` which is a space-separated list of distributions for which we'll dispatch Artifactory repository URLs, and which can be set to `soss` on production
 * `primary_repositories`, which is a space-separated list of repository names under the Artifactory base URL to dispatch by default, and which can be set to `soss-deb-stable soss-docker-stable soss-generic-stable soss-oci-stable soss-python-stable` (or similar) on production

This should be enough to get the common case working, which is urgent: `CIBuildBehaviour` should be able to use this configuration to construct a list of URLs including credentials, and dispatch them only to builds in the correct distribution.  We'll certainly need to work out a way for builds to specify additional repositories, but that's less urgent and we can work it out later.  We need to get the common case working this week if at all possible.

I'm making one significant assumption here, which is that it's OK to send URLs of Artifactory repositories for all package types to all builds, even though (for example) `soss-generic-stable` won't have PyPI-compatible indexes.  The reason for this design is that we can assume that there'll be a certain amount of overlap (e.g. Python wheel builds may want to install some dependencies via apt, which should come from `soss-deb-stable` for preference), and I don't think the Launchpad side should be in the business of trying to work out which ones are needed.  My suggestion here is that lpcraft should probe which URLs are usable for a given index type: for example it can make a request to `$URL/dists/$SERIES/InRelease` for an appropriate series to figure out whether it's usable by apt.

I definitely suggest making sure that both the Launchpad and lpcraft sides of this protocol work before landing the Launchpad side (we'll also need code in the launchpad-buildd side, but that will just be glue code and unit tests will do for that).  Please talk to me ASAP if you have any questions about this.  I realize it's a bit hacky as designs go, but it should get things working for the security team, and configuration at the lazr.config level is fairly easy to undo or change later.
-- 
https://code.launchpad.net/~jugmac00/launchpad/+git/launchpad/+merge/420028
Your team Launchpad code reviewers is requested to review the proposed merge of ~jugmac00/launchpad:pass-artifactory-credentials into launchpad:master.