launchpad-reviewers team mailing list archive

[Colin Watson <mp+413081@xxxxxxxxxxxxxxxxxx>]

Re-running single jobs in GitLab CI works just fine though - I used that myself only the other day. :-)

The Python bits were just an example of environment differences we might want to test, so don't necessarily get too hung up on the precise details of my example.  I don't know whether we'll specifically want to support multiple versions of Python on focal for the commercial project at hand here, although it would certainly be useful for general testing of Python libraries.  However, a large part of the point of the work we're doing for the commercial project in question is extending our data model to be able to support multiple versions of packages within a single Ubuntu series (see e.g. https://docs.google.com/document/d/1D4tPQMe1Lk5_5t_TJT7J_c7fdHKLoAP4jU_3lsDK4ew/edit#heading=h.l5tgouppxg74), so I'm confident that something analogous to this is going to come up.

Leaving aside multiple versions, I can also easily imagine the security team wanting to run regression tests with a matrix of different environment properties of various kinds (not necessarily environment variables in particular).  For example, the first case I found of this in qa-regression-testing was https://git.launchpad.net/qa-regression-testing/tree/build_testing/apache2/run_framework.sh.  That particular script is no longer relevant because Apache isn't packaged that way any more, but you can still get the general idea: it has a set of tests that it runs for each Apache multi-processing module.  This seems like a natural application for a test matrix.

Whether we have single or multiple containers isn't yet clear, and in general I think it depends on some external factors.  For example, in the specific case of Python 3.8 vs. a backported Python 3.9 on focal, it would probably be possible to use a single job with a single container, as long as any other packages needed don't conflict.  But let's say you wanted to write a test that says that something works both with Firefox as a deb and as a snap, or maybe that something works with either libcurl4-gnutls-dev or libcurl4-openssl-dev (which conflict); it would make sense for those sorts of things to be in a test matrix, but you probably can't run all the tests in the same container.  (These are not current business examples because details are difficult to talk about in public, but just the sorts of things I can easily imagine people wanting to do when regression-testing security updates.)

I haven't worked out how we'd render multiple jobs in a matrix in the UI.  Maybe we'd take what IIRC is the GHA approach of rendering display versions of the bits that differ between entries in the matrix, but I haven't thought through that in detail.

I think it might be a good idea to get you onto some of the wider-scoped project calls here, just so that you can start getting a sense of the business objectives.
-- 
https://code.launchpad.net/~cjwatson/lpcraft/+git/lpcraft/+merge/413081
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/lpcraft:run-one into lpcraft:main.