ubuntu-docker-images team mailing list archive

[Bryce Harrington <bryce.harrington@xxxxxxxxxxxxx>]

On Mon, Jun 14, 2021 at 04:16:14PM -0400, Sergio Durigan Junior wrote:
> Hi,
> 
> Now that we have an all-green OCI test matrix, it's time to discuss the
> frequency with which we want to auto-trigger the tests.
> 
> A few things worth considering:
> 
> 1) We don't have an automatic way to know when an image is updated.  We
> could keep track of the digests that were previously tested and compare
> them with the digests currently available in a registry, and then skip
> the test if they are the same.  I'm sure there are implications to this
> that I'm not foreseeing right now.
> 
> 2) We don't have unlimited resources, of course.  I *think* that running
> all tests once a day would be fine, but I don't know for sure.  We don't
> want our tests to affect other CI projects.  Also, see (4).
> 
> 3) Dockerhub/AWS have rate-limiting in place even when we authenticate
> before running the tests.  I *think* that running everything daily
> should be fine, but I don't know for sure.  Also, see (4).
> 
> 4) Ideally, we have to take into account the fact that we plan to expand
> the number of images during the upcoming cycles, which means that we
> will eventually have dozens of images to test.  Nowadays, an LTS image
> requires 8 unit test runs (one for each arch * one for each namespace) +
> 1 standalone run, whereas a non-LTS image requires 4 unit test runs (one
> for each arch) + 1 standalone run.  If we consider that the long term
> plan is to reach 60 images, you can see how problematic it will be.
> 
> 
> Maybe the best mid/long-term plan would be to work on tooling that would
> allow us to detect when an image is updated and decide whether or not we
> should run the tests for it.  I mean, we already have this knowledge
> coded into some of our scripts; it's just a matter of putting something
> together now.
> 
> For now, given that we won't be working on these images every day, we
> can have the tests running once or twice a week in order to save
> resources.  If we see that it's not possible to schedule all of them at
> once (due to rate-limiting, for example), we can schedule the unit tests
> one day and the standalone tests the next.
> 
> WDYT?

Given that the software in the images should in theory change super
infrequently, then a longer interval between test auto-runs makes
sense.  If a given image really only changes no more than once a month,
then having 30 days of test runes isn't going to add a lot of value.
So I'd look at even maybe just once a week.

As long as there's a manual way to trigger tests to run, e.g. at upload
time or after a new base image change has been released, then can just
include a retrigger step as part of the upload+review process.  That'd
be the ideal context to deal with test failures, and my guess is going
to detect 95% of the actual problems the oci images would hit in
practice.

Bryce