ubuntu-docker-images team mailing list archive

[Sergio Durigan Junior <sergio.durigan@xxxxxxxxxxxxx>]

Hi,

Now that we have an all-green OCI test matrix, it's time to discuss the
frequency with which we want to auto-trigger the tests.

A few things worth considering:

1) We don't have an automatic way to know when an image is updated.  We
could keep track of the digests that were previously tested and compare
them with the digests currently available in a registry, and then skip
the test if they are the same.  I'm sure there are implications to this
that I'm not foreseeing right now.

2) We don't have unlimited resources, of course.  I *think* that running
all tests once a day would be fine, but I don't know for sure.  We don't
want our tests to affect other CI projects.  Also, see (4).

3) Dockerhub/AWS have rate-limiting in place even when we authenticate
before running the tests.  I *think* that running everything daily
should be fine, but I don't know for sure.  Also, see (4).

4) Ideally, we have to take into account the fact that we plan to expand
the number of images during the upcoming cycles, which means that we
will eventually have dozens of images to test.  Nowadays, an LTS image
requires 8 unit test runs (one for each arch * one for each namespace) +
1 standalone run, whereas a non-LTS image requires 4 unit test runs (one
for each arch) + 1 standalone run.  If we consider that the long term
plan is to reach 60 images, you can see how problematic it will be.


Maybe the best mid/long-term plan would be to work on tooling that would
allow us to detect when an image is updated and decide whether or not we
should run the tests for it.  I mean, we already have this knowledge
coded into some of our scripts; it's just a matter of putting something
together now.

For now, given that we won't be working on these images every day, we
can have the tests running once or twice a week in order to save
resources.  If we see that it's not possible to schedule all of them at
once (due to rate-limiting, for example), we can schedule the unit tests
one day and the standalone tests the next.

WDYT?

-- 
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0  EB2F 106D A1C8 C3CB BF14