On Mon, Jun 14, 2021 at 04:16:14PM -0400, Sergio Durigan Junior wrote:
Hi,
Now that we have an all-green OCI test matrix, it's time to discuss the
frequency with which we want to auto-trigger the tests.
A few things worth considering:
1) We don't have an automatic way to know when an image is updated. We
could keep track of the digests that were previously tested and compare
them with the digests currently available in a registry, and then skip
the test if they are the same. I'm sure there are implications to this
that I'm not foreseeing right now.
2) We don't have unlimited resources, of course. I *think* that running
all tests once a day would be fine, but I don't know for sure. We don't
want our tests to affect other CI projects. Also, see (4).
3) Dockerhub/AWS have rate-limiting in place even when we authenticate
before running the tests. I *think* that running everything daily
should be fine, but I don't know for sure. Also, see (4).
4) Ideally, we have to take into account the fact that we plan to expand
the number of images during the upcoming cycles, which means that we
will eventually have dozens of images to test. Nowadays, an LTS image
requires 8 unit test runs (one for each arch * one for each namespace) +
1 standalone run, whereas a non-LTS image requires 4 unit test runs (one
for each arch) + 1 standalone run. If we consider that the long term
plan is to reach 60 images, you can see how problematic it will be.
Maybe the best mid/long-term plan would be to work on tooling that would
allow us to detect when an image is updated and decide whether or not we
should run the tests for it. I mean, we already have this knowledge
coded into some of our scripts; it's just a matter of putting something
together now.
For now, given that we won't be working on these images every day, we
can have the tests running once or twice a week in order to save
resources. If we see that it's not possible to schedule all of them at
once (due to rate-limiting, for example), we can schedule the unit tests
one day and the standalone tests the next.
WDYT?
Given that the software in the images should in theory change super
infrequently, then a longer interval between test auto-runs makes
sense. If a given image really only changes no more than once a month,
then having 30 days of test runes isn't going to add a lot of value.
So I'd look at even maybe just once a week.
As long as there's a manual way to trigger tests to run, e.g. at upload
time or after a new base image change has been released, then can just
include a retrigger step as part of the upload+review process. That'd
be the ideal context to deal with test failures, and my guess is going
to detect 95% of the actual problems the oci images would hit in
practice.
