canonical-ubuntu-qa team mailing list archive

[Brian Murray <mp+456376@xxxxxxxxxxxxxxxxxx>]

This is looking good. One other thing we talked about is that the autopkgtest-cloud-worker units actually pull from the amqp server and inputs push items onto the queue.

We also talked about using debci-$release-$arch as those are well understood variables. It might also be good to add the other types of queues e.g. debci-ppa-$release-$arch, debci-upstream-$release-$arch, debci-huge-$release-$arch.

I've also a few in-line comments.

Diff comments:

> diff --git a/docs/architecture.rst b/docs/architecture.rst
> index 4a31322..2f4c446 100644
> --- a/docs/architecture.rst
> +++ b/docs/architecture.rst
> @@ -11,18 +11,22 @@ Overview
>          subgraph cluster_autopkgtestcloud {
>              subgraph cluster_prodstack {
>                  node [shape=Mrecord]
> -                web [label="{autopkgtest-web web UI|results browser for developers|shows which tests are running|receives requests to run tests from developers or GitHub PRs|can be scaled out}"]
> +                web [label="{autopkgtest-web web UI|* results browser for developers\n* shows which tests are running\n* receives requests to run tests from developers or GitHub PRs\n* can be scaled out}"]
>                  rabbitmq [label="{RabbitMQ AMQP server|debci-focal-amd64|...}"]
> -                cloudworker [label="{cloud worker|call autopkgtest with ssh+nova (openstack) runner}"]
> -                lxdworker [label="{cloud worker (lxd)|call autopkgtest with lxd runner}"]
> +                cloudworker [label="{autopkgtest-cloud-worker|call autopkgtest with ssh+nova (openstack) runner}"]
> +                lxdworker [label="{autopkgtest-cloud-worker (lxd)|call autopkgtest with lxd runner}"]

Let's use the application name here which is autopkgtest-lxd-worker.

>                  swift [label="{OpenStack Swift object store (test results)|autopkgtest-focal|...}"]
> -                haproxyweb [label="{autopkgtest.ubuntu.com haproxy|provides SSL termination|https://autopkgtest.ubuntu.com}";]
> -                haproxylxd [label="{haproxy-armhf-lxd|mediates access to lxd cluster to avoid SPOF}"]
> +                swiftprivate [label="{OpenStack Swift object store|private PPA test results, embargoed CVEs}"]
> +                haproxyweb [label="{HAproxy|* provides SSL termination\n* https://autopkgtest.ubuntu.com}";]
> +                runner [label="{OpenStack Instances|dynamically allocated test runners|amd64, arm64, i386, ppc64el, s390x}"]

Only the amd64 instances (which run i386) tests are actually in prodstack5. All port architectures are in scalingstack at this point in time.

>  
> +                "cloudworker" -> "runner"
>                  "rabbitmq" -> "cloudworker"
>                  "rabbitmq" -> "lxdworker"
>                  "cloudworker" -> "swift"
> +                "cloudworker" -> "swiftprivate"
>                  "lxdworker" -> "swift"
> +                "lxdworker" -> "swiftprivate"
>                  "swift" -> "web" [label="download new results into database"]
>                  "web" -> "swift" [label="link to logs and artifacts" style="dotted"]
>                  "haproxyweb" -> "web" [label="one proxy, many backends"]
> @@ -32,10 +36,8 @@ Overview
>              }
>              subgraph cluster_scalingstack {
>                  node [shape=Mrecord]
> -                runner [label="{dynamically allocated test runners|amd64, arm64, i386, ppc64el, s390x|openstack instances}"]
> -                static [label="{statically provisioned lxd runners|arm64 host|armhf tests|many of these|running lxd in an lxd cluster}"]
> +                static [label="{statically provisioned lxd runners|arm64 host|armhf tests|all of these running lxd in an lxd cluster}"]

I'm pretty sure that these are not in a lxd cluster there is some documentation which indicates the lxd cluster is unused.

>  
> -                "cloudworker" -> "runner"
>                  "lxdworker" -> "static"
>  
>                  label="'scalingstack' workload OpenStack cloud"
> @@ -46,9 +48,19 @@ Overview
>          subgraph cluster_external {
>              node [shape=Mrecord]
>              archive [label="{Ubuntu developers|packages in -proposed}"]
> -            britney [label="{britney|gates pkgs from devel-proposed to release pocket|completed builds|installability|autopkgtests|generates update_excuses.html/update_output.txt}"]
> +            britney [label="{britney|* gates pkgs from -proposed to release pocket\n* completed builds\n* installability\n* autopkgtests\n* generates update_excuses.html/update_output.txt}"]

"non-regressing(?) autopkgtests" something like that would be clarify that not all autopkgtests need to pass.

> +            
>              "archive" -> "britney" [label="Ubuntu developers upload packages"]
>              "britney" -> "rabbitmq" [label="send AMQP requests for tests"]
> +            
> +            archivesecure [label="{Ubuntu security team|packages in ???}"]

??? == PPAs

> +            britneysecure [label="{britney-secure}"]

It's actually referred to as security-britney.

> +            "archivesecure" -> "britneysecure" [label="Ubuntu security team upload packages"]
> +            "britneysecure" -> "rabbitmq" [label="send AMQP requests for tests"]
> +
> +            thirdinput [label="{??? Third input}"]

Ubuntu developers can upload packages to PPA and then request that tests be run using the packages in those PPAs. The tests are actually requested via the web UI and then go to amqp but that seems overly complicated.

A fourth input is packages being sync'ed from Debian to Ubuntu and britney ends up queuing tests for those package syncs.

> +            "thirdinput" -> "rabbitmq" [label="send AMQP requests for tests"]
> +
>  
>              color=green
>              label="external inputs"


-- 
https://code.launchpad.net/~hyask/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/456376
Your team Canonical's Ubuntu QA is requested to review the proposed merge of ~hyask/autopkgtest-cloud:skia/docs_archi_diagram into autopkgtest-cloud:master.