canonical-ubuntu-qa team mailing list archive

[Skia <mp+466127@xxxxxxxxxxxxxxxxxx>]

Skia has proposed merging ~hyask/autopkgtest-cloud:skia/doc_rabbitmq_web_access into autopkgtest-cloud:master.

Requested reviews:
  Canonical's Ubuntu QA (canonical-ubuntu-qa)

For more details, see:
https://code.launchpad.net/~hyask/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/466127

Quick documentation update.
-- 
Your team Canonical's Ubuntu QA is requested to review the proposed merge of ~hyask/autopkgtest-cloud:skia/doc_rabbitmq_web_access into autopkgtest-cloud:master.

diff --git a/docs/administration.rst b/docs/administration.rst
index 4673fd1..e899b03 100644
--- a/docs/administration.rst
+++ b/docs/administration.rst
@@ -131,7 +131,7 @@ On the cloud/lxd controller, run::
 Watching one cloud/arch
 ^^^^^^^^^^^^^^^^^^^^^^^
 
-.. code-block::
+.. code-block:: bash
 
  journalctl -u autopkgtest@<cloud>-<arch>-*.service
 
@@ -262,7 +262,7 @@ for an example, search for ``TEST_UPSTREAM``.
 
 However you want to structure your test, ensure that it works locally with a command like
 
-.. code-block::
+.. code-block:: bash
 
  autopkgtest --apt-upgrade https://coolcode.projects.org/foo.git \
     --env UPSTREAM_PULL_REQUEST=1234 --env TEST_UPSTREAM=1 -- \
@@ -341,7 +341,7 @@ On the autopkgtest side on the controller:
 3. Run ``mojo run`` to deploy the updated configuration, as normal.
 4. Verify that the files got updated on the servers:
 
-.. code-block::
+.. code-block:: bash
 
   juju run --application autopkgtest-web 'cat ~ubuntu/github-status-credentials.txt; cat ~ubuntu/github-secrets.json'
 
@@ -355,7 +355,8 @@ on multiple architectures or multiple Ubuntu releases.
 
 
 Queue Cleanup
-^^^^^^^^^^^^^
+-------------
+
 Regular queue cleanup can become necessary when the queues are quite large.
 The best way to go about doing this is by first downloading the queues.json:
 
@@ -374,14 +375,14 @@ obsoleted package versions, e.g.:
 Here, you can see that ``dpkg/1.22.6ubuntu2`` has been obsoleted by ``dpkg/1.22.6ubuntu4``.
 So, the workflow now would be to remove said package from the queue::
 
-.. code-block::
+.. code-block:: bash
 
   ./filter-amqp -v debci-huge-noble-$arch "dpkg/1.22.6ubuntu2\b"
 
 However, this gets tedious with lots of obsoleted packages in the queue. So an approach,
 when you have lots of obsoleted packages, would be like so:
 
-.. code-block::
+.. code-block:: bash
 
   packages="package1/trigger-2.3.0 package2/trigger-2.4.3..." # obviously with more packages
   for pkg in $packages; do for arch in amd64 arm64 s390x ppc64el armhf i386; do ./filter-amqp -v debci-huge-noble-$arch "$pkg\b"; done; done
@@ -389,7 +390,7 @@ when you have lots of obsoleted packages, would be like so:
 This way you can remove all the packages in one command on every architecture.
 
 Resizing /tmp partitions
-^^^^^^^^^^^^^^^^^^^^^^^^
+------------------------
 
 When running an instance of autopkgtest-cloud, you may find that the `/tmp` partitions for the
 autopkgtest-cloud-worker units can get quite full.
@@ -424,7 +425,7 @@ The steps to increase the `/tmp` volume size are detailed below.
 Before doing any of the steps detailed in this section, it's important to make sure no tests
 are currently running on the cloud worker with the partition you want to resize.
 
-.. code-block::
+.. code-block:: bash
 
   # on the worker machine with the volume you intend to resize
   chmod -x autopkgtest-cloud/worker/worker
@@ -433,7 +434,7 @@ are currently running on the cloud worker with the partition you want to resize.
 
 First check that this specific version of openstack is available via:
 
-.. code-block::
+.. code-block:: bash
 
   openstack --os-volume-api-version 3.42 volume list
 
@@ -441,7 +442,7 @@ The command should not fail.
 
 To resize a volume:
 
-.. code-block::
+.. code-block:: bash
 
   # get the 'openstack' volume id
   juju storage --volume # the volume id is in the "Provider ID" column
@@ -473,14 +474,14 @@ openstack server. Keep this in mind if you're using the __DEFAULT__ volume type
 
 
 Killing running tests
-^^^^^^^^^^^^^^^^^^^^^
+---------------------
 
 In order to kill a currently running test, grab the test uuid. This can be seen in
 `running.json` or on the `/running` page.
 
 `ssh` to a worker unit, and run:
 
-.. code-block::
+.. code-block:: bash
 
   ps aux | grep runner | grep $uuid
   # grab the PID from the process
@@ -489,3 +490,26 @@ In order to kill a currently running test, grab the test uuid. This can be seen 
 This will kill the autopkgtest process, and then the worker will `ack` the test request
 message, causing the test to not be re-queued, and then the worker will also ensure
 that the openstack server used for the test is deleted.
+
+Access the RabbitMQ web UI
+--------------------------
+
+RabbitMQ by default provides a web UI that is very convenient to look at various
+metrics gathered by the process. This can help diagnose issues such as queues
+not being cleaned correctly, and growing indefinitely.
+
+Since it's not exposed on a publicly reached port in any way, you need to setup
+SSH tunnelling to access it.
+
+.. code-block:: bash
+
+  # First setup a tunnel to the bastion (disable `byobu` to avoid indefinitely running tunnels)
+  LC_BYOBU=0 ssh ubuntu-qa-bastion-ps5.internal -L 15672:localhost:15672
+  # Then, from the right environment on the bastion, setup a second tunnel to the RabbitMQ unit
+  ssh -i ~/.ssh/id_ed25519 -L 15672:localhost:15672 ubuntu@10.136.6.239
+
+Now point your local browser to http://localhost:15672 and you should have
+access to the UI. Grab the credentials by asking a team member.
+
+**NOTE**: Don't forget to close the tunnels when you're done, especially if you
+usually have a `tmux`/`byobu` session running wrapping the second tunnel!
diff --git a/docs/lxd.rst b/docs/lxd.rst
index 665c84c..a4b397b 100644
--- a/docs/lxd.rst
+++ b/docs/lxd.rst
@@ -25,7 +25,7 @@ Managing cluster nodes
 
 To deploy a new node, after having sourced the cloud ``.rc`` file (``source ~/.scalingstack/bos03-arm64.rc``, for example) on the bastion itself:
 
-.. code-block::
+.. code-block:: bash
 
     $ IMAGE=$(openstack image list --format csv -c Name --quote none | grep auto-sync/ubuntu-jammy-daily-arm64-server | tail -n1)
     $ NET_ID=$(openstack network show net_prod-proposed-migration --format json | jq -r '."id"')