canonical-ubuntu-qa team mailing list archive

[Tim Andersson <mp+465424@xxxxxxxxxxxxxxxxxx>]

Tim Andersson has proposed merging ~andersson123/autopkgtest-cloud:worker-dont-remove-queue-item-systemctl-restart into autopkgtest-cloud:master.

Requested reviews:
  Canonical's Ubuntu QA (canonical-ubuntu-qa)
Related bugs:
  Bug #2064582 in Auto Package Testing: "sudo systemctl restart autopkgtest.target loses currently running jobs  with new kill test changes"
  https://bugs.launchpad.net/auto-package-testing/+bug/2064582

For more details, see:
https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/465424
-- 
Your team Canonical's Ubuntu QA is requested to review the proposed merge of ~andersson123/autopkgtest-cloud:worker-dont-remove-queue-item-systemctl-restart into autopkgtest-cloud:master.

diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
index 809f7ad..51a48dc 100755
--- a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
@@ -1332,13 +1332,31 @@ def request(msg):
                     # tests can be requested to be killed with:
                     # kill -15 $pid
                     logging.warning(
-                        "autopkgtest exited with code: %i\nack-ing message: %s",
+                        "autopkgtest exited with code: %i",
                         code,
-                        body.encode(),
                     )
                     running_test = False
                     # we ack the message so it doesn't go back in the queue
-                    msg.channel.basic_ack(msg.delivery_tag)
+                    if code == -15:
+                        logging.info(
+                            "autopkgtest has been killed with SIGTERM, indicating a restart or stop, requeue-ing message %s"
+                            % body
+                        )
+                        msg.channel.basic_reject(
+                            msg.delivery_tag, requeue=True
+                        )
+                    elif code == -9:
+                        logging.info(
+                            "autopkgtest has been killed intentionally, removing message %s from the queue"
+                            % body
+                        )
+                        msg.channel.basic_ack(msg.delivery_tag)
+                    else:
+                        logging.info(
+                            "autopkgtest has hit an obscure failure mode, removing message %s from the queue"
+                            % body
+                        )
+                        msg.channel.basic_ack(msg.delivery_tag)
                     logging.info(
                         "Killing openstack server with uuid %s",
                         test_uuid,
diff --git a/charms/focal/autopkgtest-cloud-worker/units/autopkgtest@.service b/charms/focal/autopkgtest-cloud-worker/units/autopkgtest@.service
index a94f14d..172ecf7 100644
--- a/charms/focal/autopkgtest-cloud-worker/units/autopkgtest@.service
+++ b/charms/focal/autopkgtest-cloud-worker/units/autopkgtest@.service
@@ -33,6 +33,8 @@ RestartSec=5min
 Restart=on-failure
 StartLimitInterval=10m
 StartLimitBurst=3
+KillSignal=SIGTERM
+RestartKillSignal=SIGTERM
 
 [Install]
 WantedBy=autopkgtest.target
diff --git a/docs/administration.rst b/docs/administration.rst
index e899b03..302fcc7 100644
--- a/docs/administration.rst
+++ b/docs/administration.rst
@@ -485,6 +485,9 @@ In order to kill a currently running test, grab the test uuid. This can be seen 
 
   ps aux | grep runner | grep $uuid
   # grab the PID from the process
+  # If you want to kill the test and remove the test request from the queue:
+  kill -9 $pid
+  # If you want to kill the test and preserve the test request in the queue:
   kill -15 $pid
 
 This will kill the autopkgtest process, and then the worker will `ack` the test request