canonical-ubuntu-qa team mailing list archive

[Tim Andersson <mp+468152@xxxxxxxxxxxxxxxxxx>]

Tim Andersson has proposed merging ~andersson123/autopkgtest-cloud:remove-rabbitmq-auto-restart into autopkgtest-cloud:master.

Commit message:
fix: rabbitmq: remove auto-restart for rabbitmq server

For the last few years, rabbitmq was auto-restarting after using up 2GiB
of ram.

This was a longstanding issue, in which the root cause was addressed in
the following commit:
https://git.launchpad.net/autopkgtest-cloud/commit/?id=f019281e0fe38d3f298b933b9fd9fcb243795a7a

The root cause of the issue was the worker code sending status updates
to a queue at a rate which the consumer couldn't keep up with, causing
the rabbitmq queue to grow in size in perpetuity. You can see a more
complete description of the issue in the message of the commit above.

That being said, we can now remove the script that sets up the service
which auto-restarts the rabbitmq-server.service (with glee).

Requested reviews:
  Canonical's Ubuntu QA (canonical-ubuntu-qa)

For more details, see:
https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/468152

This MP removes the script which set up the service on the rabbitmq server which restarted the rabbitmq-server.service when it began to use up more than 2GiB of RAM.
-- 
Your team Canonical's Ubuntu QA is requested to review the proposed merge of ~andersson123/autopkgtest-cloud:remove-rabbitmq-auto-restart into autopkgtest-cloud:master.

diff --git a/mojo/rabbitmq-cleanup-setup b/mojo/rabbitmq-cleanup-setup
deleted file mode 100644
index 8fb4ea9..0000000
--- a/mojo/rabbitmq-cleanup-setup
+++ /dev/null
@@ -1,86 +0,0 @@
-#!/bin/bash
-
-##################################################################
-# Get machine number for rabbitmq-server application
-machine=$(juju status --format=json rabbitmq-server | jq --monochrome-output --raw-output '.applications["rabbitmq-server"].units | map(.["machine"])[]')
-
-##################################################################
-# set file names
-service_name="maybe-restart-rabbitmq"
-rabbitmq_mdir="/home/ubuntu/${service_name}"
-
-##################################################################
-# Create script
-cat > /tmp/"${service_name}" << EOM
-#!/bin/sh
-
-set -e
-
-# Set a limit 2 GiB
-LIMIT=$((2 * 1024 * 1024 * 1024))
-
-if [ $(cat /sys/fs/cgroup/memory/system.slice/rabbitmq-server.service/memory.usage_in_bytes 2>/dev/null || echo 0) -ge $LIMIT ]; then
-	echo "RabbitMQ uses too much memory, restarting"
-	timeout 30 rabbitmq-diagnostics memory_breakdown || true
-	systemctl restart rabbitmq-server.service
-fi
-
-EOM
-
-chmod +x /tmp/"${service_name}"
-
-# Create service file
-cat > /tmp/"${service_name}".service << EOM
-[Unit]
-Description=Check rabbitmq memory use and restart if needed
-
-[Service]
-Type=oneshot
-ExecStart=/usr/local/sbin/maybe-restart-rabbitmq
-EOM
-
-# Create timer file
-cat > /tmp/"${service_name}".timer << EOM
-[Unit]
-Description=Check rabbitmq every 15 mins
-
-[Timer]
-OnBootSec=15min
-OnUnitInactiveSec=15min
-
-[Install]
-WantedBy=multi-user.target
-EOM
-
-##################################################################
-# Create directory for files
-juju exec --machine "${machine}" "mkdir -p ${rabbitmq_mdir}"
-
-
-##################################################################
-# Copy script to machine
-juju scp /tmp/"${service_name}" "${machine}":"${rabbitmq_mdir}/"
-
-# Copy service to machine
-juju scp /tmp/"${service_name}".service "${machine}":"${rabbitmq_mdir}/"
-
-# Copy timer to machine
-juju scp /tmp/"${service_name}".timer "${machine}":"${rabbitmq_mdir}/"
-
-##################################################################
-# Set necessary symbolic links
-
-# Set link for script
-juju exec --machine "${machine}" "sudo ln -sf ${rabbitmq_mdir}/${service_name} /usr/local/sbin/"
-
-# Set link for service
-juju exec --machine "${machine}" "sudo ln -sf ${rabbitmq_mdir}/${service_name}.service /etc/systemd/system/"
-
-# Set link for timer
-juju exec --machine "${machine}" "sudo ln -sf ${rabbitmq_mdir}/${service_name}.timer /etc/systemd/system/"
-
-##################################################################
-# Enable systemd unit
-juju exec --machine "${machine}" "sudo systemctl daemon-reload"
-
-juju exec --machine "${machine}" "sudo systemctl restart ${service_name}.service"