sts-sponsors team mailing list archive

[Matthew Ruffell <1874075@xxxxxxxxxxxxxxxxxx>]

Performing verification for rabbitmq-server in -proposed for xenial.

I installed 3.5.7-1ubuntu0.16.04.4 from -proposed, and followed the
instructions I wrote in comment #59.

I sent a message to host 1, checked that host 1 was master, and host 2
was the slave. I shut host 1 down, and then sent a message to host 2.
Host 2 became the new master. I shut host 2 down. I then started host 1
up again.

The rabbit service came up in a "activating" state, due to
ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait kicking in. The
rabbitmq in xenial does not offer verbose logging in its
/var/log/rabbitmq/rabbit.log file about waiting for host 2, so instead I
focused on the systemd service. The service will time out after 60
seconds, something I noted and explained previously, which myself and
@ddstreet find to be an acceptable timeout. When the timeout occurs,
/usr/lib/rabbitmq/bin/rabbitmq-server-wait exits with an error code,
which causes systemd to restart the service, as expected. After a 10
second wait, the service starts in the activating state, waiting for
host 2 to come up. I left the service for 15 minutes, and it timed out
and restarted successfully 15 times.

I then started host 2 up, and host 1 rejoined the cluster. Both hosts
have the rabbit service in the active state.

The package is behaving as expected on xenial, and I am happy to mark
this as verified.

** Tags removed: verification-needed verification-needed-xenial
** Tags added: verification-done-xenial

-- 
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1874075

Title:
  rabbitmq-server startup timeouts differ between SysV and systemd

Status in rabbitmq-server package in Ubuntu:
  Fix Released
Status in rabbitmq-server source package in Xenial:
  Fix Committed
Status in rabbitmq-server source package in Bionic:
  Fix Committed
Status in rabbitmq-server source package in Eoan:
  Won't Fix
Status in rabbitmq-server source package in Focal:
  Fix Released
Status in rabbitmq-server source package in Groovy:
  Fix Released
Status in rabbitmq-server package in Debian:
  New

Bug description:
  The startup timeouts were recently adjusted and synchronized between
  the SysV and systemd startup files.

  https://github.com/rabbitmq/rabbitmq-server-release/pull/129

  The new startup files should be included in this package.

  [Impact]

  After starting the RabbitMQ server process, the startup script will
  wait for the server to start by calling `rabbitmqctl wait` and will
  time out after 10 s.

  The startup time of the server depends on how quickly the Mnesia
  database becomes available and the server will time out after
  `mnesia_table_loading_retry_timeout` ms times
  `mnesia_table_loading_retry_limit` retries. By default this wait is
  30,000 ms times 10 retries, i.e. 300 s.

  The mismatch between these two timeout values might lead to the
  startup script failing prematurely while the server is still waiting
  for the Mnesia tables.

  This change introduces variable `RABBITMQ_STARTUP_TIMEOUT` and the
  `--timeout` option into the startup script. The default value for this
  timeout is set to 10 minutes (600 seconds).

  This change also updates the systemd service file to match the timeout
  values between the two service management methods.

  [Scope]

  Upstream patch: https://github.com/rabbitmq/rabbitmq-server-
  release/pull/129

  * Fix is not included in the Debian package
  * Fix is not included in any Ubuntu series

  * Groovy and Focal can apply the upstream patch as is
  * Bionic and Xenial need an additional fix in the systemd service file
    to set the `RABBITMQ_STARTUP_TIMEOUT` variable for the
    `rabbitmq-server-wait` helper script.

  [Test Case]

  In a clustered setup with two nodes, A and B.

  1. create queue on A
  2. shut down B
  3. shut down A
  4. boot B

  The broker on B will wait for A. The systemd service will wait for 10
  seconds and then fail. Boot A and the rabbitmq-server process on B
  will complete startup.

  [Regression Potential]

  This change alters the behavior of the startup scripts when the Mnesia
  database takes long to become available. This might lead to failures
  further down the service dependency chain.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1874075/+subscriptions