yahoo-eng-team team mailing list archive

[James Page <james.page@xxxxxxxxxx>]

** Also affects: python-amqp (Ubuntu Trusty)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1472712

Title:
  [SRU] Using SSL with rabbitmq prevents communication between nova-
  compute and conductor after latest nova updates

Status in OpenStack Compute (nova):
  Invalid
Status in oslo.messaging:
  Invalid
Status in python-amqp package in Ubuntu:
  In Progress
Status in python-amqp source package in Trusty:
  New

Bug description:
  [Impact]

  Current oslo.messaging and python-amqp results in repeated connection
  timeouts in the amqp transport layer (SSLError) and thus excessive
  reconnect attempts. This is a known issues that was fixed in python-
  amqp 1.4.4.

  [Test Case]

  Deploy openstack using current Trusty versions + this version of
  python-amqp. Once up and running, check the following:

   - number of rabbitmq connections - with single nova-compute,
  conductor etc I see approx 20 connections whereas previously i saw
  well over 100 and rising.

      sudo rabbitmqctl list_connections

  - check that messages are being consumed from openstack queues

      sudo rabbitmqctl list_queues -p openstack consumers messages name

  - also check e.g. nova-compute and nova-conductor logs and verify that
  the erros menioned below no longer appear

  [Regression Potential]

  None.

  [Other Info]
   
  None.

  ---- ---- ----- ----

  On the latest update of the Ubuntu OpenStack packages, it was
  discovered that the nova-compute/nova-conductor
  (1:2014.1.4-0ubuntu2.1) packages encountered a bug with using SSL to
  connect to rabbitmq.

  When this problem occurs, the compute node cannot connect to the
  controller, and this message is constantly displayed:

  WARNING nova.conductor.api [req-4022395c-9501-47cf-bf8e-476e1cc58772
  None None] Timed out waiting for nova-conductor. Is it running? Or did
  this service start before nova-conductor?

  Investigation revealed that having rabbitmq configured with SSL was
  the root cause of this problem.  This seems to have been introduced
  with the current version of the nova packages.   Rabbitmq was not
  updated as part of this distribution update, but the messaging library
  (python-oslo.messaging 1.3.0-0ubuntu1.1) was updated.   So the problem
  could exist in any of these components.

  Versions installed:
  Openstack version: Icehouse
  Ubuntu 14.04.2 LTS
  nova-conductor    1:2014.1.4-0ubuntu2.1
  nova-compute        1:2014.1.4-0ubuntu2.1
  rabbitmq-server  3.2.4-1
  openssl:amd64/trusty-security   1.0.1f-1ubuntu2.15

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1472712/+subscriptions