yahoo-eng-team team mailing list archive

[Will Foster <wfoster@xxxxxxxxxx>]

Public bug reported:

This is related to an upgrade from Havana -> Icehouse.

It seems that nova-compute is passing the wrong protocol header to rabbitmq 
which results in nova compute services never starting up, this is despite the fact that the nova.conf
driver for messaging is:

   rpc_backend=nova.openstack.common.rpc.impl_kombu

nova-compute logs the following and never starts up:

   2014-11-26 14:15:12.817 8585 ERROR oslo.messaging._drivers.impl_qpid
   [-] Unable to connect to AMQP server: client: 0-10, server: 9-1.
   Sleeping 5 seconds

This implies that the wrong driver is being used (hence the protocol
mismatch causing the connection termination?)

We see that the client (compute) wants to send protocol id major 1, protocol
id minor 1, version major 0, version minor 10, which rabbitmq doesn't
like (controller).

=ERROR REPORT==== 26-Nov-2014::01:32:33 ===
closing AMQP connection <0.2090.0> (192.168.0.7:50588 ->
192.168.0.222:5672)
{bad_version,{1,1,0,10}}

The configs are clean, and not telling it to use qpid so we believe
there may be some protocol field buried in the DB somewhere that it's
referencing still in place from Havana/QPID.

We believe that somewhere in the DB it's referencing some kind of protocol field
and it's being used regardless of what is in the config files or the driver it's using.

Of note, it also references the old controller IP address which is not
in the configs so we've brought up a virtual interface with that old IP
(192.168.0.222) to get past it for now.

Attached is a TCPDUMP of the activity.
We have been able to run sucessful db_sync for nova, neutron, keystone, glance, cinder
during the upgrade however this is blocking nova from starting.

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "TCPDUMP of AMQP traffic"
   https://bugs.launchpad.net/bugs/1396689/+attachment/4269099/+files/odd_qpid_to_rabbit.bin

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1396689

Title:
  Nova ignores configs, uses wrong driver/protocol for AMQP from Havana
  -> Icehouse DB upgrade

Status in OpenStack Compute (Nova):
  New

Bug description:
  This is related to an upgrade from Havana -> Icehouse.

  It seems that nova-compute is passing the wrong protocol header to rabbitmq 
  which results in nova compute services never starting up, this is despite the fact that the nova.conf
  driver for messaging is:

     rpc_backend=nova.openstack.common.rpc.impl_kombu

  nova-compute logs the following and never starts up:

     2014-11-26 14:15:12.817 8585 ERROR oslo.messaging._drivers.impl_qpid
     [-] Unable to connect to AMQP server: client: 0-10, server: 9-1.
     Sleeping 5 seconds

  This implies that the wrong driver is being used (hence the protocol
  mismatch causing the connection termination?)

  We see that the client (compute) wants to send protocol id major 1, protocol
  id minor 1, version major 0, version minor 10, which rabbitmq doesn't
  like (controller).

  =ERROR REPORT==== 26-Nov-2014::01:32:33 ===
  closing AMQP connection <0.2090.0> (192.168.0.7:50588 ->
  192.168.0.222:5672)
  {bad_version,{1,1,0,10}}

  The configs are clean, and not telling it to use qpid so we believe
  there may be some protocol field buried in the DB somewhere that it's
  referencing still in place from Havana/QPID.

  We believe that somewhere in the DB it's referencing some kind of protocol field
  and it's being used regardless of what is in the config files or the driver it's using.

  Of note, it also references the old controller IP address which is not
  in the configs so we've brought up a virtual interface with that old IP
  (192.168.0.222) to get past it for now.

  Attached is a TCPDUMP of the activity.
  We have been able to run sucessful db_sync for nova, neutron, keystone, glance, cinder
  during the upgrade however this is blocking nova from starting.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1396689/+subscriptions