yahoo-eng-team team mailing list archive

[Mark McClain <1177654@xxxxxxxxxxxxxxxxxx>]

** Changed in: neutron
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1177654

Title:
  quantum agent-list status different across nodes.

Status in OpenStack Neutron (virtual network service):
  Invalid

Bug description:
  Hi,

  This issue has only shown up since I add the other 3 nodes, a single
  node install doesn't seem to have the issues below.

  I have 4 x Quantum Network nodes vcr01 vcr02 vcr03 vcr04.

  I have OVS, DHCP agents installed on each. I have also copied the
  quantum.conf etc files so they are identical on each node.

  on vcr01 I run 
  quantum agent-list
  +--------------------------------------+--------------------+-------------------------------+-------+----------------+
  | id                                   | agent_type         | host                          | alive | admin_state_up |
  +--------------------------------------+--------------------+-------------------------------+-------+----------------+
  | 0630d2b3-7696-4b3d-9203-f7e162426707 | Open vSwitch agent | vcr01.bne01.cloud.labs.local  | xxx   | True           |
  | 37f76ec5-2581-4e6d-9df7-cd7c7302af2a | Open vSwitch agent | vcr04.bne01.cloud.labs.local  | xxx   | True           |
  | 3a1e22da-d4ba-408f-8e84-0698b9d26c43 | DHCP agent         | vcr02.bne01.cloud.labs.local  | xxx   | True           |
  | 458401b2-821d-45b1-be91-b64222ba32d0 | Open vSwitch agent | vcr03.bne01.cloud.labs.local  | xxx   | True           |
  | 7ce64466-619c-462f-9dc6-71ebaed4bf4f | DHCP agent         | vcr04.bne01.cloud.labs.local  | xxx   | True           |
  | 8e18207f-cd13-4044-9e00-293f06ce9ac1 | DHCP agent         | vcr01.bne01.cloud.labs.local  | :-)   | True           |
  | c18efffd-ecb5-4866-983a-d3ee4e9e115d | Open vSwitch agent | vcr02.bne01.cloud.labs.local  | :-)   | True           |
  | d5dd0afa-b3ef-430a-892a-a3093b5b0a02 | DHCP agent         | vcr03.bne01.cloud.labs.local  | xxx   | True           |
  | f9f97da8-b905-4400-b920-7c7883b25fd6 | Open vSwitch agent | pvmh01.bne01.cloud.labs.local | xxx   | True           |
  +--------------------------------------+--------------------+-------------------------------+-------+----------------+

  on vcr02 i run 
  quantum agent-list

  +--------------------------------------+--------------------+-------------------------------+-------+----------------+
  | id                                   | agent_type         | host                          | alive | admin_state_up |
  +--------------------------------------+--------------------+-------------------------------+-------+----------------+
  | 0630d2b3-7696-4b3d-9203-f7e162426707 | Open vSwitch agent | vcr01.bne01.cloud.labs.local  | :-)   | True           |
  | 37f76ec5-2581-4e6d-9df7-cd7c7302af2a | Open vSwitch agent | vcr04.bne01.cloud.labs.local  | :-)   | True           |
  | 3a1e22da-d4ba-408f-8e84-0698b9d26c43 | DHCP agent         | vcr02.bne01.cloud.labs.local  | :-)   | True           |
  | 458401b2-821d-45b1-be91-b64222ba32d0 | Open vSwitch agent | vcr03.bne01.cloud.labs.local  | :-)   | True           |
  | 7ce64466-619c-462f-9dc6-71ebaed4bf4f | DHCP agent         | vcr04.bne01.cloud.labs.local  | :-)   | True           |
  | 8e18207f-cd13-4044-9e00-293f06ce9ac1 | DHCP agent         | vcr01.bne01.cloud.labs.local  | :-)   | True           |
  | c18efffd-ecb5-4866-983a-d3ee4e9e115d | Open vSwitch agent | vcr02.bne01.cloud.labs.local  | :-)   | True           |
  | d5dd0afa-b3ef-430a-892a-a3093b5b0a02 | DHCP agent         | vcr03.bne01.cloud.labs.local  | :-)   | True           |
  | f9f97da8-b905-4400-b920-7c7883b25fd6 | Open vSwitch agent | pvmh01.bne01.cloud.labs.local | :-)   | True           |
  +--------------------------------------+--------------------+-------------------------------+-------+----------------+

  on vcr03 i get the same result as vcr02

  
  on vcr04 i get a similar result as to vcr01

  Strangely enough the agent-list result pretty much changes inconstantly on vcr01 & vcr04. 
  ie: the xxx and :-) status changes allot. as per below.

  
  http://pastie.org/pastes/7816343/text

  
  log files seem ok i don't see any "errors" my eyes might be painted on.

  
  I tried turning off the vm's dumping the quantum database and recreating it powed on the vm's and they all came up as :-) however after a reboot of one of the nodes the behavior as above returned.

  I have keystone setup as UUID as well.

  
  quantum.conf http://pastie.org/pastes/7816357/text

  api-paste.ini
  http://pastie.org/pastes/7816369/text?key=shmkvpqhvkq6dnvpneajpq

  dhcp_agent.ini
  http://pastie.org/pastes/7816377/text?key=lfiqbqwuaxvnjiiqowdmq

  plugins/openvswitch/ovs_quantum_plugin.ini
  http://pastie.org/pastes/7816381/text?key=mk1gq4c3xqymeud49zgw

  -------------
  Other thoughts 

  might it be rabbitmq? i'm using mirrored queues with the rabbitmq
  repo.

  rabbitmqctl delete_user guest
  rabbitmqctl add_user openstack password
  rabbitmqctl set_permissions -p / openstack ".*" ".*" ".*"

  rabbitmqctl set_policy HA '^(?!amq\.).*' '{"ha-mode": "all"}'

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1177654/+subscriptions