yahoo-eng-team team mailing list archive

[Adam Gandelman <1172922@xxxxxxxxxxxxxxxxxx>]

** Changed in: quantum/grizzly
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to quantum.
https://bugs.launchpad.net/bugs/1172922

Title:
  Nova and quantum don't close qpid connections after certain error
  conditions

Status in OpenStack Compute (Nova):
  Triaged
Status in Oslo - a Library of Common OpenStack Code:
  Fix Released
Status in OpenStack Quantum (virtual network service):
  Fix Released
Status in quantum grizzly series:
  Fix Released

Bug description:
  In conditions where nova gets a connection to qpid but there is a
  problem at the application layer (such as a bad qpid password is
  specified), nova will continuously retry the connection without
  closing previous connections.

  Steps to reproduce:
  - Preconditions: At least 1 nova compute node using qpid for messaging, with authentication turned on.
  - Specify a wrong qpid password in nova.conf on the compute node
  - nova will continuously retry with the wrong password and print errors such as this to the compute log:
  2013-04-25 16:37:52.269 ERROR nova.openstack.common.rpc.impl_qpid [req-1d15f33c-5b2d-4ee1-aaa1-ab0140a56608 None None] Unable to connect to AMQP server: connection-forced: Authentication failed(320). Sleeping 60 seconds
  - Each time nova retries the connection, it will create another connection to qpid and not close the previous connections.

  
  I think this exception case needs to be updated in https://github.com/openstack/nova/blob/master/nova/openstack/common/rpc/impl_qpid.py to close the connection:

              except qpid_exceptions.ConnectionError, e:
                  msg_dict = dict(e=e, delay=delay)
                  msg = _("Unable to connect to AMQP server: %(e)s. "
                          "Sleeping %(delay)s seconds") % msg_dict
                  LOG.error(msg)
                  time.sleep(delay)
                  delay = min(2 * delay, 60)

  Note that quantum uses the same code for qpid.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1172922/+subscriptions