yahoo-eng-team team mailing list archive

[OpenStack Infra <1558721@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/294230
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9d21b5ad7edbf9ac1fd9254e97f56966f25de8e6
Submitter: Jenkins
Branch:    master

commit 9d21b5ad7edbf9ac1fd9254e97f56966f25de8e6
Author: Alex Oughton <alex.oughton@xxxxxxxxxxxxx>
Date:   Fri Mar 18 11:12:10 2016 -0500

    Close XenAPI sessions in neutron-rootwrap-xen-dom0
    
    Neutron with XenServer properly doesn't close XenAPI sessions.
    If it creates these sessions so rapidly, the XenServer host eventually
    exceeds its maximum allowed number of connections.
    This patch adds a close process for session.
    
    Closes-Bug: 1558721
    Change-Id: Ida90a970c649745c492c28c41c4a151e4d940aa6


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1558721

Title:
  neutron-rootwrap-xen-dom0 not properly closing XenAPI sessions

Status in neutron:
  Fix Released

Bug description:
  Hello,

  When using OpenStack Liberty with XenServer, neutron is not properly
  closing its XenAPI sessions. Since it creates these so rapidly, the
  XenServer host eventually exceeds its maximum allowed number of
  connections:

  Mar 17 11:39:05 compute3 xapi:
  [debug|compute3.openstack.lab.eco.rackspace.com|25 db_gc|DB GC
  D:bb694b976766|db_gc] Number of disposable sessions in group
  'external' in database (401/401) exceeds limit (400): will delete the
  oldest

  This occurs roughly once per minute, with many sessions being
  invalidated. The effect is that any long-running hypervisor operations
  (for example a live-migration) will fail with an "unauthorized" error,
  as their session was invalidated while they were still running:

  2016-03-17 11:43:34.483 14310 ERROR nova.virt.xenapi.vmops Failure: ['INTERNAL_ERROR', 'Storage_interface.Internal_error("Http_client.Http_error(\\"401\\", \\"{ frame = false; method = POST; uri = /services/SM;
  query = [ session_id=OpaqueRef:8663a5b7-928e-6ef5-e312-9f430b553c7f ]; content_length = [  ]; transfer encoding = ; version = 1.0; cookie = [  ]; task = ; subtask_of = ; content-type = ; host = ; user_agent = xe
  n-api-libs/1.0 }\\")")']

  The fix is to add a line to neutron-rootwrap-xen-dom0 to have it
  properly close the sessions.

  Before:

  def run_command(url, username, password, user_args, cmd_input):
      try:
          session = XenAPI.Session(url)
          session.login_with_password(username, password)
          host = session.xenapi.session.get_this_host(session.handle)
          result = session.xenapi.host.call_plugin(
              host, 'netwrap', 'run_command',
              {'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})
          return json.loads(result)
      except Exception as e:
          traceback.print_exc()
          sys.exit(RC_XENAPI_ERROR)

  After:

  def run_command(url, username, password, user_args, cmd_input):
      try:
          session = XenAPI.Session(url)
          session.login_with_password(username, password)
          host = session.xenapi.session.get_this_host(session.handle)
          result = session.xenapi.host.call_plugin(
              host, 'netwrap', 'run_command',
              {'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})
          session.xenapi.session.logout()
          return json.loads(result)
      except Exception as e:
          traceback.print_exc()
          sys.exit(RC_XENAPI_ERROR)

  
  After making this change, the logs still show the sessions being rapidly created, but it also shows them being destroyed. The "exceeds limit" error no longer occurs, and live-migrations now succeed.

  
  Regards,

  Alex Oughton

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1558721/+subscriptions