yahoo-eng-team team mailing list archive

[Alex Oughton <alex.oughton@xxxxxxxxxxxxx>]

Public bug reported:

Hello,

When using OpenStack Liberty with XenServer, neutron is not properly
closing its XenAPI sessions. Since it creates these so rapidly, the
XenServer host eventually exceeds its maximum allowed number of
connections:

Mar 17 11:39:05 compute3 xapi:
[debug|compute3.openstack.lab.eco.rackspace.com|25 db_gc|DB GC
D:bb694b976766|db_gc] Number of disposable sessions in group 'external'
in database (401/401) exceeds limit (400): will delete the oldest

This occurs roughly once per minute, with many sessions being
invalidated. The effect is that any long-running hypervisor operations
(for example a live-migration) will fail with an "unauthorized" error,
as their session was invalidated while they were still running:

2016-03-17 11:43:34.483 14310 ERROR nova.virt.xenapi.vmops Failure: ['INTERNAL_ERROR', 'Storage_interface.Internal_error("Http_client.Http_error(\\"401\\", \\"{ frame = false; method = POST; uri = /services/SM;
query = [ session_id=OpaqueRef:8663a5b7-928e-6ef5-e312-9f430b553c7f ]; content_length = [  ]; transfer encoding = ; version = 1.0; cookie = [  ]; task = ; subtask_of = ; content-type = ; host = ; user_agent = xe
n-api-libs/1.0 }\\")")']

The fix is to add a line to neutron-rootwrap-xen-dom0 to have it
properly close the sessions.

Before:

def run_command(url, username, password, user_args, cmd_input):
    try:
        session = XenAPI.Session(url)
        session.login_with_password(username, password)
        host = session.xenapi.session.get_this_host(session.handle)
        result = session.xenapi.host.call_plugin(
            host, 'netwrap', 'run_command',
            {'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})
        return json.loads(result)
    except Exception as e:
        traceback.print_exc()
        sys.exit(RC_XENAPI_ERROR)

After:

def run_command(url, username, password, user_args, cmd_input):
    try:
        session = XenAPI.Session(url)
        session.login_with_password(username, password)
        host = session.xenapi.session.get_this_host(session.handle)
        result = session.xenapi.host.call_plugin(
            host, 'netwrap', 'run_command',
            {'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})
        session.xenapi.session.logout()
        return json.loads(result)
    except Exception as e:
        traceback.print_exc()
        sys.exit(RC_XENAPI_ERROR)


After making this change, the logs still show the sessions being rapidly created, but it also shows them being destroyed. The "exceeds limit" error no longer occurs, and live-migrations now succeed.


Regards,

Alex Oughton

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1558721

Title:
  neutron-rootwrap-xen-dom0 not properly closing XenAPI sessions

Status in neutron:
  New

Bug description:
  Hello,

  When using OpenStack Liberty with XenServer, neutron is not properly
  closing its XenAPI sessions. Since it creates these so rapidly, the
  XenServer host eventually exceeds its maximum allowed number of
  connections:

  Mar 17 11:39:05 compute3 xapi:
  [debug|compute3.openstack.lab.eco.rackspace.com|25 db_gc|DB GC
  D:bb694b976766|db_gc] Number of disposable sessions in group
  'external' in database (401/401) exceeds limit (400): will delete the
  oldest

  This occurs roughly once per minute, with many sessions being
  invalidated. The effect is that any long-running hypervisor operations
  (for example a live-migration) will fail with an "unauthorized" error,
  as their session was invalidated while they were still running:

  2016-03-17 11:43:34.483 14310 ERROR nova.virt.xenapi.vmops Failure: ['INTERNAL_ERROR', 'Storage_interface.Internal_error("Http_client.Http_error(\\"401\\", \\"{ frame = false; method = POST; uri = /services/SM;
  query = [ session_id=OpaqueRef:8663a5b7-928e-6ef5-e312-9f430b553c7f ]; content_length = [  ]; transfer encoding = ; version = 1.0; cookie = [  ]; task = ; subtask_of = ; content-type = ; host = ; user_agent = xe
  n-api-libs/1.0 }\\")")']

  The fix is to add a line to neutron-rootwrap-xen-dom0 to have it
  properly close the sessions.

  Before:

  def run_command(url, username, password, user_args, cmd_input):
      try:
          session = XenAPI.Session(url)
          session.login_with_password(username, password)
          host = session.xenapi.session.get_this_host(session.handle)
          result = session.xenapi.host.call_plugin(
              host, 'netwrap', 'run_command',
              {'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})
          return json.loads(result)
      except Exception as e:
          traceback.print_exc()
          sys.exit(RC_XENAPI_ERROR)

  After:

  def run_command(url, username, password, user_args, cmd_input):
      try:
          session = XenAPI.Session(url)
          session.login_with_password(username, password)
          host = session.xenapi.session.get_this_host(session.handle)
          result = session.xenapi.host.call_plugin(
              host, 'netwrap', 'run_command',
              {'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})
          session.xenapi.session.logout()
          return json.loads(result)
      except Exception as e:
          traceback.print_exc()
          sys.exit(RC_XENAPI_ERROR)

  
  After making this change, the logs still show the sessions being rapidly created, but it also shows them being destroyed. The "exceeds limit" error no longer occurs, and live-migrations now succeed.

  
  Regards,

  Alex Oughton

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1558721/+subscriptions