yahoo-eng-team team mailing list archive

[Alex Oughton <alex.oughton@xxxxxxxxxxxxx>]

This was already backported to Kilo:
https://git.openstack.org/cgit/openstack/neutron/commit/?id=2b6d1bba5d4c9e30f8389635c963fdcd3707b409

** Changed in: neutron/kilo
       Status: New => Fix Released

** Changed in: neutron/kilo
     Assignee: (unassigned) => Alex Oughton (alex-oughton)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1558721

Title:
  neutron-rootwrap-xen-dom0 not properly closing XenAPI sessions

Status in neutron:
  Fix Released
Status in neutron kilo series:
  Fix Released

Bug description:
  Hello,

  When using OpenStack Liberty with XenServer, neutron is not properly
  closing its XenAPI sessions. Since it creates these so rapidly, the
  XenServer host eventually exceeds its maximum allowed number of
  connections:

  Mar 17 11:39:05 compute3 xapi:
  [debug|compute3.openstack.lab.eco.rackspace.com|25 db_gc|DB GC
  D:bb694b976766|db_gc] Number of disposable sessions in group
  'external' in database (401/401) exceeds limit (400): will delete the
  oldest

  This occurs roughly once per minute, with many sessions being
  invalidated. The effect is that any long-running hypervisor operations
  (for example a live-migration) will fail with an "unauthorized" error,
  as their session was invalidated while they were still running:

  2016-03-17 11:43:34.483 14310 ERROR nova.virt.xenapi.vmops Failure: ['INTERNAL_ERROR', 'Storage_interface.Internal_error("Http_client.Http_error(\\"401\\", \\"{ frame = false; method = POST; uri = /services/SM;
  query = [ session_id=OpaqueRef:8663a5b7-928e-6ef5-e312-9f430b553c7f ]; content_length = [  ]; transfer encoding = ; version = 1.0; cookie = [  ]; task = ; subtask_of = ; content-type = ; host = ; user_agent = xe
  n-api-libs/1.0 }\\")")']

  The fix is to add a line to neutron-rootwrap-xen-dom0 to have it
  properly close the sessions.

  Before:

  def run_command(url, username, password, user_args, cmd_input):
      try:
          session = XenAPI.Session(url)
          session.login_with_password(username, password)
          host = session.xenapi.session.get_this_host(session.handle)
          result = session.xenapi.host.call_plugin(
              host, 'netwrap', 'run_command',
              {'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})
          return json.loads(result)
      except Exception as e:
          traceback.print_exc()
          sys.exit(RC_XENAPI_ERROR)

  After:

  def run_command(url, username, password, user_args, cmd_input):
      try:
          session = XenAPI.Session(url)
          session.login_with_password(username, password)
          host = session.xenapi.session.get_this_host(session.handle)
          result = session.xenapi.host.call_plugin(
              host, 'netwrap', 'run_command',
              {'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})
          session.xenapi.session.logout()
          return json.loads(result)
      except Exception as e:
          traceback.print_exc()
          sys.exit(RC_XENAPI_ERROR)

  
  After making this change, the logs still show the sessions being rapidly created, but it also shows them being destroyed. The "exceeds limit" error no longer occurs, and live-migrations now succeed.

  
  Regards,

  Alex Oughton

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1558721/+subscriptions