yahoo-eng-team team mailing list archive

[mchlumsky <martin.chlumsky@xxxxxxxxx>]

We have also noticed this problem.
We did an strace on horizon and found that the connections made by horizon to keystone were not getting closed.

Digging deeper, we found 2 very similar commits that fix connection
closure issues related to keystone.

https://github.com/openstack/python-
keystoneclient/commit/8fcacdc7c74f5ac68e8e55ea8c15918c452411fe

and

https://github.com/openstack/keystoneauth/commit/dbcbf414ac8423e97d77d0bda8157be5350530f0

I think that the commit in keystoneauth is incomplete, it's missing the
destination of the move of the _FakeRequestSession class and so it's
just a removal.

In Newton, horizon is using keystoneclient to get keystone sessions and we don't see the connection leaks.
Somewhere between Newton and Queens, horizon switched to keystoneauth and lost the _FakeRequestSession hack and so we are now seeing a regression where horizon is leaking connections again.

I am attaching a patch that we are currently testing, it re-introduces the _FakeRequestSession class into keystoneauth.
The patch is a little naive, I don't know what to do with the instance attribute "adapters" of _FakeRequestSession, I just set it to an empty list because it is being referenced by code elsewhere. It seems to be working for now.


** Patch added: "session.py.patch"
   https://bugs.launchpad.net/horizon/+bug/1780164/+attachment/5184962/+files/session.py.patch

** Also affects: keystoneauth
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1780164

Title:
  httpd leaks open files

Status in OpenStack Dashboard (Horizon):
  New
Status in keystoneauth:
  New

Bug description:
  horizon version 13.0.0-1.el7 (Queens) on centos  7.4.1708

  After some time working on the dashboard, it stops working and throw
  this error in the error log

  [Wed Jul 04 22:49:33.744241 2018] [:error] [pid 23924] [remote
  10.144.187.237:52] IOError: [Errno 24] Too many open files:
  '/usr/share/openstack-
  dashboard/openstack_dashboard/templates/500.html'

  if we check the open files of this process 23924

   ls -l /proc/23924/fd | wc -l
  1023

  even if we increase the nofile limit of this process, it doesn't help as the open files is always increasing
  the problem is cleared if we restart the httpd process, but then the open files will increase again

  how to reproduce

  1- login to the dashboard
  2- get the pid from the error_log file
  [Wed Jul 04 22:50:58.620832 2018] [:error] [pid 23924] INFO openstack_auth.forms Login successful for user "demo" using domain "default"
  3- browse the dashboard different menus, specially the network topology tab
  4- monitor the open files no. with ls -l /proc/<pid>/fd | wc -l

  observation
  the no. is always increasing

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1780164/+subscriptions