sts-sponsors team mailing list archive

["Guilherme G. Piccoli" <1750013@xxxxxxxxxxxxxxxxxx>]

Dan, thanks for looking at the patch and for your question!


I tried to explain this a bit in comment #1, not too much detailed though:

"[...]closing session should be somehow added to gc [...] and then (the gc) performs
the clean-up.

To include a session in gc, the function session_add_to_gc_queue() should
be called.

[...] (this) patch proactively puts a session in gc on session_remove_fifo().
This function is always called in the event of Release a session - when the ssh ends, for example.
[...]"


Being a bit more specific: when SSH ends, for example, a Release event is sent
through dbus and systemd-logind captures it, in the function manager_message_handler().

>From there, the function session_remove_fifo() is called. That point is our "bootstrap"
to add the closing session on gc, our first addition there. But unfortunately due to
an unpredictable timing of cgroup become empty for that session, the gc might fail
to remove the session right in that moment. So, we need to re-add the closing session to
gc from...within the gc! In my experiments, 2 or 3 re-additions are enough to get the session removed.

It wouldn't be necessary to do these 2 steps above IF the cgroup empty notification
was working fine. In that case, after the cgroup for the closing session become empty,
a notification handler would be triggered in the systemd-logind and that handler would
add the session to gc, and then gc would clean it up.
But, since in Trusty we have upstart as our init system (and systemd is highly patched to co-exist in this fashion) and cgroup empty notifications were broken in systemd (until a complete refactor of cgroup handling in systemd), our approach here is the less expensive one.

In summary, the above 2 steps [adding the session to gc in
session_remove_fifo() and re-adding it from within gc] are the most
important parts of this patch - without them we leak entire sessions and
eventually the logind process gets OOM'ed.

Thanks,


Guilherme

-- 
You received this bug notification because you are a member of STS
Sponsors, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1750013

Title:
  systemd-logind: memory leaks on session's connections (trusty-only)

Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Trusty:
  In Progress
Status in systemd source package in Xenial:
  Fix Released
Status in systemd source package in Artful:
  Fix Released
Status in systemd source package in Bionic:
  Fix Released

Bug description:
  Below the SRU request form. Please refer to the Original Description
  to a more comprehensive explanation of the problem observed.

  
  [Impact] 

   * systemd-logind tool is leaking memory at each session connected. The 
   issues happens in systemd from Trusty (14.04) only.

   * Three issues observed:
    - systemd-logind is leaking entire sessions, i.e, the sessions are not 
      feeed after they're closed. In order to fix that, we proactively add 
      the sessions to systemd garbage collector (gc) when they are closed. 
      Also, part of the fix is to make cgmanager package a dependency. Refer 
      to comment #1 to a more thorough explanation of the issue and the fix.

    - a small memory leak was observed in the session creation logic of 
      systemd-logind. The fix for that is the addition of an appropriate 
      free() call. Refer to comment #2 to more details on the issue and fix.

    - another small memory leak was observed in the cgmanager glue code of 
      systemd-logind - this code is only present in this specific Ubuntu 
      release of the package, due to necessary compatibility layer with 
      upstart init system. The fix is to properly call free() in 2 
      functions. Refer to comment #3 to a deep exposition of the issue and 
      the fix.

  
  [Test Case]

   * The basic test-case is to run the following loop from a remote machine:
     while true; do ssh <hostname-target> "whoami"; done

   * It's possible to watch the increase in memory consumption from 
     "systemd-logind" process in the target machine. One can use the
     "ps uax" command to verify the RSS of the process, or count its 
     anonymous pages from /proc/<logind_pid>/smaps.

  
  [Regression Potential] 

   * Since the fixes are small and not intrusive, the potential for 
     regressions are low. More regression considerations on comments #1, #2 
     and #3 for each fix.

   * A potential small regressson is performance-wise, since now we add 
     sessions to garbage collector proactively.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1750013/+subscriptions