touch-packages team mailing list archive

["Aaron C. de Bruyn" <aaron@xxxxxxxxxxxx>]

Er.  Apparently it's a feature, not a bug.

"rsyslog was still a running process, it just stopped logging both
remotely and to local files. We found a few discussions of this problem
from 2012, 2011 and 2009 but they didn’t entirely cover the problem.
However, the common thread was a connectivity issue causing problems
with the queuing. Although not definitive, that every server in just one
of our data centres saw this problem gave weight to a network based
issue which may have caused rsyslog to hang for all actions, even though
it was a network issue and we still had disk based logging enabled.

After discussing the issue with the Papertrail support guys, in order to
combat this we decided to enable reliable f0rwarding which means rsyslog
will queue log lines in memory and then to disk if the remote server
cannot be reached, posting them when connectivity returns. This is
necessary because syslog over TCP is not entirely reliable" (See
https://blog.serverdensity.com/reliable-forwarding-with-rsyslog/)

** Changed in: rsyslog (Ubuntu)
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1389356

Title:
  rsyslogd stops responding, doesn't log data, and eventually hangs the
  system

Status in “rsyslog” package in Ubuntu:
  Invalid

Bug description:
  This is happening on a wide variety of client systems that are all
  dissimilar--some are virtual machines, some are physical machines, and
  it spans 11.04, 12.04, 13.10, and 14.04.

  It started about two months ago.

  Approximately once per week, we will start getting calls from all our
  clients that services running on linux boxes are unavailable or
  extremely slow.

  Attempts to access the boxes via SSH will either not work (hang and
  then timeout after ~2 minutes) or succeed (after hanging for ~1
  minute).  Then the shell prompt takes a while (maybe 30 seconds) to
  display.

  After spending several frustrating hours with one particular box, I noticed the following:
  * Very low disk IO (i.e. the box isn't hammering the disk)
  * Memory usage was appropriate
  * Network IO was appropriate and responsive (ping, traceroute, wget, etc...)
  * Logs were all 'empty'.  Last log data was in the log from the previous evening (i.e. /var/log/syslog.1 has a final entry at 8:36 PM PST from the previous night)

  Running the command 'restart rsyslogd' immediately returns the box to
  normal operation.

  After a few more testing sessions, I can see that rsyslogd is running
  on all these boxes, it just appears to be unresponsive.

  The issue happens fairly regularly--every 7-10 days, and it happens on
  multiple disparate systems on different networks at approximately the
  same time.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/1389356/+subscriptions