touch-packages team mailing list archive

[Vitaly Sedelnik <1481671@xxxxxxxxxxxxxxxxxx>]

Linux team - please try to reproduce this issue given the info provided
by @abubyr.

** Also affects: mos/6.0-updates
   Importance: Undecided
       Status: New

** No longer affects: mos/6.0-updates

** Changed in: mos
    Milestone: None => 6.0-updates

** Changed in: mos
       Status: Incomplete => Confirmed

** Changed in: mos
     Assignee: Andrey Bubyr (abubyr) => MOS Linux (mos-linux)

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to rsyslog in Ubuntu.
https://bugs.launchpad.net/bugs/1481671

Title:
  [mos] rsyslog periodically crashes with *** glibc detected ***
  rsyslogd: double free or corruption (fasttop)  preventing log into the
  system

Status in Mirantis OpenStack:
  Confirmed
Status in rsyslog package in Ubuntu:
  New

Bug description:
  Rsyslog crashes every 10-14 days on MogoDB nodes with coredump in /var/log/messages:
  <46>Aug  4 15:01:56 node-115 rsyslogd: rsyslogd's groupid changed to 103
  <46>Aug  4 15:01:56 node-115 rsyslogd: rsyslogd's userid changed to 101
  <6>Aug  4 15:02:12 node-115 kernel: imklog 5.8.6, log source = /proc/kmsg started.
  <46>Aug  4 15:02:12 node-115 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="58757" x-info="http://www.
  rsyslog.com"] start
  <46>Aug  4 15:02:12 node-115 rsyslogd: rsyslogd's groupid changed to 103
  <46>Aug  4 15:02:12 node-115 rsyslogd: rsyslogd's userid changed to 101
  <6>Aug  4 15:02:13 node-115 kernel: imklog 5.8.6, log source = /proc/kmsg started.
  <46>Aug  4 15:02:13 node-115 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="58773" x-info="http://www.
  rsyslog.com"] start
  <46>Aug  4 15:02:13 node-115 rsyslogd: rsyslogd's groupid changed to 103
  <46>Aug  4 15:02:13 node-115 rsyslogd: rsyslogd's userid changed to 101
  <6>Aug  4 15:02:21 node-115 kernel: imklog 5.8.6, log source = /proc/kmsg started.
  <46>Aug  4 15:02:21 node-115 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="58788" x-info="http://www.
  rsyslog.com"] start
  <46>Aug  4 15:02:21 node-115 rsyslogd: rsyslogd's groupid changed to 103
  <46>Aug  4 15:02:21 node-115 rsyslogd: rsyslogd's userid changed to 101
  *** glibc detected *** rsyslogd: double free or corruption (fasttop): 0x00007f5e88032100 ***
  ======= Backtrace: =========
  /lib/x86_64-linux-gnu/libc.so.6(+0x7e846)[0x7f5ea2101846]
  rsyslogd[0x42ee36]
  rsyslogd[0x431697]
  rsyslogd[0x431ce7]
  rsyslogd(wtiWorker+0xc3)[0x42b513]
  rsyslogd[0x42a78a]
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f5ea2855e9a]
  /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f5ea217731d]
  ======= Memory map: ========
  00400000-00456000 r-xp 00000000 08:03 8132346                            /usr/sbin/rsyslogd
  00656000-00658000 r--p 00056000 08:03 8132346                            /usr/sbin/rsyslogd
  00658000-0065e000 rw-p 00058000 08:03 8132346                            /usr/sbin/rsyslogd
  0065e000-0065f000 rw-p 00000000 00:00 0
  00f18000-00f39000 rw-p 00000000 00:00 0                                  [heap]
  00f39000-00f7b000 rw-p 00000000 00:00 0                                  [heap]
  7f5e84000000-7f5e84040000 rw-p 00000000 00:00 0
  7f5e84040000-7f5e88000000 ---p 00000000 00:00 0
  7f5e88000000-7f5e88043000 rw-p 00000000 00:00 0
  7f5e88043000-7f5e8c000000 ---p 00000000 00:00 0
  7f5e8c000000-7f5e8c1eb000 rw-p 00000000 00:00 0
  7f5e8c1eb000-7f5e90000000 ---p 00000000 00:00 0
  < ... memory map ... >
  Full memory map could be found in attached 'messages' log.
  When rsyslog crashes in this manner it is impossible to login into the system. Process /bin/login successfully authenticates user, but does not born shell process, constantly waiting for rsyslog reading login log message from Unix socket.

  Obviously Fuel master reports that node in 'False' state (because it
  becomes non-sshable too).

  Workaround: fully restart rsyslog after crash (service rsyslog
  restart) if some active login session is present or reboot the server
  if not.

  MOS 6.0, Ubuntu 12.04. Rsyslog v. 5.8.6 (from Fuel node repos).
  Rsyslog config files placed in rsyslog folder of attached archive.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mos/+bug/1481671/+subscriptions