papercuts-ninja team mailing list archive
-
papercuts-ninja team
-
Mailing list archive
-
Message #01503
[Bug 1630516] Re: Logrotate fails repeatedly with /etc/cron.daily/logrotate: error: error creating output file /var/log/... File exists run-parts: /etc/cron.daily/logrotate exited with return code 1
I'll just point out that this can perhaps be considered a security bug,
because if log rotations stop happening, the disk will inevitably fill
up sooner or later, which means security patches will stop being
applied, which means the machine will become insecure. You might say
that that is a bit tenuous, but it is the actual real effect and a
genuine problem I have been having with Ubuntu servers.
--
You received this bug notification because you are a member of Papercuts
Ninjas, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1630516
Title:
Logrotate fails repeatedly with /etc/cron.daily/logrotate: error:
error creating output file /var/log/... File exists run-parts:
/etc/cron.daily/logrotate exited with return code 1
Status in logrotate package in Ubuntu:
Confirmed
Status in logrotate package in Debian:
Fix Released
Bug description:
Good afternoon.
I have started seeing something very similar to Debian Dug 734688 "Logs are not rotated for a month" but in the latest Ubuntu LTS (16.04). I seem to have
$ logrotate --version
logrotate 3.8.7
bundled in it. A few weeks ago I started getting root emails such as this:
> Subject: Cron <root@warden> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
>
> /etc/cron.daily/logrotate:
> error: error creating output file /var/log/munin/munin-node.log.1: File exists
> run-parts: /etc/cron.daily/logrotate exited with return code 1
When I inspected the area of concern I was able to see that there was an existing .1 file.
manager@warden:/var/log/munin$ ll
total 580
drwxr-xr-x 2 munin adm 4096 Sep 27 06:31 ./
drwxr-xr-x 13 root syslog 4096 Oct 5 06:26 ../
-rw-r--r-- 1 root root 3440 Sep 26 13:39 munin-node-configure.log
-rw-r--r-- 1 root root 490251 Oct 5 10:25 munin-node.log
-rw-r--r-- 1 root root 56598 Sep 21 02:01 munin-node.log.1
-rw-r--r-- 1 root root 24576 Aug 31 02:01 munin-node.log.2
-rw-r--r-- 1 root root 1906 Sep 19 06:25 munin-node.log.8.gz
The contents of the munin-node.log file seem to run from the 19th September until today. Unlike other parts of this bug the .1 and .2 files do not seem to be already compressed.
I deleted all but the munin-node.log file to see if it would resolve the problem and was going to leave it at that. Then I noticed that I have had another Ubuntu machine which has been sending similar emails for the past week:
> Subject: Cron <root@trac> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
>
> /etc/cron.daily/logrotate:
> error: error creating output file /var/log/syslog.1.gz: File exists
> run-parts: /etc/cron.daily/logrotate exited with return code 1
Different file, different machine but a very similar error message.
Checking on the syslog file I can see that it better fits with other reports on this bug as my duplicated .1 files has a corresponding .1.gz file.
manager@trac:/var/log$ ll syslog*
-rw-r----- 1 syslog adm 918492 Oct 5 10:30 syslog
-rw-r----- 1 syslog adm 125819 Sep 30 06:25 syslog.1
-rw-r----- 1 syslog adm 20638 Oct 2 02:01 syslog.1.gz
-rw-r----- 1 syslog adm 41989 Sep 30 02:00 syslog.2.gz
-rw-r----- 1 syslog adm 18654 Sep 28 02:01 syslog.3.gz
-rw-r----- 1 syslog adm 31720 Sep 26 06:40 syslog.4.gz
-rw-r----- 1 syslog adm 33151 Sep 25 02:01 syslog.5.gz
-rw-r----- 1 syslog adm 17290 Sep 23 02:01 syslog.6.gz
-rw-r----- 1 syslog adm 39275 Sep 21 06:35 syslog.7.gz
There has been some speculation that a full or nearly full /var
partition would cause this issue. I can confirm that /var is part of
/ on my systems and that presently both of them have several gigabytes
of space. I run Munin an Icinga to monitor system state. Neither of
these show / being completely full in the past month. They have both
had /boot fill significantly. Trac had a highest use value of / being
99.28% full in the past year but warden has only had a peak of 33% in
the past year.
A quick search of the internet suggests a couple of other people reporting similar issues:
https://github.com/gitlabhq/gitlabhq/issues/6894
http://raspberrypi.stackexchange.com/questions/22545/why-are-system-logs-not-rotating
I do not believe that I have altered by logrotate configuration files but here is a copy of the ones I know about:
$ cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly
# use the syslog group by default, since this is the owning group
# of /var/log/syslog.
su root syslog
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# uncomment this if you want your log files compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
missingok
monthly
create 0664 root utmp
rotate 1
}
/var/log/btmp {
missingok
monthly
create 0660 root utmp
rotate 1
}
# system-specific logs may be configured here
$ cat /etc/logrotate.d/rsyslog
/var/log/syslog
{
rotate 7
daily
missingok
notifempty
delaycompress
compress
postrotate
reload rsyslog >/dev/null 2>&1 || true
endscript
}
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
reload rsyslog >/dev/null 2>&1 || true
endscript
}
manager@trac:/var/log$ cat /etc/logrotate.d/munin-node
/var/log/munin/munin-node.log {
daily
missingok
rotate 7
compress
copytruncate
notifempty
create 640 root root
}
I am confident that is all the files which are relative to the configuration:
manager@trac:/var/log$ rgrep syslog /etc/logrotate.*
/etc/logrotate.conf:# use the syslog group by default, since this is the owning group
/etc/logrotate.conf:# of /var/log/syslog.
/etc/logrotate.conf:su root syslog
/etc/logrotate.d/ufw: invoke-rc.d rsyslog reload >/dev/null 2>&1 || true
/etc/logrotate.d/rsyslog:/var/log/syslog
/etc/logrotate.d/rsyslog: reload rsyslog >/dev/null 2>&1 || true
/etc/logrotate.d/rsyslog: reload rsyslog >/dev/null 2>&1 || true
manager@trac:/var/log$ rgrep munin /etc/logrotate.*
/etc/logrotate.d/munin-node:/var/log/munin/munin-node.log
If there is anything else I can do to help diagnose the issue please
let me know.
Usual boiler plate bits:
"trac" machine:
$ lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
$ apt-cache policy logrotate
logrotate:
Installed: 3.8.7-1ubuntu1
Candidate: 3.8.7-1ubuntu1
Version table:
*** 3.8.7-1ubuntu1 0
500 http://gb.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
100 /var/lib/dpkg/status
"warden" machine:
$ lsb_release -rd
Description: Ubuntu 16.04.1 LTS
Release: 16.04
$ apt-cache policy logrotate
logrotate:
Installed: 3.8.7-2ubuntu2
Candidate: 3.8.7-2ubuntu2
Version table:
*** 3.8.7-2ubuntu2 500
500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
100 /var/lib/dpkg/status
N: Ignoring file '50unattended-upgrades.ucf-dist' in directory '/etc/apt/apt.conf.d/' as it has an invalid filename extension
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: logrotate 3.8.7-1ubuntu1
ProcVersionSignature: Ubuntu 4.2.0-42.49~14.04.1-generic 4.2.8-ckt12
Uname: Linux 4.2.0-42-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.21
Architecture: amd64
Date: Wed Oct 5 11:14:53 2016
InstallationDate: Installed on 2015-12-14 (295 days ago)
InstallationMedia: Ubuntu-Server 14.04.3 LTS "Trusty Tahr" - Beta amd64 (20150805)
SourcePackage: logrotate
UpgradeStatus: No upgrade log present (probably fresh install)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/logrotate/+bug/1630516/+subscriptions
