kernel-packages team mailing list archive

Thread
Date
Re: [Bug 1423672] Re: ext4 turned read-only during logrotate daily run

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Simon Déziel <1423672@xxxxxxxxxxxxxxxxxx>
Date: Mon, 09 Mar 2015 15:06:48 -0000
Reply-to: Bug 1423672 <1423672@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
On 03/04/2015 05:07 PM, Simon Déziel wrote:
> Before loosing anyone's time on this issue, I'll first make sure the
> hypervisor's RAM is sane. So I'll get back after some memtest86+ then
> I'll look at the 4.0 kernel.

After 15+ hours of memtest86+ (8 full passes), I don't think the RAM is
at fault. I ran a fsck on every VMs and many of them had issues. I'm
still using the default Ubuntu kernel for now.

One weird thing I noticed is that apparently all the affected VMs have
trouble with the /etc/cron.daily/logrotate job:

 error: error creating output file /var/log/syslog.1.gz: File exists
 run-parts: /etc/cron.daily/logrotate exited with return code 1

On those, there was a very old /var/log/syslog.1 and a more recent .1.gz
one too. I removed both and will see if it helps.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1423672

Title:
  ext4 turned read-only during logrotate daily run

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  I noticed that one of my VM had this "dmesg -T" output:

  [Tue Feb 17 09:53:27 2015] systemd-udevd[5433]: starting version 204
  [Thu Feb 19 06:25:08 2015] EXT4-fs error (device vda1): ext4_mb_generate_buddy:756: group 5, block bitmap and bg descriptor inconsistent: 16446 vs 16445 free clusters
  [Thu Feb 19 06:25:09 2015] Aborting journal on device vda1-8.
  [Thu Feb 19 06:25:09 2015] EXT4-fs (vda1): Remounting filesystem read-only
  [Thu Feb 19 06:25:09 2015] ------------[ cut here ]------------
  [Thu Feb 19 06:25:09 2015] WARNING: CPU: 0 PID: 9946 at /build/buildd/linux-3.13.0/fs/ext4/ext4_jbd2.c:259 __ext4_handle_dirty_metadata+0x1a2/0x1c0()
  [Thu Feb 19 06:25:09 2015] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_filter ip_tables x_tables serio_raw psmouse floppy
  [Thu Feb 19 06:25:09 2015] CPU: 0 PID: 9946 Comm: logrotate Not tainted 3.13.0-45-generic #74-Ubuntu
  [Thu Feb 19 06:25:09 2015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
  [Thu Feb 19 06:25:09 2015]  0000000000000009 ffff880003a11aa0 ffffffff81720eb6 0000000000000000
  [Thu Feb 19 06:25:09 2015]  ffff880003a11ad8 ffffffff810677cd ffff880000c41340 0000000000000000
  [Thu Feb 19 06:25:09 2015]  ffff88000a58e000 ffffffff81835280 0000000000001302 ffff880003a11ae8
  [Thu Feb 19 06:25:09 2015] Call Trace:
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81720eb6>] dump_stack+0x45/0x56
  [Thu Feb 19 06:25:09 2015]  [<ffffffff810677cd>] warn_slowpath_common+0x7d/0xa0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff810678aa>] warn_slowpath_null+0x1a/0x20
  [Thu Feb 19 06:25:09 2015]  [<ffffffff8126e862>] __ext4_handle_dirty_metadata+0x1a2/0x1c0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81246a5a>] ? ext4_dirty_inode+0x2a/0x60
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81277086>] ext4_free_blocks+0x646/0xbf0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff810aacc5>] ? wake_up_bit+0x25/0x30
  [Thu Feb 19 06:25:09 2015]  [<ffffffff812685b5>] ext4_ext_rm_leaf+0x505/0x8f0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81267527>] ? __ext4_ext_check+0x197/0x370
  [Thu Feb 19 06:25:09 2015]  [<ffffffff8126ad00>] ? ext4_ext_remove_space+0xc0/0x7e0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff8126af5c>] ext4_ext_remove_space+0x31c/0x7e0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff8126d300>] ext4_ext_truncate+0xb0/0xe0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81244eb9>] ext4_truncate+0x379/0x3c0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81245a18>] ext4_evict_inode+0x408/0x4d0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff811d8f60>] evict+0xb0/0x1b0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff811d9775>] iput+0xf5/0x180
  [Thu Feb 19 06:25:09 2015]  [<ffffffff811d4698>] __dentry_kill+0x1a8/0x200
  [Thu Feb 19 06:25:09 2015]  [<ffffffff811d4795>] dput+0xa5/0x180
  [Thu Feb 19 06:25:09 2015]  [<ffffffff811bf7e6>] __fput+0x176/0x260
  [Thu Feb 19 06:25:09 2015]  [<ffffffff811bf91e>] ____fput+0xe/0x10
  [Thu Feb 19 06:25:09 2015]  [<ffffffff810882f7>] task_work_run+0xa7/0xe0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81013ed7>] do_notify_resume+0x97/0xb0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81731c2a>] int_signal+0x12/0x17
  [Thu Feb 19 06:25:09 2015] ---[ end trace ebff9843d81b5c41 ]---
  [Thu Feb 19 06:25:09 2015] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
  [Thu Feb 19 06:25:09 2015] IP: [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
  [Thu Feb 19 06:25:09 2015] PGD 167067 PUD 161067 PMD 0 
  [Thu Feb 19 06:25:09 2015] Oops: 0000 [#1] SMP 
  [Thu Feb 19 06:25:09 2015] Modules linked in: nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack iptable_filter ip_tables x_tables serio_raw psmouse floppy
  [Thu Feb 19 06:25:09 2015] CPU: 0 PID: 9946 Comm: logrotate Tainted: G        W     3.13.0-45-generic #74-Ubuntu
  [Thu Feb 19 06:25:09 2015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
  [Thu Feb 19 06:25:09 2015] task: ffff880009ac4800 ti: ffff880003a10000 task.ti: ffff880003a10000
  [Thu Feb 19 06:25:09 2015] RIP: 0010:[<ffffffff8125d4c1>]  [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
  [Thu Feb 19 06:25:09 2015] RSP: 0000:ffff880003a11a58  EFLAGS: 00010292
  [Thu Feb 19 06:25:09 2015] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000086
  [Thu Feb 19 06:25:09 2015] RDX: 0000000000001302 RSI: ffffffff81a6e81f RDI: 0000000000000000
  [Thu Feb 19 06:25:09 2015] RBP: ffff880003a11ae8 R08: ffffffff81a78568 R09: 0000000000000005
  [Thu Feb 19 06:25:09 2015] R10: 00000000ffffffe2 R11: ffff880003a117ce R12: 0000000000000086
  [Thu Feb 19 06:25:09 2015] R13: ffffffff81835280 R14: 0000000000001302 R15: ffffffff81a78568
  [Thu Feb 19 06:25:09 2015] FS:  00007f74eaca4840(0000) GS:ffff88000b800000(0000) knlGS:0000000000000000
  [Thu Feb 19 06:25:09 2015] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  [Thu Feb 19 06:25:09 2015] CR2: 0000000001de4000 CR3: 0000000009b45000 CR4: 00000000000006f0
  [Thu Feb 19 06:25:09 2015] Stack:
  [Thu Feb 19 06:25:09 2015]  ffff880003a11a60 0000000000000103 ebff9843d81b5c41 000000000000321e
  [Thu Feb 19 06:25:09 2015]  00000000000014d8 0000000000000092 000000000000020e ffff88000a58e000
  [Thu Feb 19 06:25:09 2015]  ffff880003a11ae8 ffffffff8126e372 ffffffff810677df ffff880000c41340
  [Thu Feb 19 06:25:09 2015] Call Trace:
  [Thu Feb 19 06:25:09 2015]  [<ffffffff8126e372>] ? ext4_journal_abort_handle+0x42/0xc0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff810677df>] ? warn_slowpath_common+0x8f/0xa0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff8126e7cf>] __ext4_handle_dirty_metadata+0x10f/0x1c0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81277086>] ext4_free_blocks+0x646/0xbf0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff810aacc5>] ? wake_up_bit+0x25/0x30
  [Thu Feb 19 06:25:09 2015]  [<ffffffff812685b5>] ext4_ext_rm_leaf+0x505/0x8f0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81267527>] ? __ext4_ext_check+0x197/0x370
  [Thu Feb 19 06:25:09 2015]  [<ffffffff8126ad00>] ? ext4_ext_remove_space+0xc0/0x7e0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff8126af5c>] ext4_ext_remove_space+0x31c/0x7e0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff8126d300>] ext4_ext_truncate+0xb0/0xe0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81244eb9>] ext4_truncate+0x379/0x3c0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81245a18>] ext4_evict_inode+0x408/0x4d0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff811d8f60>] evict+0xb0/0x1b0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff811d9775>] iput+0xf5/0x180
  [Thu Feb 19 06:25:09 2015]  [<ffffffff811d4698>] __dentry_kill+0x1a8/0x200
  [Thu Feb 19 06:25:09 2015]  [<ffffffff811d4795>] dput+0xa5/0x180
  [Thu Feb 19 06:25:09 2015]  [<ffffffff811bf7e6>] __fput+0x176/0x260
  [Thu Feb 19 06:25:09 2015]  [<ffffffff811bf91e>] ____fput+0xe/0x10
  [Thu Feb 19 06:25:09 2015]  [<ffffffff810882f7>] task_work_run+0xa7/0xe0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81013ed7>] do_notify_resume+0x97/0xb0
  [Thu Feb 19 06:25:09 2015]  [<ffffffff81731c2a>] int_signal+0x12/0x17
  [Thu Feb 19 06:25:09 2015] Code: 48 89 e5 41 57 4d 89 c7 41 56 41 89 d6 41 55 49 89 f5 48 c7 c6 1f e8 a6 81 41 54 49 89 cc 53 48 89 fb 48 83 ec 68 4c 89 4c 24 60 <48> 8b 47 28 48 8b 57 40 48 8b 80 f8 02 00 00 48 8b 40 68 89 90 
  [Thu Feb 19 06:25:09 2015] RIP  [<ffffffff8125d4c1>] __ext4_error_inode+0x31/0x160
  [Thu Feb 19 06:25:09 2015]  RSP <ffff880003a11a58>
  [Thu Feb 19 06:25:09 2015] CR2: 0000000000000028
  [Thu Feb 19 06:25:10 2015] ---[ end trace ebff9843d81b5c42 ]---

  cron.daily jobs fired at 6:25:01 apparently:

  # tail -n2 /var/log/syslog
  Feb 19 06:17:01 git CRON[9848]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
  Feb 19 06:25:01 git CRON[9853]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))

  # run-parts --test /etc/cron.daily
  /etc/cron.daily/apt
  /etc/cron.daily/autoremove
  /etc/cron.daily/dpkg
  /etc/cron.daily/hdd-backup
  /etc/cron.daily/logrotate
  /etc/cron.daily/passwd
  /etc/cron.daily/upstart

  It seems like all the jobs ran and the upstart one somehow triggered
  the crash:

  # ls -alt /var/log/upstart/ | head
  total 272
  drwxrwxr-x 5 root syslog 4096 Feb 19 06:25 ..
  drwxr-xr-x 2 root root   4096 Feb 14 06:25 .
  -rw-r----- 1 root root    591 Feb 12 12:26 ureadahead.log.1.gz
  -rw-r----- 1 root root    178 Feb 12 12:25 mountall.log.1.gz

  
  Now that I have collected some information (sorry, I don't have ubuntu-bug installed on the VM) I'll reboot it and see how it goes.

  
  More information on the VM:

  # lsb_release -rd
  Description:	Ubuntu 14.04.2 LTS
  Release:	14.04
  # apt-cache policy linux-image-3.13.0-45-generic
  linux-image-3.13.0-45-generic:
    Installed: 3.13.0-45.74
    Candidate: 3.13.0-45.74
    Version table:
   *** 3.13.0-45.74 0
          500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
          100 /var/lib/dpkg/status
  --- 
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Feb 19 14:34 seq
   crw-rw---- 1 root audio 116, 33 Feb 19 14:34 timer
  AplayDevices: Error: [Errno 2] No such file or directory
  ApportVersion: 2.14.1-0ubuntu3.7
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory
  AudioDevicesInUse: Error: [Errno 2] No such file or directory
  CRDA: Error: [Errno 2] No such file or directory
  CurrentDmesg:
   [   13.891047] init: console-font main process (855) terminated with status 71
   [   13.952825] init: plymouth-splash main process (870) terminated with status 1
   [  217.853139] random: nonblocking pool is initialized
  DistroRelease: Ubuntu 14.04
  IwConfig: Error: [Errno 2] No such file or directory
  Lspci: Error: [Errno 2] No such file or directory
  Lsusb: Error: [Errno 2] No such file or directory
  MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
  Package: linux (not installed)
  PciMultimedia:
   
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcFB:
   
  ProcKernelCmdLine: root=UUID=cb9cbdad-c668-4503-85db-fcf9b02f3495 ro console=tty0 console=ttyS0,38400
  ProcVersionSignature: Ubuntu 3.13.0-45.74-generic 3.13.11-ckt13
  RelatedPackageVersions:
   linux-restricted-modules-3.13.0-45-generic N/A
   linux-backports-modules-3.13.0-45-generic  N/A
   linux-firmware                             N/A
  RfKill: Error: [Errno 2] No such file or directory
  Tags:  trusty
  Uname: Linux 3.13.0-45-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups:
   
  _MarkForUpload: True
  dmi.bios.date: 01/01/2011
  dmi.bios.vendor: Bochs
  dmi.bios.version: Bochs
  dmi.chassis.type: 1
  dmi.chassis.vendor: Bochs
  dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-2.0:cvnBochs:ct1:cvr:
  dmi.product.name: Standard PC (i440FX + PIIX, 1996)
  dmi.product.version: pc-i440fx-2.0
  dmi.sys.vendor: QEMU

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1423672/+subscriptions
References

[Bug 1423672] [NEW] ext4 turned read-only following a cronjob run
From: Simon Déziel, 2015-02-19
[Bug 1423672] Re: ext4 turned read-only during logrotate daily run
From: Simon Déziel, 2015-03-04