kernel-packages team mailing list archive
-
kernel-packages team
-
Mailing list archive
-
Message #175281
[Bug 1576599] Re: XFS Oopsing 4.2.0-35-generic Kernel - Block out of range
apport information
** Changed in: linux (Ubuntu)
Status: Incomplete => Confirmed
** Tags added: apport-collected trusty uec-images
** Description changed:
After updating the Kernel on two of our Trusty Ceph servers from
3.19.0-33-generic (vivid) to 4.2.0-35-generic (wily), both servers have
now Oopsed multiple times accessing some of their XFS filesystems.
Once this error occurs, the XFS filesystem becomes permanently
inaccessible. Even after a reboot, any attempt to access the filesystem
in question either by mounting it, or attempting an xfs_repair will
trigger the same Oops again.
These filesystems did not immediately fail until several hours after the
upgrade. I suspect that this is related to some housekeeping being
triggered within the filesystem.
This is the call trace from the moment a filesystem first failed:
Apr 28 22:47:25 ceph-store5 kernel: [26692.804773] ------------[ cut here ]------------
Apr 28 22:47:25 ceph-store5 kernel: [26692.810046] WARNING: CPU: 8 PID: 5195 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
Apr 28 22:47:25 ceph-store5 kernel: [26692.824715] Modules linked in: bridge openvswitch xfs dcdbas ipmi_devintf intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev dm_multipath sb_edac scsi_dh edac_core input_leds mei_me lpc_ich mei ipmi_si ipmi_msghandler 8250_fintek wmi acpi_power_meter 8021q acpi_pad mac_hid garp mrp stp llc shpchp lp parport btrfs bcache raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx hid_generic xor usbhid hid raid6_pq bnx2x raid1 ptp mpt2sas raid0 pps_core multipath raid_class mdio scsi_transport_sas nvme libcrc32c linear
Apr 28 22:47:25 ceph-store5 kernel: [26692.891939] CPU: 8 PID: 5195 Comm: ceph-osd Tainted: G D W 4.2.0-35-generic #40~14.04.1-Ubuntu
Apr 28 22:47:25 ceph-store5 kernel: [26692.902689] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
Apr 28 22:47:25 ceph-store5 kernel: [26692.911303] 0000000000000000 ffff8800c7bf74a8 ffffffff817bcbf8 0000000000000000
Apr 28 22:47:25 ceph-store5 kernel: [26692.919677] ffffffffc0a7f300 ffff8800c7bf74e8 ffffffff81079b5a ffff8800c7bf7508
Apr 28 22:47:25 ceph-store5 kernel: [26692.928033] ffff88100a28ed80 0000000000000008 0000000000000000 ffff8800c7bf75f8
Apr 28 22:47:25 ceph-store5 kernel: [26692.936441] Call Trace:
Apr 28 22:47:25 ceph-store5 kernel: [26692.939208] [<ffffffff817bcbf8>] dump_stack+0x63/0x81
Apr 28 22:47:25 ceph-store5 kernel: [26692.944992] [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
Apr 28 22:47:25 ceph-store5 kernel: [26692.951752] [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
Apr 28 22:47:25 ceph-store5 kernel: [26692.958346] [<ffffffffc0a3e12e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.965221] [<ffffffffc0a3e1aa>] xfs_buf_get_map+0x2a/0x260 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.972194] [<ffffffffc0a0130a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.979763] [<ffffffffc0a69f26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.987412] [<ffffffffc0a17380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.994564] [<ffffffffc0a01d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.002302] [<ffffffffc0a3dd3f>] ? xfs_buf_trylock+0x1f/0x90 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.009379] [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.016229] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
Apr 28 22:47:25 ceph-store5 kernel: [26693.022814] [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.029669] [<ffffffffc0a020a2>] xfs_alloc_vextent+0x1d2/0x660 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.036915] [<ffffffffc0a12746>] xfs_bmap_btalloc+0x376/0x6b0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.044074] [<ffffffffc0a12aa4>] xfs_bmap_alloc+0x24/0x40 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.050833] [<ffffffffc0a13456>] xfs_bmapi_write+0x486/0x9a0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.057882] [<ffffffffc0a1d1ce>] xfs_da_grow_inode_int+0xae/0x2d0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.065403] [<ffffffff811d1bc2>] ? __kmalloc+0x1d2/0x280
Apr 28 22:47:25 ceph-store5 kernel: [26693.071509] [<ffffffffc0a5b080>] ? kmem_alloc+0x70/0xf0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.078078] [<ffffffffc0a1d427>] xfs_da_grow_inode+0x37/0x90 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.085129] [<ffffffffc0a08562>] xfs_attr_shortform_to_leaf+0xa2/0x2b0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.093151] [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.100199] [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.107251] [<ffffffffc0a047eb>] xfs_attr_set+0x31b/0x3e0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.114009] [<ffffffffc0a5acc2>] xfs_xattr_set+0x42/0x60 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.120647] [<ffffffff81212a42>] generic_setxattr+0x62/0x80
Apr 28 22:47:25 ceph-store5 kernel: [26693.127107] [<ffffffff812134e1>] __vfs_setxattr_noperm+0x61/0x1a0
Apr 28 22:47:25 ceph-store5 kernel: [26693.134045] [<ffffffff812136c7>] vfs_setxattr+0xa7/0xb0
Apr 28 22:47:25 ceph-store5 kernel: [26693.145904] [<ffffffff8121376d>] ? setxattr+0x9d/0x1b0
Apr 28 22:47:25 ceph-store5 kernel: [26693.151781] [<ffffffff812137fa>] setxattr+0x12a/0x1b0
Apr 28 22:47:25 ceph-store5 kernel: [26693.157626] [<ffffffffc0a73fcb>] ? xfs_acl_exists+0x4b/0x60 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.164599] [<ffffffffc0a74505>] ? posix_acl_access_exists+0x15/0x20 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.172451] [<ffffffffc0a5af52>] ? xfs_vn_listxattr+0xf2/0x160 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.179694] [<ffffffff811f1089>] ? __sb_start_write+0x49/0xf0
Apr 28 22:47:25 ceph-store5 kernel: [26693.186278] [<ffffffffc0a5ad30>] ? xfs_xattr_get+0x50/0x50 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.193122] [<ffffffff81213a90>] SyS_fsetxattr+0xa0/0xd0
Apr 28 22:47:25 ceph-store5 kernel: [26693.199193] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
Apr 28 22:47:25 ceph-store5 kernel: [26693.206471] ---[ end trace c0a9568b8830fc8b ]---
Apr 28 22:47:25 ceph-store5 kernel: [26693.211904] XFS (bcache7): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0
Apr 28 22:47:25 ceph-store5 kernel: [26693.221992] ------------[ cut here ]------------
I notice that xfs_da_grow_inode. Am I right in thinking that this was a
new feature add in the 4.2 kernel?
I expect that the history of these filesystems is important. These would
more have been formatted on a 3.16 kernel. Although I can't inspect
these failed filesystems, I can give information about the other
filesystems on the same servers, which I an 99% sure are identical:
root@ceph-store5:~# xfs_info /var/lib/ceph/osd/ceph-35
meta-data=/dev/bcache6 isize=512 agcount=4, agsize=244188661 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1
data = bsize=4096 blocks=976754644, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=476930, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
==== Attempt to repair a filesystem (after a reboot) ====
root@ceph-store5:~# xfs_repair /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed. Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair. If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.
root@ceph-store5:~# mount -o noatime,inode64 /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c /var/lib/ceph/osd/ceph-39/
Killed
Dmesg output (dmesg -T):
[Fri Apr 29 10:18:45 2016] XFS (bcache2): Mounting V5 Filesystem
[Fri Apr 29 10:18:45 2016] XFS (bcache2): Starting recovery (logdev: internal)
[Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0
[Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------
[Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
[Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
[Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Not tainted 4.2.0-35-generic #40~14.04.1-Ubuntu
[Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 10:18:49 2016] 0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000
[Fri Apr 29 10:18:49 2016] ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928
[Fri Apr 29 10:18:49 2016] ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18
[Fri Apr 29 10:18:49 2016] Call Trace:
[Fri Apr 29 10:18:49 2016] [<ffffffff817bcbf8>] dump_stack+0x63/0x81
[Fri Apr 29 10:18:49 2016] [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
[Fri Apr 29 10:18:49 2016] [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
[Fri Apr 29 10:18:49 2016] [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc06121aa>] xfs_buf_get_map+0x2a/0x260 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d530a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
[Fri Apr 29 10:18:49 2016] [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 10:18:49 2016] [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f2879>] mount_fs+0x39/0x160
[Fri Apr 29 10:18:49 2016] [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
[Fri Apr 29 10:18:49 2016] [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
[Fri Apr 29 10:18:49 2016] [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
[Fri Apr 29 10:18:49 2016] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e0 ]---
[Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0
[Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------
[Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
[Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
[Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G W 4.2.0-35-generic #40~14.04.1-Ubuntu
[Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 10:18:49 2016] 0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000
[Fri Apr 29 10:18:49 2016] ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928
[Fri Apr 29 10:18:49 2016] ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18
[Fri Apr 29 10:18:49 2016] Call Trace:
[Fri Apr 29 10:18:49 2016] [<ffffffff817bcbf8>] dump_stack+0x63/0x81
[Fri Apr 29 10:18:49 2016] [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
[Fri Apr 29 10:18:49 2016] [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
[Fri Apr 29 10:18:49 2016] [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc061235d>] xfs_buf_get_map+0x1dd/0x260 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
[Fri Apr 29 10:18:49 2016] [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 10:18:49 2016] [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f2879>] mount_fs+0x39/0x160
[Fri Apr 29 10:18:49 2016] [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
[Fri Apr 29 10:18:49 2016] [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
[Fri Apr 29 10:18:49 2016] [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
[Fri Apr 29 10:18:49 2016] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e1 ]---
[Fri Apr 29 10:18:49 2016] BUG: unable to handle kernel NULL pointer dereference at 00000000000000f8
[Fri Apr 29 10:18:49 2016] IP: [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] PGD 564eec067 PUD 11408b067 PMD 0
[Fri Apr 29 10:18:49 2016] Oops: 0000 [#1] SMP
[Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
[Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G W 4.2.0-35-generic #40~14.04.1-Ubuntu
[Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 10:18:49 2016] task: ffff88080967ee00 ti: ffff88018e118000 task.ti: ffff88018e118000
[Fri Apr 29 10:18:49 2016] RIP: 0010:[<ffffffffc063e6f6>] [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] RSP: 0018:ffff88018e11ba08 EFLAGS: 00010292
[Fri Apr 29 10:18:49 2016] RAX: 0000000000000000 RBX: ffff88018e11bb18 RCX: 00000000003d2d52
[Fri Apr 29 10:18:49 2016] RDX: 00000000003d2d51 RSI: 0000000000000000 RDI: ffff880afa4e4828
[Fri Apr 29 10:18:49 2016] RBP: ffff88018e11ba28 R08: 000000000001a4f0 R09: ffff88080f8da4f0
[Fri Apr 29 10:18:49 2016] R10: ffffffffc061164a R11: ffffea001281d600 R12: ffff880afa4e4828
[Fri Apr 29 10:18:49 2016] R13: ffff880afa4e4828 R14: 0000000000000000 R15: 0000000000000008
[Fri Apr 29 10:18:49 2016] FS: 00007f42d515f880(0000) GS:ffff88080f8c0000(0000) knlGS:0000000000000000
[Fri Apr 29 10:18:49 2016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Fri Apr 29 10:18:49 2016] CR2: 00000000000000f8 CR3: 0000000139a06000 CR4: 00000000000406e0
[Fri Apr 29 10:18:49 2016] Stack:
[Fri Apr 29 10:18:49 2016] ffff88018e11bb18 ffff880afa4e4828 ffff88075d649ec0 ffff880164bc8800
[Fri Apr 29 10:18:49 2016] ffff88018e11bb08 ffffffffc05d5d26 ffff88015e77eb80 0000000200000000
[Fri Apr 29 10:18:49 2016] ffff88018e11ba98 ffffffff811d1075 ffff88015e77eb80 0000000000000000
[Fri Apr 29 10:18:49 2016] Call Trace:
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d5d26>] xfs_alloc_fix_freelist+0x216/0x3c0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
[Fri Apr 29 10:18:49 2016] [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 10:18:49 2016] [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f2879>] mount_fs+0x39/0x160
[Fri Apr 29 10:18:49 2016] [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
[Fri Apr 29 10:18:49 2016] [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
[Fri Apr 29 10:18:50 2016] [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
[Fri Apr 29 10:18:50 2016] [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
[Fri Apr 29 10:18:50 2016] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[Fri Apr 29 10:18:50 2016] Code: 13 48 85 d2 75 eb e9 5f ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 56 49 89 f6 41 55 49 89 fd 41 54 53 <4c> 8b a6 f8 00 00 00 66 66 66 66 90 41 f6 44 24 78 04 74 4f 5b
[Fri Apr 29 10:18:50 2016] RIP [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
[Fri Apr 29 10:18:50 2016] RSP <ffff88018e11ba08>
[Fri Apr 29 10:18:50 2016] CR2: 00000000000000f8
[Fri Apr 29 10:18:50 2016] ---[ end trace ce3e7a80324237e2 ]---
root@ceph-store5:~#
Thanks to the magic of Ceph, I've so far managed to avoid data-loss and
serious downtime, but I suspect there's a timebomb here waiting for
anyone who upgrades from an older kernel to 4.2.0+.
- Please let me know is there's any further information I might be able to
- provide to track this down.
+ Please let me know is there's any further information I might be able to provide to track this down.
+ ---
+ AlsaDevices:
+ total 0
+ crw-rw---- 1 root audio 116, 1 Apr 29 01:29 seq
+ crw-rw---- 1 root audio 116, 33 Apr 29 01:29 timer
+ AplayDevices: Error: [Errno 2] No such file or directory
+ ApportVersion: 2.14.1-0ubuntu3.19
+ Architecture: amd64
+ ArecordDevices: Error: [Errno 2] No such file or directory
+ AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
+ CRDA: Error: [Errno 2] No such file or directory
+ DistroRelease: Ubuntu 14.04
+ MachineType: Dell Inc. PowerEdge R720xd
+ Package: linux (not installed)
+ PciMultimedia:
+
+ ProcEnviron:
+ TERM=xterm-256color
+ PATH=(custom, no user)
+ XDG_RUNTIME_DIR=<set>
+ LANG=en_US.UTF-8
+ SHELL=/bin/bash
+ ProcFB:
+
+ ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-35-generic root=UUID=f022f2e6-eeba-4e27-b286-6acd2c4bd1ea ro net.ifnames=1 biosdevname=0 console=tty1 console=ttyS1,115200 initrd=ubuntu/amd64/generic/trusty/release/di-initrd BOOTIF=01-b0-83-fe-e6-17-cc nomdmonddf nomdmonisw crashkernel=384M-:128M
+ ProcVersionSignature: Ubuntu 4.2.0-35.40~14.04.1-generic 4.2.8-ckt5
+ RelatedPackageVersions:
+ linux-restricted-modules-4.2.0-35-generic N/A
+ linux-backports-modules-4.2.0-35-generic N/A
+ linux-firmware 1.127.22
+ RfKill: Error: [Errno 2] No such file or directory
+ Tags: trusty uec-images
+ Uname: Linux 4.2.0-35-generic x86_64
+ UpgradeStatus: No upgrade log present (probably fresh install)
+ UserGroups:
+
+ _MarkForUpload: True
+ dmi.bios.date: 07/09/2014
+ dmi.bios.vendor: Dell Inc.
+ dmi.bios.version: 2.4.3
+ dmi.board.name: 0HJK12
+ dmi.board.vendor: Dell Inc.
+ dmi.board.version: A03
+ dmi.chassis.asset.tag: 707
+ dmi.chassis.type: 23
+ dmi.chassis.vendor: Dell Inc.
+ dmi.modalias: dmi:bvnDellInc.:bvr2.4.3:bd07/09/2014:svnDellInc.:pnPowerEdgeR720xd:pvr:rvnDellInc.:rn0HJK12:rvrA03:cvnDellInc.:ct23:cvr:
+ dmi.product.name: PowerEdge R720xd
+ dmi.sys.vendor: Dell Inc.
** Attachment added: "BootDmesg.txt"
https://bugs.launchpad.net/bugs/1576599/+attachment/4651224/+files/BootDmesg.txt
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1576599
Title:
XFS Oopsing 4.2.0-35-generic Kernel - Block out of range
Status in linux package in Ubuntu:
Confirmed
Bug description:
After updating the Kernel on two of our Trusty Ceph servers from
3.19.0-33-generic (vivid) to 4.2.0-35-generic (wily), both servers
have now Oopsed multiple times accessing some of their XFS
filesystems.
Once this error occurs, the XFS filesystem becomes permanently
inaccessible. Even after a reboot, any attempt to access the
filesystem in question either by mounting it, or attempting an
xfs_repair will trigger the same Oops again.
These filesystems did not immediately fail until several hours after
the upgrade. I suspect that this is related to some housekeeping being
triggered within the filesystem.
This is the call trace from the moment a filesystem first failed:
Apr 28 22:47:25 ceph-store5 kernel: [26692.804773] ------------[ cut here ]------------
Apr 28 22:47:25 ceph-store5 kernel: [26692.810046] WARNING: CPU: 8 PID: 5195 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
Apr 28 22:47:25 ceph-store5 kernel: [26692.824715] Modules linked in: bridge openvswitch xfs dcdbas ipmi_devintf intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev dm_multipath sb_edac scsi_dh edac_core input_leds mei_me lpc_ich mei ipmi_si ipmi_msghandler 8250_fintek wmi acpi_power_meter 8021q acpi_pad mac_hid garp mrp stp llc shpchp lp parport btrfs bcache raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx hid_generic xor usbhid hid raid6_pq bnx2x raid1 ptp mpt2sas raid0 pps_core multipath raid_class mdio scsi_transport_sas nvme libcrc32c linear
Apr 28 22:47:25 ceph-store5 kernel: [26692.891939] CPU: 8 PID: 5195 Comm: ceph-osd Tainted: G D W 4.2.0-35-generic #40~14.04.1-Ubuntu
Apr 28 22:47:25 ceph-store5 kernel: [26692.902689] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
Apr 28 22:47:25 ceph-store5 kernel: [26692.911303] 0000000000000000 ffff8800c7bf74a8 ffffffff817bcbf8 0000000000000000
Apr 28 22:47:25 ceph-store5 kernel: [26692.919677] ffffffffc0a7f300 ffff8800c7bf74e8 ffffffff81079b5a ffff8800c7bf7508
Apr 28 22:47:25 ceph-store5 kernel: [26692.928033] ffff88100a28ed80 0000000000000008 0000000000000000 ffff8800c7bf75f8
Apr 28 22:47:25 ceph-store5 kernel: [26692.936441] Call Trace:
Apr 28 22:47:25 ceph-store5 kernel: [26692.939208] [<ffffffff817bcbf8>] dump_stack+0x63/0x81
Apr 28 22:47:25 ceph-store5 kernel: [26692.944992] [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
Apr 28 22:47:25 ceph-store5 kernel: [26692.951752] [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
Apr 28 22:47:25 ceph-store5 kernel: [26692.958346] [<ffffffffc0a3e12e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.965221] [<ffffffffc0a3e1aa>] xfs_buf_get_map+0x2a/0x260 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.972194] [<ffffffffc0a0130a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.979763] [<ffffffffc0a69f26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.987412] [<ffffffffc0a17380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.994564] [<ffffffffc0a01d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.002302] [<ffffffffc0a3dd3f>] ? xfs_buf_trylock+0x1f/0x90 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.009379] [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.016229] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
Apr 28 22:47:25 ceph-store5 kernel: [26693.022814] [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.029669] [<ffffffffc0a020a2>] xfs_alloc_vextent+0x1d2/0x660 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.036915] [<ffffffffc0a12746>] xfs_bmap_btalloc+0x376/0x6b0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.044074] [<ffffffffc0a12aa4>] xfs_bmap_alloc+0x24/0x40 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.050833] [<ffffffffc0a13456>] xfs_bmapi_write+0x486/0x9a0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.057882] [<ffffffffc0a1d1ce>] xfs_da_grow_inode_int+0xae/0x2d0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.065403] [<ffffffff811d1bc2>] ? __kmalloc+0x1d2/0x280
Apr 28 22:47:25 ceph-store5 kernel: [26693.071509] [<ffffffffc0a5b080>] ? kmem_alloc+0x70/0xf0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.078078] [<ffffffffc0a1d427>] xfs_da_grow_inode+0x37/0x90 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.085129] [<ffffffffc0a08562>] xfs_attr_shortform_to_leaf+0xa2/0x2b0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.093151] [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.100199] [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.107251] [<ffffffffc0a047eb>] xfs_attr_set+0x31b/0x3e0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.114009] [<ffffffffc0a5acc2>] xfs_xattr_set+0x42/0x60 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.120647] [<ffffffff81212a42>] generic_setxattr+0x62/0x80
Apr 28 22:47:25 ceph-store5 kernel: [26693.127107] [<ffffffff812134e1>] __vfs_setxattr_noperm+0x61/0x1a0
Apr 28 22:47:25 ceph-store5 kernel: [26693.134045] [<ffffffff812136c7>] vfs_setxattr+0xa7/0xb0
Apr 28 22:47:25 ceph-store5 kernel: [26693.145904] [<ffffffff8121376d>] ? setxattr+0x9d/0x1b0
Apr 28 22:47:25 ceph-store5 kernel: [26693.151781] [<ffffffff812137fa>] setxattr+0x12a/0x1b0
Apr 28 22:47:25 ceph-store5 kernel: [26693.157626] [<ffffffffc0a73fcb>] ? xfs_acl_exists+0x4b/0x60 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.164599] [<ffffffffc0a74505>] ? posix_acl_access_exists+0x15/0x20 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.172451] [<ffffffffc0a5af52>] ? xfs_vn_listxattr+0xf2/0x160 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.179694] [<ffffffff811f1089>] ? __sb_start_write+0x49/0xf0
Apr 28 22:47:25 ceph-store5 kernel: [26693.186278] [<ffffffffc0a5ad30>] ? xfs_xattr_get+0x50/0x50 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.193122] [<ffffffff81213a90>] SyS_fsetxattr+0xa0/0xd0
Apr 28 22:47:25 ceph-store5 kernel: [26693.199193] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
Apr 28 22:47:25 ceph-store5 kernel: [26693.206471] ---[ end trace c0a9568b8830fc8b ]---
Apr 28 22:47:25 ceph-store5 kernel: [26693.211904] XFS (bcache7): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0
Apr 28 22:47:25 ceph-store5 kernel: [26693.221992] ------------[ cut here ]------------
I notice that xfs_da_grow_inode. Am I right in thinking that this was
a new feature add in the 4.2 kernel?
I expect that the history of these filesystems is important. These
would more have been formatted on a 3.16 kernel. Although I can't
inspect these failed filesystems, I can give information about the
other filesystems on the same servers, which I an 99% sure are
identical:
root@ceph-store5:~# xfs_info /var/lib/ceph/osd/ceph-35
meta-data=/dev/bcache6 isize=512 agcount=4, agsize=244188661 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1
data = bsize=4096 blocks=976754644, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=476930, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
==== Attempt to repair a filesystem (after a reboot) ====
root@ceph-store5:~# xfs_repair /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed. Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair. If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.
root@ceph-store5:~# mount -o noatime,inode64 /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c /var/lib/ceph/osd/ceph-39/
Killed
Dmesg output (dmesg -T):
[Fri Apr 29 10:18:45 2016] XFS (bcache2): Mounting V5 Filesystem
[Fri Apr 29 10:18:45 2016] XFS (bcache2): Starting recovery (logdev: internal)
[Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0
[Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------
[Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
[Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
[Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Not tainted 4.2.0-35-generic #40~14.04.1-Ubuntu
[Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 10:18:49 2016] 0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000
[Fri Apr 29 10:18:49 2016] ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928
[Fri Apr 29 10:18:49 2016] ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18
[Fri Apr 29 10:18:49 2016] Call Trace:
[Fri Apr 29 10:18:49 2016] [<ffffffff817bcbf8>] dump_stack+0x63/0x81
[Fri Apr 29 10:18:49 2016] [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
[Fri Apr 29 10:18:49 2016] [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
[Fri Apr 29 10:18:49 2016] [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc06121aa>] xfs_buf_get_map+0x2a/0x260 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d530a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
[Fri Apr 29 10:18:49 2016] [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 10:18:49 2016] [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f2879>] mount_fs+0x39/0x160
[Fri Apr 29 10:18:49 2016] [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
[Fri Apr 29 10:18:49 2016] [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
[Fri Apr 29 10:18:49 2016] [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
[Fri Apr 29 10:18:49 2016] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e0 ]---
[Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0
[Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------
[Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
[Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
[Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G W 4.2.0-35-generic #40~14.04.1-Ubuntu
[Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 10:18:49 2016] 0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000
[Fri Apr 29 10:18:49 2016] ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928
[Fri Apr 29 10:18:49 2016] ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18
[Fri Apr 29 10:18:49 2016] Call Trace:
[Fri Apr 29 10:18:49 2016] [<ffffffff817bcbf8>] dump_stack+0x63/0x81
[Fri Apr 29 10:18:49 2016] [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
[Fri Apr 29 10:18:49 2016] [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
[Fri Apr 29 10:18:49 2016] [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc061235d>] xfs_buf_get_map+0x1dd/0x260 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
[Fri Apr 29 10:18:49 2016] [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 10:18:49 2016] [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f2879>] mount_fs+0x39/0x160
[Fri Apr 29 10:18:49 2016] [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
[Fri Apr 29 10:18:49 2016] [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
[Fri Apr 29 10:18:49 2016] [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
[Fri Apr 29 10:18:49 2016] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e1 ]---
[Fri Apr 29 10:18:49 2016] BUG: unable to handle kernel NULL pointer dereference at 00000000000000f8
[Fri Apr 29 10:18:49 2016] IP: [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] PGD 564eec067 PUD 11408b067 PMD 0
[Fri Apr 29 10:18:49 2016] Oops: 0000 [#1] SMP
[Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
[Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G W 4.2.0-35-generic #40~14.04.1-Ubuntu
[Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 10:18:49 2016] task: ffff88080967ee00 ti: ffff88018e118000 task.ti: ffff88018e118000
[Fri Apr 29 10:18:49 2016] RIP: 0010:[<ffffffffc063e6f6>] [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] RSP: 0018:ffff88018e11ba08 EFLAGS: 00010292
[Fri Apr 29 10:18:49 2016] RAX: 0000000000000000 RBX: ffff88018e11bb18 RCX: 00000000003d2d52
[Fri Apr 29 10:18:49 2016] RDX: 00000000003d2d51 RSI: 0000000000000000 RDI: ffff880afa4e4828
[Fri Apr 29 10:18:49 2016] RBP: ffff88018e11ba28 R08: 000000000001a4f0 R09: ffff88080f8da4f0
[Fri Apr 29 10:18:49 2016] R10: ffffffffc061164a R11: ffffea001281d600 R12: ffff880afa4e4828
[Fri Apr 29 10:18:49 2016] R13: ffff880afa4e4828 R14: 0000000000000000 R15: 0000000000000008
[Fri Apr 29 10:18:49 2016] FS: 00007f42d515f880(0000) GS:ffff88080f8c0000(0000) knlGS:0000000000000000
[Fri Apr 29 10:18:49 2016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Fri Apr 29 10:18:49 2016] CR2: 00000000000000f8 CR3: 0000000139a06000 CR4: 00000000000406e0
[Fri Apr 29 10:18:49 2016] Stack:
[Fri Apr 29 10:18:49 2016] ffff88018e11bb18 ffff880afa4e4828 ffff88075d649ec0 ffff880164bc8800
[Fri Apr 29 10:18:49 2016] ffff88018e11bb08 ffffffffc05d5d26 ffff88015e77eb80 0000000200000000
[Fri Apr 29 10:18:49 2016] ffff88018e11ba98 ffffffff811d1075 ffff88015e77eb80 0000000000000000
[Fri Apr 29 10:18:49 2016] Call Trace:
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d5d26>] xfs_alloc_fix_freelist+0x216/0x3c0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
[Fri Apr 29 10:18:49 2016] [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 10:18:49 2016] [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f2879>] mount_fs+0x39/0x160
[Fri Apr 29 10:18:49 2016] [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
[Fri Apr 29 10:18:49 2016] [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
[Fri Apr 29 10:18:50 2016] [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
[Fri Apr 29 10:18:50 2016] [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
[Fri Apr 29 10:18:50 2016] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[Fri Apr 29 10:18:50 2016] Code: 13 48 85 d2 75 eb e9 5f ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 56 49 89 f6 41 55 49 89 fd 41 54 53 <4c> 8b a6 f8 00 00 00 66 66 66 66 90 41 f6 44 24 78 04 74 4f 5b
[Fri Apr 29 10:18:50 2016] RIP [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
[Fri Apr 29 10:18:50 2016] RSP <ffff88018e11ba08>
[Fri Apr 29 10:18:50 2016] CR2: 00000000000000f8
[Fri Apr 29 10:18:50 2016] ---[ end trace ce3e7a80324237e2 ]---
root@ceph-store5:~#
Thanks to the magic of Ceph, I've so far managed to avoid data-loss
and serious downtime, but I suspect there's a timebomb here waiting
for anyone who upgrades from an older kernel to 4.2.0+.
Please let me know is there's any further information I might be able to provide to track this down.
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Apr 29 01:29 seq
crw-rw---- 1 root audio 116, 33 Apr 29 01:29 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.19
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
MachineType: Dell Inc. PowerEdge R720xd
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-35-generic root=UUID=f022f2e6-eeba-4e27-b286-6acd2c4bd1ea ro net.ifnames=1 biosdevname=0 console=tty1 console=ttyS1,115200 initrd=ubuntu/amd64/generic/trusty/release/di-initrd BOOTIF=01-b0-83-fe-e6-17-cc nomdmonddf nomdmonisw crashkernel=384M-:128M
ProcVersionSignature: Ubuntu 4.2.0-35.40~14.04.1-generic 4.2.8-ckt5
RelatedPackageVersions:
linux-restricted-modules-4.2.0-35-generic N/A
linux-backports-modules-4.2.0-35-generic N/A
linux-firmware 1.127.22
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty uec-images
Uname: Linux 4.2.0-35-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 07/09/2014
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 2.4.3
dmi.board.name: 0HJK12
dmi.board.vendor: Dell Inc.
dmi.board.version: A03
dmi.chassis.asset.tag: 707
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr2.4.3:bd07/09/2014:svnDellInc.:pnPowerEdgeR720xd:pvr:rvnDellInc.:rn0HJK12:rvrA03:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R720xd
dmi.sys.vendor: Dell Inc.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1576599/+subscriptions
References
