kernel-packages team mailing list archive

[Launchpad Bug Tracker <1576599@xxxxxxxxxxxxxxxxxx>]

You have been subscribed to a public bug:

After updating the Kernel on two of our Trusty Ceph servers from
3.19.0-33-generic (vivid) to 4.2.0-35-generic (wily), both servers have
now Oopsed multiple times accessing some of their XFS filesystems.

Once this error occurs, the XFS filesystem becomes permanently
inaccessible. Even after a reboot, any attempt to access the filesystem
in question either by mounting it, or attempting an xfs_repair will
trigger the same Oops again.

These filesystems did not immediately fail until several hours after the
upgrade. I suspect that this is related to some housekeeping being
triggered within the filesystem.

This is the call trace from the moment a filesystem first failed:
Apr 28 22:47:25 ceph-store5 kernel: [26692.804773] ------------[ cut here ]------------
Apr 28 22:47:25 ceph-store5 kernel: [26692.810046] WARNING: CPU: 8 PID: 5195 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
Apr 28 22:47:25 ceph-store5 kernel: [26692.824715] Modules linked in: bridge openvswitch xfs dcdbas ipmi_devintf intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev dm_multipath sb_edac scsi_dh edac_core input_leds mei_me lpc_ich mei ipmi_si ipmi_msghandler 8250_fintek wmi acpi_power_meter 8021q acpi_pad mac_hid garp mrp stp llc shpchp lp parport btrfs bcache raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx hid_generic xor usbhid hid raid6_pq bnx2x raid1 ptp mpt2sas raid0 pps_core multipath raid_class mdio scsi_transport_sas nvme libcrc32c linear
Apr 28 22:47:25 ceph-store5 kernel: [26692.891939] CPU: 8 PID: 5195 Comm: ceph-osd Tainted: G      D W       4.2.0-35-generic #40~14.04.1-Ubuntu
Apr 28 22:47:25 ceph-store5 kernel: [26692.902689] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
Apr 28 22:47:25 ceph-store5 kernel: [26692.911303]  0000000000000000 ffff8800c7bf74a8 ffffffff817bcbf8 0000000000000000
Apr 28 22:47:25 ceph-store5 kernel: [26692.919677]  ffffffffc0a7f300 ffff8800c7bf74e8 ffffffff81079b5a ffff8800c7bf7508
Apr 28 22:47:25 ceph-store5 kernel: [26692.928033]  ffff88100a28ed80 0000000000000008 0000000000000000 ffff8800c7bf75f8
Apr 28 22:47:25 ceph-store5 kernel: [26692.936441] Call Trace:
Apr 28 22:47:25 ceph-store5 kernel: [26692.939208]  [<ffffffff817bcbf8>] dump_stack+0x63/0x81
Apr 28 22:47:25 ceph-store5 kernel: [26692.944992]  [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
Apr 28 22:47:25 ceph-store5 kernel: [26692.951752]  [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
Apr 28 22:47:25 ceph-store5 kernel: [26692.958346]  [<ffffffffc0a3e12e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.965221]  [<ffffffffc0a3e1aa>] xfs_buf_get_map+0x2a/0x260 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.972194]  [<ffffffffc0a0130a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.979763]  [<ffffffffc0a69f26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.987412]  [<ffffffffc0a17380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.994564]  [<ffffffffc0a01d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.002302]  [<ffffffffc0a3dd3f>] ? xfs_buf_trylock+0x1f/0x90 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.009379]  [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.016229]  [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
Apr 28 22:47:25 ceph-store5 kernel: [26693.022814]  [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.029669]  [<ffffffffc0a020a2>] xfs_alloc_vextent+0x1d2/0x660 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.036915]  [<ffffffffc0a12746>] xfs_bmap_btalloc+0x376/0x6b0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.044074]  [<ffffffffc0a12aa4>] xfs_bmap_alloc+0x24/0x40 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.050833]  [<ffffffffc0a13456>] xfs_bmapi_write+0x486/0x9a0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.057882]  [<ffffffffc0a1d1ce>] xfs_da_grow_inode_int+0xae/0x2d0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.065403]  [<ffffffff811d1bc2>] ? __kmalloc+0x1d2/0x280
Apr 28 22:47:25 ceph-store5 kernel: [26693.071509]  [<ffffffffc0a5b080>] ? kmem_alloc+0x70/0xf0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.078078]  [<ffffffffc0a1d427>] xfs_da_grow_inode+0x37/0x90 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.085129]  [<ffffffffc0a08562>] xfs_attr_shortform_to_leaf+0xa2/0x2b0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.093151]  [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.100199]  [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.107251]  [<ffffffffc0a047eb>] xfs_attr_set+0x31b/0x3e0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.114009]  [<ffffffffc0a5acc2>] xfs_xattr_set+0x42/0x60 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.120647]  [<ffffffff81212a42>] generic_setxattr+0x62/0x80
Apr 28 22:47:25 ceph-store5 kernel: [26693.127107]  [<ffffffff812134e1>] __vfs_setxattr_noperm+0x61/0x1a0
Apr 28 22:47:25 ceph-store5 kernel: [26693.134045]  [<ffffffff812136c7>] vfs_setxattr+0xa7/0xb0
Apr 28 22:47:25 ceph-store5 kernel: [26693.145904]  [<ffffffff8121376d>] ? setxattr+0x9d/0x1b0
Apr 28 22:47:25 ceph-store5 kernel: [26693.151781]  [<ffffffff812137fa>] setxattr+0x12a/0x1b0
Apr 28 22:47:25 ceph-store5 kernel: [26693.157626]  [<ffffffffc0a73fcb>] ? xfs_acl_exists+0x4b/0x60 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.164599]  [<ffffffffc0a74505>] ? posix_acl_access_exists+0x15/0x20 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.172451]  [<ffffffffc0a5af52>] ? xfs_vn_listxattr+0xf2/0x160 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.179694]  [<ffffffff811f1089>] ? __sb_start_write+0x49/0xf0
Apr 28 22:47:25 ceph-store5 kernel: [26693.186278]  [<ffffffffc0a5ad30>] ? xfs_xattr_get+0x50/0x50 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.193122]  [<ffffffff81213a90>] SyS_fsetxattr+0xa0/0xd0
Apr 28 22:47:25 ceph-store5 kernel: [26693.199193]  [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
Apr 28 22:47:25 ceph-store5 kernel: [26693.206471] ---[ end trace c0a9568b8830fc8b ]---
Apr 28 22:47:25 ceph-store5 kernel: [26693.211904] XFS (bcache7): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0 
Apr 28 22:47:25 ceph-store5 kernel: [26693.221992] ------------[ cut here ]------------

I notice that xfs_da_grow_inode. Am I right in thinking that this was a
new feature add in the 4.2 kernel?

I expect that the history of these filesystems is important. These would
more have been formatted on a 3.16 kernel. Although I can't inspect
these failed filesystems, I can give information about the other
filesystems on the same servers, which I an 99% sure are identical:

root@ceph-store5:~# xfs_info /var/lib/ceph/osd/ceph-35
meta-data=/dev/bcache6           isize=512    agcount=4, agsize=244188661 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1
data     =                       bsize=4096   blocks=976754644, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=476930, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

 ==== Attempt to repair a filesystem (after a reboot) ====

root@ceph-store5:~# xfs_repair /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

root@ceph-store5:~# mount -o noatime,inode64 /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c /var/lib/ceph/osd/ceph-39/
Killed

Dmesg output (dmesg -T):

[Fri Apr 29 10:18:45 2016] XFS (bcache2): Mounting V5 Filesystem
[Fri Apr 29 10:18:45 2016] XFS (bcache2): Starting recovery (logdev: internal)
[Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0 
[Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------
[Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
[Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
[Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Not tainted 4.2.0-35-generic #40~14.04.1-Ubuntu
[Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 10:18:49 2016]  0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000
[Fri Apr 29 10:18:49 2016]  ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928
[Fri Apr 29 10:18:49 2016]  ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18
[Fri Apr 29 10:18:49 2016] Call Trace:
[Fri Apr 29 10:18:49 2016]  [<ffffffff817bcbf8>] dump_stack+0x63/0x81
[Fri Apr 29 10:18:49 2016]  [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
[Fri Apr 29 10:18:49 2016]  [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
[Fri Apr 29 10:18:49 2016]  [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc06121aa>] xfs_buf_get_map+0x2a/0x260 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc05d530a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
[Fri Apr 29 10:18:49 2016]  [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 10:18:49 2016]  [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
[Fri Apr 29 10:18:49 2016]  [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffff811f2879>] mount_fs+0x39/0x160
[Fri Apr 29 10:18:49 2016]  [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
[Fri Apr 29 10:18:49 2016]  [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
[Fri Apr 29 10:18:49 2016]  [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
[Fri Apr 29 10:18:49 2016]  [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
[Fri Apr 29 10:18:49 2016]  [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
[Fri Apr 29 10:18:49 2016]  [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e0 ]---
[Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0 
[Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------
[Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
[Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
[Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G        W       4.2.0-35-generic #40~14.04.1-Ubuntu
[Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 10:18:49 2016]  0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000
[Fri Apr 29 10:18:49 2016]  ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928
[Fri Apr 29 10:18:49 2016]  ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18
[Fri Apr 29 10:18:49 2016] Call Trace:
[Fri Apr 29 10:18:49 2016]  [<ffffffff817bcbf8>] dump_stack+0x63/0x81
[Fri Apr 29 10:18:49 2016]  [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
[Fri Apr 29 10:18:49 2016]  [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
[Fri Apr 29 10:18:49 2016]  [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc061235d>] xfs_buf_get_map+0x1dd/0x260 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
[Fri Apr 29 10:18:49 2016]  [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 10:18:49 2016]  [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
[Fri Apr 29 10:18:49 2016]  [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffff811f2879>] mount_fs+0x39/0x160
[Fri Apr 29 10:18:49 2016]  [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
[Fri Apr 29 10:18:49 2016]  [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
[Fri Apr 29 10:18:49 2016]  [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
[Fri Apr 29 10:18:49 2016]  [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
[Fri Apr 29 10:18:49 2016]  [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
[Fri Apr 29 10:18:49 2016]  [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e1 ]---
[Fri Apr 29 10:18:49 2016] BUG: unable to handle kernel NULL pointer dereference at 00000000000000f8
[Fri Apr 29 10:18:49 2016] IP: [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] PGD 564eec067 PUD 11408b067 PMD 0 
[Fri Apr 29 10:18:49 2016] Oops: 0000 [#1] SMP 
[Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
[Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G        W       4.2.0-35-generic #40~14.04.1-Ubuntu
[Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 10:18:49 2016] task: ffff88080967ee00 ti: ffff88018e118000 task.ti: ffff88018e118000
[Fri Apr 29 10:18:49 2016] RIP: 0010:[<ffffffffc063e6f6>]  [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] RSP: 0018:ffff88018e11ba08  EFLAGS: 00010292
[Fri Apr 29 10:18:49 2016] RAX: 0000000000000000 RBX: ffff88018e11bb18 RCX: 00000000003d2d52
[Fri Apr 29 10:18:49 2016] RDX: 00000000003d2d51 RSI: 0000000000000000 RDI: ffff880afa4e4828
[Fri Apr 29 10:18:49 2016] RBP: ffff88018e11ba28 R08: 000000000001a4f0 R09: ffff88080f8da4f0
[Fri Apr 29 10:18:49 2016] R10: ffffffffc061164a R11: ffffea001281d600 R12: ffff880afa4e4828
[Fri Apr 29 10:18:49 2016] R13: ffff880afa4e4828 R14: 0000000000000000 R15: 0000000000000008
[Fri Apr 29 10:18:49 2016] FS:  00007f42d515f880(0000) GS:ffff88080f8c0000(0000) knlGS:0000000000000000
[Fri Apr 29 10:18:49 2016] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Fri Apr 29 10:18:49 2016] CR2: 00000000000000f8 CR3: 0000000139a06000 CR4: 00000000000406e0
[Fri Apr 29 10:18:49 2016] Stack:
[Fri Apr 29 10:18:49 2016]  ffff88018e11bb18 ffff880afa4e4828 ffff88075d649ec0 ffff880164bc8800
[Fri Apr 29 10:18:49 2016]  ffff88018e11bb08 ffffffffc05d5d26 ffff88015e77eb80 0000000200000000
[Fri Apr 29 10:18:49 2016]  ffff88018e11ba98 ffffffff811d1075 ffff88015e77eb80 0000000000000000
[Fri Apr 29 10:18:49 2016] Call Trace:
[Fri Apr 29 10:18:49 2016]  [<ffffffffc05d5d26>] xfs_alloc_fix_freelist+0x216/0x3c0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
[Fri Apr 29 10:18:49 2016]  [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 10:18:49 2016]  [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
[Fri Apr 29 10:18:49 2016]  [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
[Fri Apr 29 10:18:49 2016]  [<ffffffff811f2879>] mount_fs+0x39/0x160
[Fri Apr 29 10:18:49 2016]  [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
[Fri Apr 29 10:18:49 2016]  [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
[Fri Apr 29 10:18:49 2016]  [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
[Fri Apr 29 10:18:50 2016]  [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
[Fri Apr 29 10:18:50 2016]  [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
[Fri Apr 29 10:18:50 2016]  [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[Fri Apr 29 10:18:50 2016] Code: 13 48 85 d2 75 eb e9 5f ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 56 49 89 f6 41 55 49 89 fd 41 54 53 <4c> 8b a6 f8 00 00 00 66 66 66 66 90 41 f6 44 24 78 04 74 4f 5b 
[Fri Apr 29 10:18:50 2016] RIP  [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
[Fri Apr 29 10:18:50 2016]  RSP <ffff88018e11ba08>
[Fri Apr 29 10:18:50 2016] CR2: 00000000000000f8
[Fri Apr 29 10:18:50 2016] ---[ end trace ce3e7a80324237e2 ]---
root@ceph-store5:~# 

Thanks to the magic of Ceph, I've so far managed to avoid data-loss and
serious downtime, but I suspect there's a timebomb here waiting for
anyone who upgrades from an older kernel to 4.2.0+.

Please let me know is there's any further information I might be able to
provide to track this down.

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: New

-- 
XFS Oopsing 4.2.0-35-generic Kernel - Block out of range
https://bugs.launchpad.net/bugs/1576599
You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.