kernel-packages team mailing list archive

Thread
Date
[Bug 1576599] Missing required logs.

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Brad Figg <brad.figg@xxxxxxxxxxxxx>
Date: Fri, 29 Apr 2016 09:30:59 -0000
Reply-to: Bug 1576599 <1576599@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This bug is missing log files that will aid in diagnosing the problem.
>From a terminal window please run:

apport-collect 1576599

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable
to run this command, please add a comment stating that fact and change
the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the
Ubuntu Kernel Team.

** Changed in: linux (Ubuntu)
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1576599

Title:
  XFS Oopsing 4.2.0-35-generic Kernel - Block out of range

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  After updating the Kernel on two of our Trusty Ceph servers from
  3.19.0-33-generic (vivid) to 4.2.0-35-generic (wily), both servers
  have now Oopsed multiple times accessing some of their XFS
  filesystems.

  Once this error occurs, the XFS filesystem becomes permanently
  inaccessible. Even after a reboot, any attempt to access the
  filesystem in question either by mounting it, or attempting an
  xfs_repair will trigger the same Oops again.

  These filesystems did not immediately fail until several hours after
  the upgrade. I suspect that this is related to some housekeeping being
  triggered within the filesystem.

  This is the call trace from the moment a filesystem first failed:
  Apr 28 22:47:25 ceph-store5 kernel: [26692.804773] ------------[ cut here ]------------
  Apr 28 22:47:25 ceph-store5 kernel: [26692.810046] WARNING: CPU: 8 PID: 5195 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
  Apr 28 22:47:25 ceph-store5 kernel: [26692.824715] Modules linked in: bridge openvswitch xfs dcdbas ipmi_devintf intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev dm_multipath sb_edac scsi_dh edac_core input_leds mei_me lpc_ich mei ipmi_si ipmi_msghandler 8250_fintek wmi acpi_power_meter 8021q acpi_pad mac_hid garp mrp stp llc shpchp lp parport btrfs bcache raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx hid_generic xor usbhid hid raid6_pq bnx2x raid1 ptp mpt2sas raid0 pps_core multipath raid_class mdio scsi_transport_sas nvme libcrc32c linear
  Apr 28 22:47:25 ceph-store5 kernel: [26692.891939] CPU: 8 PID: 5195 Comm: ceph-osd Tainted: G      D W       4.2.0-35-generic #40~14.04.1-Ubuntu
  Apr 28 22:47:25 ceph-store5 kernel: [26692.902689] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
  Apr 28 22:47:25 ceph-store5 kernel: [26692.911303]  0000000000000000 ffff8800c7bf74a8 ffffffff817bcbf8 0000000000000000
  Apr 28 22:47:25 ceph-store5 kernel: [26692.919677]  ffffffffc0a7f300 ffff8800c7bf74e8 ffffffff81079b5a ffff8800c7bf7508
  Apr 28 22:47:25 ceph-store5 kernel: [26692.928033]  ffff88100a28ed80 0000000000000008 0000000000000000 ffff8800c7bf75f8
  Apr 28 22:47:25 ceph-store5 kernel: [26692.936441] Call Trace:
  Apr 28 22:47:25 ceph-store5 kernel: [26692.939208]  [<ffffffff817bcbf8>] dump_stack+0x63/0x81
  Apr 28 22:47:25 ceph-store5 kernel: [26692.944992]  [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
  Apr 28 22:47:25 ceph-store5 kernel: [26692.951752]  [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
  Apr 28 22:47:25 ceph-store5 kernel: [26692.958346]  [<ffffffffc0a3e12e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26692.965221]  [<ffffffffc0a3e1aa>] xfs_buf_get_map+0x2a/0x260 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26692.972194]  [<ffffffffc0a0130a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26692.979763]  [<ffffffffc0a69f26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26692.987412]  [<ffffffffc0a17380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26692.994564]  [<ffffffffc0a01d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.002302]  [<ffffffffc0a3dd3f>] ? xfs_buf_trylock+0x1f/0x90 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.009379]  [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.016229]  [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
  Apr 28 22:47:25 ceph-store5 kernel: [26693.022814]  [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.029669]  [<ffffffffc0a020a2>] xfs_alloc_vextent+0x1d2/0x660 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.036915]  [<ffffffffc0a12746>] xfs_bmap_btalloc+0x376/0x6b0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.044074]  [<ffffffffc0a12aa4>] xfs_bmap_alloc+0x24/0x40 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.050833]  [<ffffffffc0a13456>] xfs_bmapi_write+0x486/0x9a0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.057882]  [<ffffffffc0a1d1ce>] xfs_da_grow_inode_int+0xae/0x2d0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.065403]  [<ffffffff811d1bc2>] ? __kmalloc+0x1d2/0x280
  Apr 28 22:47:25 ceph-store5 kernel: [26693.071509]  [<ffffffffc0a5b080>] ? kmem_alloc+0x70/0xf0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.078078]  [<ffffffffc0a1d427>] xfs_da_grow_inode+0x37/0x90 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.085129]  [<ffffffffc0a08562>] xfs_attr_shortform_to_leaf+0xa2/0x2b0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.093151]  [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.100199]  [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.107251]  [<ffffffffc0a047eb>] xfs_attr_set+0x31b/0x3e0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.114009]  [<ffffffffc0a5acc2>] xfs_xattr_set+0x42/0x60 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.120647]  [<ffffffff81212a42>] generic_setxattr+0x62/0x80
  Apr 28 22:47:25 ceph-store5 kernel: [26693.127107]  [<ffffffff812134e1>] __vfs_setxattr_noperm+0x61/0x1a0
  Apr 28 22:47:25 ceph-store5 kernel: [26693.134045]  [<ffffffff812136c7>] vfs_setxattr+0xa7/0xb0
  Apr 28 22:47:25 ceph-store5 kernel: [26693.145904]  [<ffffffff8121376d>] ? setxattr+0x9d/0x1b0
  Apr 28 22:47:25 ceph-store5 kernel: [26693.151781]  [<ffffffff812137fa>] setxattr+0x12a/0x1b0
  Apr 28 22:47:25 ceph-store5 kernel: [26693.157626]  [<ffffffffc0a73fcb>] ? xfs_acl_exists+0x4b/0x60 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.164599]  [<ffffffffc0a74505>] ? posix_acl_access_exists+0x15/0x20 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.172451]  [<ffffffffc0a5af52>] ? xfs_vn_listxattr+0xf2/0x160 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.179694]  [<ffffffff811f1089>] ? __sb_start_write+0x49/0xf0
  Apr 28 22:47:25 ceph-store5 kernel: [26693.186278]  [<ffffffffc0a5ad30>] ? xfs_xattr_get+0x50/0x50 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.193122]  [<ffffffff81213a90>] SyS_fsetxattr+0xa0/0xd0
  Apr 28 22:47:25 ceph-store5 kernel: [26693.199193]  [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
  Apr 28 22:47:25 ceph-store5 kernel: [26693.206471] ---[ end trace c0a9568b8830fc8b ]---
  Apr 28 22:47:25 ceph-store5 kernel: [26693.211904] XFS (bcache7): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0 
  Apr 28 22:47:25 ceph-store5 kernel: [26693.221992] ------------[ cut here ]------------

  I notice that xfs_da_grow_inode. Am I right in thinking that this was
  a new feature add in the 4.2 kernel?

  I expect that the history of these filesystems is important. These
  would more have been formatted on a 3.16 kernel. Although I can't
  inspect these failed filesystems, I can give information about the
  other filesystems on the same servers, which I an 99% sure are
  identical:

  root@ceph-store5:~# xfs_info /var/lib/ceph/osd/ceph-35
  meta-data=/dev/bcache6           isize=512    agcount=4, agsize=244188661 blks
           =                       sectsz=512   attr=2, projid32bit=1
           =                       crc=1        finobt=1
  data     =                       bsize=4096   blocks=976754644, imaxpct=5
           =                       sunit=0      swidth=0 blks
  naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
  log      =internal               bsize=4096   blocks=476930, version=2
           =                       sectsz=512   sunit=0 blks, lazy-count=1
  realtime =none                   extsz=4096   blocks=0, rtextents=0

   ==== Attempt to repair a filesystem (after a reboot) ====

  root@ceph-store5:~# xfs_repair /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c
  Phase 1 - find and verify superblock...
  Phase 2 - using internal log
          - zero log...
  ERROR: The filesystem has valuable metadata changes in a log which needs to
  be replayed.  Mount the filesystem to replay the log, and unmount it before
  re-running xfs_repair.  If you are unable to mount the filesystem, then use
  the -L option to destroy the log and attempt a repair.
  Note that destroying the log may cause corruption -- please attempt a mount
  of the filesystem before doing this.

  root@ceph-store5:~# mount -o noatime,inode64 /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c /var/lib/ceph/osd/ceph-39/
  Killed

  Dmesg output (dmesg -T):

  [Fri Apr 29 10:18:45 2016] XFS (bcache2): Mounting V5 Filesystem
  [Fri Apr 29 10:18:45 2016] XFS (bcache2): Starting recovery (logdev: internal)
  [Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0 
  [Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------
  [Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
  [Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
  [Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Not tainted 4.2.0-35-generic #40~14.04.1-Ubuntu
  [Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
  [Fri Apr 29 10:18:49 2016]  0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000
  [Fri Apr 29 10:18:49 2016]  ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928
  [Fri Apr 29 10:18:49 2016]  ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18
  [Fri Apr 29 10:18:49 2016] Call Trace:
  [Fri Apr 29 10:18:49 2016]  [<ffffffff817bcbf8>] dump_stack+0x63/0x81
  [Fri Apr 29 10:18:49 2016]  [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc06121aa>] xfs_buf_get_map+0x2a/0x260 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d530a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811f2879>] mount_fs+0x39/0x160
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
  [Fri Apr 29 10:18:49 2016]  [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
  [Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e0 ]---
  [Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0 
  [Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------
  [Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
  [Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
  [Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G        W       4.2.0-35-generic #40~14.04.1-Ubuntu
  [Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
  [Fri Apr 29 10:18:49 2016]  0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000
  [Fri Apr 29 10:18:49 2016]  ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928
  [Fri Apr 29 10:18:49 2016]  ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18
  [Fri Apr 29 10:18:49 2016] Call Trace:
  [Fri Apr 29 10:18:49 2016]  [<ffffffff817bcbf8>] dump_stack+0x63/0x81
  [Fri Apr 29 10:18:49 2016]  [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc061235d>] xfs_buf_get_map+0x1dd/0x260 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811f2879>] mount_fs+0x39/0x160
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
  [Fri Apr 29 10:18:49 2016]  [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
  [Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e1 ]---
  [Fri Apr 29 10:18:49 2016] BUG: unable to handle kernel NULL pointer dereference at 00000000000000f8
  [Fri Apr 29 10:18:49 2016] IP: [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
  [Fri Apr 29 10:18:49 2016] PGD 564eec067 PUD 11408b067 PMD 0 
  [Fri Apr 29 10:18:49 2016] Oops: 0000 [#1] SMP 
  [Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
  [Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G        W       4.2.0-35-generic #40~14.04.1-Ubuntu
  [Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
  [Fri Apr 29 10:18:49 2016] task: ffff88080967ee00 ti: ffff88018e118000 task.ti: ffff88018e118000
  [Fri Apr 29 10:18:49 2016] RIP: 0010:[<ffffffffc063e6f6>]  [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
  [Fri Apr 29 10:18:49 2016] RSP: 0018:ffff88018e11ba08  EFLAGS: 00010292
  [Fri Apr 29 10:18:49 2016] RAX: 0000000000000000 RBX: ffff88018e11bb18 RCX: 00000000003d2d52
  [Fri Apr 29 10:18:49 2016] RDX: 00000000003d2d51 RSI: 0000000000000000 RDI: ffff880afa4e4828
  [Fri Apr 29 10:18:49 2016] RBP: ffff88018e11ba28 R08: 000000000001a4f0 R09: ffff88080f8da4f0
  [Fri Apr 29 10:18:49 2016] R10: ffffffffc061164a R11: ffffea001281d600 R12: ffff880afa4e4828
  [Fri Apr 29 10:18:49 2016] R13: ffff880afa4e4828 R14: 0000000000000000 R15: 0000000000000008
  [Fri Apr 29 10:18:49 2016] FS:  00007f42d515f880(0000) GS:ffff88080f8c0000(0000) knlGS:0000000000000000
  [Fri Apr 29 10:18:49 2016] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [Fri Apr 29 10:18:49 2016] CR2: 00000000000000f8 CR3: 0000000139a06000 CR4: 00000000000406e0
  [Fri Apr 29 10:18:49 2016] Stack:
  [Fri Apr 29 10:18:49 2016]  ffff88018e11bb18 ffff880afa4e4828 ffff88075d649ec0 ffff880164bc8800
  [Fri Apr 29 10:18:49 2016]  ffff88018e11bb08 ffffffffc05d5d26 ffff88015e77eb80 0000000200000000
  [Fri Apr 29 10:18:49 2016]  ffff88018e11ba98 ffffffff811d1075 ffff88015e77eb80 0000000000000000
  [Fri Apr 29 10:18:49 2016] Call Trace:
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d5d26>] xfs_alloc_fix_freelist+0x216/0x3c0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811f2879>] mount_fs+0x39/0x160
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
  [Fri Apr 29 10:18:50 2016]  [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
  [Fri Apr 29 10:18:50 2016]  [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
  [Fri Apr 29 10:18:50 2016]  [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
  [Fri Apr 29 10:18:50 2016] Code: 13 48 85 d2 75 eb e9 5f ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 56 49 89 f6 41 55 49 89 fd 41 54 53 <4c> 8b a6 f8 00 00 00 66 66 66 66 90 41 f6 44 24 78 04 74 4f 5b 
  [Fri Apr 29 10:18:50 2016] RIP  [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
  [Fri Apr 29 10:18:50 2016]  RSP <ffff88018e11ba08>
  [Fri Apr 29 10:18:50 2016] CR2: 00000000000000f8
  [Fri Apr 29 10:18:50 2016] ---[ end trace ce3e7a80324237e2 ]---
  root@ceph-store5:~# 

  Thanks to the magic of Ceph, I've so far managed to avoid data-loss
  and serious downtime, but I suspect there's a timebomb here waiting
  for anyone who upgrades from an older kernel to 4.2.0+.

  Please let me know is there's any further information I might be able
  to provide to track this down.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1576599/+subscriptions
References

[Bug 1576599] [NEW] XFS Oopsing 4.2.0-35-generic Kernel - Block out of range
From: John McEleney, 2016-04-29