kernel-packages team mailing list archive

Thread
Date
[Bug 1576599] ProcInterrupts.txt

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: John McEleney <launchpad@xxxxxxxxxxxxxxxxxxxxx>
Date: Fri, 29 Apr 2016 09:53:52 -0000
Reply-to: Bug 1576599 <1576599@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
apport information

** Attachment added: "ProcInterrupts.txt"
   https://bugs.launchpad.net/bugs/1576599/+attachment/4651230/+files/ProcInterrupts.txt

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1576599

Title:
  XFS Oopsing 4.2.0-35-generic Kernel - Block out of range

Status in linux package in Ubuntu:
  Confirmed

Bug description:
  After updating the Kernel on two of our Trusty Ceph servers from
  3.19.0-33-generic (vivid) to 4.2.0-35-generic (wily), both servers
  have now Oopsed multiple times accessing some of their XFS
  filesystems.

  Once this error occurs, the XFS filesystem becomes permanently
  inaccessible. Even after a reboot, any attempt to access the
  filesystem in question either by mounting it, or attempting an
  xfs_repair will trigger the same Oops again.

  These filesystems did not immediately fail until several hours after
  the upgrade. I suspect that this is related to some housekeeping being
  triggered within the filesystem.

  This is the call trace from the moment a filesystem first failed:
  Apr 28 22:47:25 ceph-store5 kernel: [26692.804773] ------------[ cut here ]------------
  Apr 28 22:47:25 ceph-store5 kernel: [26692.810046] WARNING: CPU: 8 PID: 5195 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
  Apr 28 22:47:25 ceph-store5 kernel: [26692.824715] Modules linked in: bridge openvswitch xfs dcdbas ipmi_devintf intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev dm_multipath sb_edac scsi_dh edac_core input_leds mei_me lpc_ich mei ipmi_si ipmi_msghandler 8250_fintek wmi acpi_power_meter 8021q acpi_pad mac_hid garp mrp stp llc shpchp lp parport btrfs bcache raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx hid_generic xor usbhid hid raid6_pq bnx2x raid1 ptp mpt2sas raid0 pps_core multipath raid_class mdio scsi_transport_sas nvme libcrc32c linear
  Apr 28 22:47:25 ceph-store5 kernel: [26692.891939] CPU: 8 PID: 5195 Comm: ceph-osd Tainted: G      D W       4.2.0-35-generic #40~14.04.1-Ubuntu
  Apr 28 22:47:25 ceph-store5 kernel: [26692.902689] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
  Apr 28 22:47:25 ceph-store5 kernel: [26692.911303]  0000000000000000 ffff8800c7bf74a8 ffffffff817bcbf8 0000000000000000
  Apr 28 22:47:25 ceph-store5 kernel: [26692.919677]  ffffffffc0a7f300 ffff8800c7bf74e8 ffffffff81079b5a ffff8800c7bf7508
  Apr 28 22:47:25 ceph-store5 kernel: [26692.928033]  ffff88100a28ed80 0000000000000008 0000000000000000 ffff8800c7bf75f8
  Apr 28 22:47:25 ceph-store5 kernel: [26692.936441] Call Trace:
  Apr 28 22:47:25 ceph-store5 kernel: [26692.939208]  [<ffffffff817bcbf8>] dump_stack+0x63/0x81
  Apr 28 22:47:25 ceph-store5 kernel: [26692.944992]  [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
  Apr 28 22:47:25 ceph-store5 kernel: [26692.951752]  [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
  Apr 28 22:47:25 ceph-store5 kernel: [26692.958346]  [<ffffffffc0a3e12e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26692.965221]  [<ffffffffc0a3e1aa>] xfs_buf_get_map+0x2a/0x260 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26692.972194]  [<ffffffffc0a0130a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26692.979763]  [<ffffffffc0a69f26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26692.987412]  [<ffffffffc0a17380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26692.994564]  [<ffffffffc0a01d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.002302]  [<ffffffffc0a3dd3f>] ? xfs_buf_trylock+0x1f/0x90 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.009379]  [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.016229]  [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
  Apr 28 22:47:25 ceph-store5 kernel: [26693.022814]  [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.029669]  [<ffffffffc0a020a2>] xfs_alloc_vextent+0x1d2/0x660 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.036915]  [<ffffffffc0a12746>] xfs_bmap_btalloc+0x376/0x6b0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.044074]  [<ffffffffc0a12aa4>] xfs_bmap_alloc+0x24/0x40 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.050833]  [<ffffffffc0a13456>] xfs_bmapi_write+0x486/0x9a0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.057882]  [<ffffffffc0a1d1ce>] xfs_da_grow_inode_int+0xae/0x2d0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.065403]  [<ffffffff811d1bc2>] ? __kmalloc+0x1d2/0x280
  Apr 28 22:47:25 ceph-store5 kernel: [26693.071509]  [<ffffffffc0a5b080>] ? kmem_alloc+0x70/0xf0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.078078]  [<ffffffffc0a1d427>] xfs_da_grow_inode+0x37/0x90 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.085129]  [<ffffffffc0a08562>] xfs_attr_shortform_to_leaf+0xa2/0x2b0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.093151]  [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.100199]  [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.107251]  [<ffffffffc0a047eb>] xfs_attr_set+0x31b/0x3e0 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.114009]  [<ffffffffc0a5acc2>] xfs_xattr_set+0x42/0x60 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.120647]  [<ffffffff81212a42>] generic_setxattr+0x62/0x80
  Apr 28 22:47:25 ceph-store5 kernel: [26693.127107]  [<ffffffff812134e1>] __vfs_setxattr_noperm+0x61/0x1a0
  Apr 28 22:47:25 ceph-store5 kernel: [26693.134045]  [<ffffffff812136c7>] vfs_setxattr+0xa7/0xb0
  Apr 28 22:47:25 ceph-store5 kernel: [26693.145904]  [<ffffffff8121376d>] ? setxattr+0x9d/0x1b0
  Apr 28 22:47:25 ceph-store5 kernel: [26693.151781]  [<ffffffff812137fa>] setxattr+0x12a/0x1b0
  Apr 28 22:47:25 ceph-store5 kernel: [26693.157626]  [<ffffffffc0a73fcb>] ? xfs_acl_exists+0x4b/0x60 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.164599]  [<ffffffffc0a74505>] ? posix_acl_access_exists+0x15/0x20 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.172451]  [<ffffffffc0a5af52>] ? xfs_vn_listxattr+0xf2/0x160 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.179694]  [<ffffffff811f1089>] ? __sb_start_write+0x49/0xf0
  Apr 28 22:47:25 ceph-store5 kernel: [26693.186278]  [<ffffffffc0a5ad30>] ? xfs_xattr_get+0x50/0x50 [xfs]
  Apr 28 22:47:25 ceph-store5 kernel: [26693.193122]  [<ffffffff81213a90>] SyS_fsetxattr+0xa0/0xd0
  Apr 28 22:47:25 ceph-store5 kernel: [26693.199193]  [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
  Apr 28 22:47:25 ceph-store5 kernel: [26693.206471] ---[ end trace c0a9568b8830fc8b ]---
  Apr 28 22:47:25 ceph-store5 kernel: [26693.211904] XFS (bcache7): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0 
  Apr 28 22:47:25 ceph-store5 kernel: [26693.221992] ------------[ cut here ]------------

  I notice that xfs_da_grow_inode. Am I right in thinking that this was
  a new feature add in the 4.2 kernel?

  I expect that the history of these filesystems is important. These
  would more have been formatted on a 3.16 kernel. Although I can't
  inspect these failed filesystems, I can give information about the
  other filesystems on the same servers, which I an 99% sure are
  identical:

  root@ceph-store5:~# xfs_info /var/lib/ceph/osd/ceph-35
  meta-data=/dev/bcache6           isize=512    agcount=4, agsize=244188661 blks
           =                       sectsz=512   attr=2, projid32bit=1
           =                       crc=1        finobt=1
  data     =                       bsize=4096   blocks=976754644, imaxpct=5
           =                       sunit=0      swidth=0 blks
  naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
  log      =internal               bsize=4096   blocks=476930, version=2
           =                       sectsz=512   sunit=0 blks, lazy-count=1
  realtime =none                   extsz=4096   blocks=0, rtextents=0

   ==== Attempt to repair a filesystem (after a reboot) ====

  root@ceph-store5:~# xfs_repair /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c
  Phase 1 - find and verify superblock...
  Phase 2 - using internal log
          - zero log...
  ERROR: The filesystem has valuable metadata changes in a log which needs to
  be replayed.  Mount the filesystem to replay the log, and unmount it before
  re-running xfs_repair.  If you are unable to mount the filesystem, then use
  the -L option to destroy the log and attempt a repair.
  Note that destroying the log may cause corruption -- please attempt a mount
  of the filesystem before doing this.

  root@ceph-store5:~# mount -o noatime,inode64 /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c /var/lib/ceph/osd/ceph-39/
  Killed

  Dmesg output (dmesg -T):

  [Fri Apr 29 10:18:45 2016] XFS (bcache2): Mounting V5 Filesystem
  [Fri Apr 29 10:18:45 2016] XFS (bcache2): Starting recovery (logdev: internal)
  [Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0 
  [Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------
  [Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
  [Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
  [Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Not tainted 4.2.0-35-generic #40~14.04.1-Ubuntu
  [Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
  [Fri Apr 29 10:18:49 2016]  0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000
  [Fri Apr 29 10:18:49 2016]  ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928
  [Fri Apr 29 10:18:49 2016]  ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18
  [Fri Apr 29 10:18:49 2016] Call Trace:
  [Fri Apr 29 10:18:49 2016]  [<ffffffff817bcbf8>] dump_stack+0x63/0x81
  [Fri Apr 29 10:18:49 2016]  [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc06121aa>] xfs_buf_get_map+0x2a/0x260 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d530a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811f2879>] mount_fs+0x39/0x160
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
  [Fri Apr 29 10:18:49 2016]  [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
  [Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e0 ]---
  [Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0 
  [Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------
  [Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
  [Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
  [Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G        W       4.2.0-35-generic #40~14.04.1-Ubuntu
  [Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
  [Fri Apr 29 10:18:49 2016]  0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000
  [Fri Apr 29 10:18:49 2016]  ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928
  [Fri Apr 29 10:18:49 2016]  ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18
  [Fri Apr 29 10:18:49 2016] Call Trace:
  [Fri Apr 29 10:18:49 2016]  [<ffffffff817bcbf8>] dump_stack+0x63/0x81
  [Fri Apr 29 10:18:49 2016]  [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc061235d>] xfs_buf_get_map+0x1dd/0x260 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811f2879>] mount_fs+0x39/0x160
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
  [Fri Apr 29 10:18:49 2016]  [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
  [Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e1 ]---
  [Fri Apr 29 10:18:49 2016] BUG: unable to handle kernel NULL pointer dereference at 00000000000000f8
  [Fri Apr 29 10:18:49 2016] IP: [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
  [Fri Apr 29 10:18:49 2016] PGD 564eec067 PUD 11408b067 PMD 0 
  [Fri Apr 29 10:18:49 2016] Oops: 0000 [#1] SMP 
  [Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
  [Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G        W       4.2.0-35-generic #40~14.04.1-Ubuntu
  [Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
  [Fri Apr 29 10:18:49 2016] task: ffff88080967ee00 ti: ffff88018e118000 task.ti: ffff88018e118000
  [Fri Apr 29 10:18:49 2016] RIP: 0010:[<ffffffffc063e6f6>]  [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
  [Fri Apr 29 10:18:49 2016] RSP: 0018:ffff88018e11ba08  EFLAGS: 00010292
  [Fri Apr 29 10:18:49 2016] RAX: 0000000000000000 RBX: ffff88018e11bb18 RCX: 00000000003d2d52
  [Fri Apr 29 10:18:49 2016] RDX: 00000000003d2d51 RSI: 0000000000000000 RDI: ffff880afa4e4828
  [Fri Apr 29 10:18:49 2016] RBP: ffff88018e11ba28 R08: 000000000001a4f0 R09: ffff88080f8da4f0
  [Fri Apr 29 10:18:49 2016] R10: ffffffffc061164a R11: ffffea001281d600 R12: ffff880afa4e4828
  [Fri Apr 29 10:18:49 2016] R13: ffff880afa4e4828 R14: 0000000000000000 R15: 0000000000000008
  [Fri Apr 29 10:18:49 2016] FS:  00007f42d515f880(0000) GS:ffff88080f8c0000(0000) knlGS:0000000000000000
  [Fri Apr 29 10:18:49 2016] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [Fri Apr 29 10:18:49 2016] CR2: 00000000000000f8 CR3: 0000000139a06000 CR4: 00000000000406e0
  [Fri Apr 29 10:18:49 2016] Stack:
  [Fri Apr 29 10:18:49 2016]  ffff88018e11bb18 ffff880afa4e4828 ffff88075d649ec0 ffff880164bc8800
  [Fri Apr 29 10:18:49 2016]  ffff88018e11bb08 ffffffffc05d5d26 ffff88015e77eb80 0000000200000000
  [Fri Apr 29 10:18:49 2016]  ffff88018e11ba98 ffffffff811d1075 ffff88015e77eb80 0000000000000000
  [Fri Apr 29 10:18:49 2016] Call Trace:
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d5d26>] xfs_alloc_fix_freelist+0x216/0x3c0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
  [Fri Apr 29 10:18:49 2016]  [<ffffffff811f2879>] mount_fs+0x39/0x160
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
  [Fri Apr 29 10:18:49 2016]  [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
  [Fri Apr 29 10:18:50 2016]  [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
  [Fri Apr 29 10:18:50 2016]  [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
  [Fri Apr 29 10:18:50 2016]  [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
  [Fri Apr 29 10:18:50 2016] Code: 13 48 85 d2 75 eb e9 5f ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 56 49 89 f6 41 55 49 89 fd 41 54 53 <4c> 8b a6 f8 00 00 00 66 66 66 66 90 41 f6 44 24 78 04 74 4f 5b 
  [Fri Apr 29 10:18:50 2016] RIP  [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
  [Fri Apr 29 10:18:50 2016]  RSP <ffff88018e11ba08>
  [Fri Apr 29 10:18:50 2016] CR2: 00000000000000f8
  [Fri Apr 29 10:18:50 2016] ---[ end trace ce3e7a80324237e2 ]---
  root@ceph-store5:~# 

  Thanks to the magic of Ceph, I've so far managed to avoid data-loss
  and serious downtime, but I suspect there's a timebomb here waiting
  for anyone who upgrades from an older kernel to 4.2.0+.

  Please let me know is there's any further information I might be able to provide to track this down.
  --- 
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Apr 29 01:29 seq
   crw-rw---- 1 root audio 116, 33 Apr 29 01:29 timer
  AplayDevices: Error: [Errno 2] No such file or directory
  ApportVersion: 2.14.1-0ubuntu3.19
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: Error: [Errno 2] No such file or directory
  DistroRelease: Ubuntu 14.04
  MachineType: Dell Inc. PowerEdge R720xd
  Package: linux (not installed)
  PciMultimedia:
   
  ProcEnviron:
   TERM=xterm-256color
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcFB:
   
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-35-generic root=UUID=f022f2e6-eeba-4e27-b286-6acd2c4bd1ea ro net.ifnames=1 biosdevname=0 console=tty1 console=ttyS1,115200 initrd=ubuntu/amd64/generic/trusty/release/di-initrd BOOTIF=01-b0-83-fe-e6-17-cc nomdmonddf nomdmonisw crashkernel=384M-:128M
  ProcVersionSignature: Ubuntu 4.2.0-35.40~14.04.1-generic 4.2.8-ckt5
  RelatedPackageVersions:
   linux-restricted-modules-4.2.0-35-generic N/A
   linux-backports-modules-4.2.0-35-generic  N/A
   linux-firmware                            1.127.22
  RfKill: Error: [Errno 2] No such file or directory
  Tags:  trusty uec-images
  Uname: Linux 4.2.0-35-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups:
   
  _MarkForUpload: True
  dmi.bios.date: 07/09/2014
  dmi.bios.vendor: Dell Inc.
  dmi.bios.version: 2.4.3
  dmi.board.name: 0HJK12
  dmi.board.vendor: Dell Inc.
  dmi.board.version: A03
  dmi.chassis.asset.tag: 707
  dmi.chassis.type: 23
  dmi.chassis.vendor: Dell Inc.
  dmi.modalias: dmi:bvnDellInc.:bvr2.4.3:bd07/09/2014:svnDellInc.:pnPowerEdgeR720xd:pvr:rvnDellInc.:rn0HJK12:rvrA03:cvnDellInc.:ct23:cvr:
  dmi.product.name: PowerEdge R720xd
  dmi.sys.vendor: Dell Inc.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1576599/+subscriptions
References

[Bug 1576599] [NEW] XFS Oopsing 4.2.0-35-generic Kernel - Block out of range
From: John McEleney, 2016-04-29