yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1373513] Re: Lvm hang during tempest tests

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: John Griffith <1373513@xxxxxxxxxxxxxxxxxx>
Date: Tue, 10 Mar 2015 15:24:13 -0000
Reply-to: Bug 1373513 <1373513@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
I've added Nova to the projects here because currently we're at a stale
mate where there seems to be a single case during unrescue that tirggers
this.  Patc is proposed but looks like it won't be accepted, want to
make sure we link this and keep it tracked although it is different than
the original bug posted by Vish, I believe it's related.

** Project changed: cinder => nova

** Changed in: nova
    Milestone: kilo-3 => None

** Also affects: cinder
   Importance: Undecided
       Status: New

** Changed in: cinder
       Status: New => In Progress

** Changed in: cinder
   Importance: Undecided => Critical

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1373513

Title:
  Lvm hang during tempest tests

Status in Cinder:
  In Progress
Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  Managed to trigger a hang in lvm create

  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.704164] INFO: task lvm:14805 blocked for more than 120 seconds.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.705096]       Not tainted 3.13.0-35-generic #62-Ubuntu
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.705839] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706871] lvm             D ffff8801ffd14440     0 14805  14804 0x00000000
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706876]  ffff880068f9dae0 0000000000000082 ffff8801a14bc800 ffff880068f9dfd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706879]  0000000000014440 0000000000014440 ffff8801a14bc800 ffff8801ffd14cd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706881]  0000000000000000 ffff88004063c280 0000000000000000 ffff8801a14bc800
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706883] Call Trace:
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706895]  [<ffffffff81722a6d>] io_schedule+0x9d/0x140
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706914]  [<ffffffff811fac94>] do_blockdev_direct_IO+0x1ce4/0x2910
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706918]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706920]  [<ffffffff811fb915>] __blockdev_direct_IO+0x55/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706922]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706924]  [<ffffffff811f61f6>] blkdev_direct_IO+0x56/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706926]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706937]  [<ffffffff8115106b>] generic_file_aio_read+0x69b/0x700
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706947]  [<ffffffff811cca78>] ? path_openat+0x158/0x640
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706953]  [<ffffffff810f3c92>] ? from_kgid_munged+0x12/0x20
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706955]  [<ffffffff811f667b>] blkdev_aio_read+0x4b/0x70
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706958]  [<ffffffff811bc99a>] do_sync_read+0x5a/0x90
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706960]  [<ffffffff811bd035>] vfs_read+0x95/0x160
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706962]  [<ffffffff811bdb49>] SyS_read+0x49/0xa0
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706966]  [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706968] INFO: task lvs:14822 blocked for more than 120 seconds.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.707774]       Not tainted 3.13.0-35-generic #62-Ubuntu
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.708507] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709535] lvs             D ffff8801ffc14440     0 14822  14821 0x00000000
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709537]  ffff880009ffdae0 0000000000000082 ffff8800095e1800 ffff880009ffdfd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709539]  0000000000014440 0000000000014440 ffff8800095e1800 ffff8801ffc14cd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709541]  0000000000000000 ffff880003d59900 0000000000000000 ffff8800095e1800
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709543] Call Trace:
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709547]  [<ffffffff81722a6d>] io_schedule+0x9d/0x140
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709549]  [<ffffffff811fac94>] do_blockdev_direct_IO+0x1ce4/0x2910
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709551]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709554]  [<ffffffff811fb915>] __blockdev_direct_IO+0x55/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709555]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709557]  [<ffffffff811f61f6>] blkdev_direct_IO+0x56/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709559]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709562]  [<ffffffff8115106b>] generic_file_aio_read+0x69b/0x700
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709564]  [<ffffffff811cca78>] ? path_openat+0x158/0x640
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709566]  [<ffffffff810f3c92>] ? from_kgid_munged+0x12/0x20
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709568]  [<ffffffff811f667b>] blkdev_aio_read+0x4b/0x70
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709570]  [<ffffffff811bc99a>] do_sync_read+0x5a/0x90
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709572]  [<ffffffff811bd035>] vfs_read+0x95/0x160
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709573]  [<ffffffff811bdb49>] SyS_read+0x49/0xa0
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709575]  [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709578] INFO: task lvs:14984 blocked for more than 120 seconds.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.710411]       Not tainted 3.13.0-35-generic #62-Ubuntu
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.711153] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712193] lvs             D ffff8801ffc94440     0 14984  14983 0x00000000
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712196]  ffff880101711ae0 0000000000000086 ffff880007d2b000 ffff880101711fd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712198]  0000000000014440 0000000000014440 ffff880007d2b000 ffff8801ffc94cd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712200]  0000000000000000 ffff880017fd2300 0000000000000000 ffff880007d2b000
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712202] Call Trace:
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712219]  [<ffffffff81722a6d>] io_schedule+0x9d/0x140
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712221]  [<ffffffff811fac94>] do_blockdev_direct_IO+0x1ce4/0x2910
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712224]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712226]  [<ffffffff811fb915>] __blockdev_direct_IO+0x55/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712228]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712230]  [<ffffffff811f61f6>] blkdev_direct_IO+0x56/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712231]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712234]  [<ffffffff8115106b>] generic_file_aio_read+0x69b/0x700
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712236]  [<ffffffff811cca78>] ? path_openat+0x158/0x640
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712238]  [<ffffffff810f3c92>] ? from_kgid_munged+0x12/0x20
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712249]  [<ffffffff811f667b>] blkdev_aio_read+0x4b/0x70
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712256]  [<ffffffff811bc99a>] do_sync_read+0x5a/0x90
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712264]  [<ffffffff811bd035>] vfs_read+0x95/0x160
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712270]  [<ffffffff811bdb49>] SyS_read+0x49/0xa0
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712276]  [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712278] INFO: task vgs:15086 blocked for more than 120 seconds.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.713149]       Not tainted 3.13.0-35-generic #62-Ubuntu
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.713870] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714919] vgs             D ffff8801ffc94440     0 15086  15061 0x00000000
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921]  ffff88014411fae0 0000000000000082 ffff8801f2f7e000 ffff88014411ffd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921]  0000000000014440 0000000000014440 ffff8801f2f7e000 ffff8801ffc94cd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921]  0000000000000000 ffff880017fd2580 0000000000000000 ffff8801f2f7e000
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921] Call Trace:
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921]  [<ffffffff81722a6d>] io_schedule+0x9d/0x140
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921]  [<ffffffff811fac94>] do_blockdev_direct_IO+0x1ce4/0x2910
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714923]  [<ffffffff811fb915>] __blockdev_direct_IO+0x55/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714924]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714926]  [<ffffffff811f61f6>] blkdev_direct_IO+0x56/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714928]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714930]  [<ffffffff8115106b>] generic_file_aio_read+0x69b/0x700
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714932]  [<ffffffff811cca78>] ? path_openat+0x158/0x640
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714934]  [<ffffffff810f3c92>] ? from_kgid_munged+0x12/0x20
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714936]  [<ffffffff811f667b>] blkdev_aio_read+0x4b/0x70
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714938]  [<ffffffff811bc99a>] do_sync_read+0x5a/0x90
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714940]  [<ffffffff811bd035>] vfs_read+0x95/0x160
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714942]  [<ffffffff811bdb49>] SyS_read+0x49/0xa0
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714944]  [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f

  Link to logs: http://logs.openstack.org/22/119522/9/check/check-
  tempest-dsvm-postgres-full/46a5f8c/logs/syslog.txt.gz#_Sep_22_21_19_01

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1373513/+subscriptions