yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1373513] Re: Lvm hang during tempest tests

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: John Griffith <1373513@xxxxxxxxxxxxxxxxxx>
Date: Tue, 17 Mar 2015 03:33:00 -0000
Reply-to: Bug 1373513 <1373513@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: New => Invalid

** Changed in: cinder
     Assignee: (unassigned) => John Griffith (john-griffith)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1373513

Title:
  Lvm hang during tempest tests

Status in Cinder:
  Triaged
Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  Managed to trigger a hang in lvm create

  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.704164] INFO: task lvm:14805 blocked for more than 120 seconds.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.705096]       Not tainted 3.13.0-35-generic #62-Ubuntu
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.705839] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706871] lvm             D ffff8801ffd14440     0 14805  14804 0x00000000
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706876]  ffff880068f9dae0 0000000000000082 ffff8801a14bc800 ffff880068f9dfd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706879]  0000000000014440 0000000000014440 ffff8801a14bc800 ffff8801ffd14cd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706881]  0000000000000000 ffff88004063c280 0000000000000000 ffff8801a14bc800
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706883] Call Trace:
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706895]  [<ffffffff81722a6d>] io_schedule+0x9d/0x140
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706914]  [<ffffffff811fac94>] do_blockdev_direct_IO+0x1ce4/0x2910
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706918]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706920]  [<ffffffff811fb915>] __blockdev_direct_IO+0x55/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706922]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706924]  [<ffffffff811f61f6>] blkdev_direct_IO+0x56/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706926]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706937]  [<ffffffff8115106b>] generic_file_aio_read+0x69b/0x700
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706947]  [<ffffffff811cca78>] ? path_openat+0x158/0x640
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706953]  [<ffffffff810f3c92>] ? from_kgid_munged+0x12/0x20
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706955]  [<ffffffff811f667b>] blkdev_aio_read+0x4b/0x70
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706958]  [<ffffffff811bc99a>] do_sync_read+0x5a/0x90
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706960]  [<ffffffff811bd035>] vfs_read+0x95/0x160
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706962]  [<ffffffff811bdb49>] SyS_read+0x49/0xa0
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706966]  [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.706968] INFO: task lvs:14822 blocked for more than 120 seconds.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.707774]       Not tainted 3.13.0-35-generic #62-Ubuntu
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.708507] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709535] lvs             D ffff8801ffc14440     0 14822  14821 0x00000000
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709537]  ffff880009ffdae0 0000000000000082 ffff8800095e1800 ffff880009ffdfd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709539]  0000000000014440 0000000000014440 ffff8800095e1800 ffff8801ffc14cd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709541]  0000000000000000 ffff880003d59900 0000000000000000 ffff8800095e1800
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709543] Call Trace:
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709547]  [<ffffffff81722a6d>] io_schedule+0x9d/0x140
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709549]  [<ffffffff811fac94>] do_blockdev_direct_IO+0x1ce4/0x2910
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709551]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709554]  [<ffffffff811fb915>] __blockdev_direct_IO+0x55/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709555]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709557]  [<ffffffff811f61f6>] blkdev_direct_IO+0x56/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709559]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709562]  [<ffffffff8115106b>] generic_file_aio_read+0x69b/0x700
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709564]  [<ffffffff811cca78>] ? path_openat+0x158/0x640
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709566]  [<ffffffff810f3c92>] ? from_kgid_munged+0x12/0x20
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709568]  [<ffffffff811f667b>] blkdev_aio_read+0x4b/0x70
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709570]  [<ffffffff811bc99a>] do_sync_read+0x5a/0x90
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709572]  [<ffffffff811bd035>] vfs_read+0x95/0x160
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709573]  [<ffffffff811bdb49>] SyS_read+0x49/0xa0
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709575]  [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.709578] INFO: task lvs:14984 blocked for more than 120 seconds.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.710411]       Not tainted 3.13.0-35-generic #62-Ubuntu
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.711153] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712193] lvs             D ffff8801ffc94440     0 14984  14983 0x00000000
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712196]  ffff880101711ae0 0000000000000086 ffff880007d2b000 ffff880101711fd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712198]  0000000000014440 0000000000014440 ffff880007d2b000 ffff8801ffc94cd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712200]  0000000000000000 ffff880017fd2300 0000000000000000 ffff880007d2b000
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712202] Call Trace:
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712219]  [<ffffffff81722a6d>] io_schedule+0x9d/0x140
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712221]  [<ffffffff811fac94>] do_blockdev_direct_IO+0x1ce4/0x2910
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712224]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712226]  [<ffffffff811fb915>] __blockdev_direct_IO+0x55/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712228]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712230]  [<ffffffff811f61f6>] blkdev_direct_IO+0x56/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712231]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712234]  [<ffffffff8115106b>] generic_file_aio_read+0x69b/0x700
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712236]  [<ffffffff811cca78>] ? path_openat+0x158/0x640
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712238]  [<ffffffff810f3c92>] ? from_kgid_munged+0x12/0x20
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712249]  [<ffffffff811f667b>] blkdev_aio_read+0x4b/0x70
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712256]  [<ffffffff811bc99a>] do_sync_read+0x5a/0x90
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712264]  [<ffffffff811bd035>] vfs_read+0x95/0x160
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712270]  [<ffffffff811bdb49>] SyS_read+0x49/0xa0
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712276]  [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.712278] INFO: task vgs:15086 blocked for more than 120 seconds.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.713149]       Not tainted 3.13.0-35-generic #62-Ubuntu
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.713870] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714919] vgs             D ffff8801ffc94440     0 15086  15061 0x00000000
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921]  ffff88014411fae0 0000000000000082 ffff8801f2f7e000 ffff88014411ffd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921]  0000000000014440 0000000000014440 ffff8801f2f7e000 ffff8801ffc94cd8
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921]  0000000000000000 ffff880017fd2580 0000000000000000 ffff8801f2f7e000
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921] Call Trace:
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921]  [<ffffffff81722a6d>] io_schedule+0x9d/0x140
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921]  [<ffffffff811fac94>] do_blockdev_direct_IO+0x1ce4/0x2910
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714921]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714923]  [<ffffffff811fb915>] __blockdev_direct_IO+0x55/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714924]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714926]  [<ffffffff811f61f6>] blkdev_direct_IO+0x56/0x60
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714928]  [<ffffffff811f5b00>] ? I_BDEV+0x10/0x10
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714930]  [<ffffffff8115106b>] generic_file_aio_read+0x69b/0x700
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714932]  [<ffffffff811cca78>] ? path_openat+0x158/0x640
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714934]  [<ffffffff810f3c92>] ? from_kgid_munged+0x12/0x20
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714936]  [<ffffffff811f667b>] blkdev_aio_read+0x4b/0x70
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714938]  [<ffffffff811bc99a>] do_sync_read+0x5a/0x90
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714940]  [<ffffffff811bd035>] vfs_read+0x95/0x160
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714942]  [<ffffffff811bdb49>] SyS_read+0x49/0xa0
  Sep 22 21:19:01 devstack-trusty-hpcloud-b5-2344254 kernel: [ 3120.714944]  [<ffffffff8172ed6d>] system_call_fastpath+0x1a/0x1f

  Link to logs: http://logs.openstack.org/22/119522/9/check/check-
  tempest-dsvm-postgres-full/46a5f8c/logs/syslog.txt.gz#_Sep_22_21_19_01

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1373513/+subscriptions