kernel-packages team mailing list archive

Thread
Date
[Bug 1273386] Re: Neutron namespace metadata proxy triggers kernel crash on Ubuntu 12.04/3.2 kernel

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: "Davanum Srinivas \(DIMS\)" <davanum@xxxxxxxxx>
Date: Tue, 28 Jan 2014 14:27:39 -0000
Reply-to: Bug 1273386 <1273386@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
pasting one trace here for the record

Jan 24 19:59:19 localhost kernel: [ 1028.120275] ------------[ cut here ]------------
Jan 24 19:59:19 localhost kernel: [ 1028.120848] kernel BUG at /build/buildd/linux-3.2.0/fs/buffer.c:2917!
Jan 24 19:59:19 localhost kernel: [ 1028.121602] invalid opcode: 0000 [#1] SMP 
Jan 24 19:59:19 localhost kernel: [ 1028.122098] CPU 1 
Jan 24 19:59:19 localhost kernel: [ 1028.122314] Modules linked in: xt_mac xt_physdev xt_conntrack ipt_REDIRECT xfs rmd160 crypto_null xfrm_user ah6 ah4 esp6 esp4 xfrm4_mode_beet xfrm4_tunnel tunnel4 xfrm4_mode_tunnel xfrm4_mode_transport xfrm6_mode_transport xfrm6_mode_ro xfrm6_mode_beet xfrm6_mode_tunnel ipcomp ipcomp6 xfrm_ipcomp xfrm6_tunnel tunnel6 af_key camellia lzo cast6 cast5 deflate zlib_deflate cts ctr gcm ccm serpent blowfish_generic blowfish_x86_64 blowfish_common twofish_generic twofish_x86_64_3way twofish_x86_64 twofish_common xcbc sha512_generic des_generic cryptd aes_x86_64 kvm ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp openvswitch(O) iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath nbd ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables psmouse serio_raw virtio_balloon acpiphp floppy
Jan 24 19:59:19 localhost kernel: [ 1028.130686] 
Jan 24 19:59:19 localhost kernel: [ 1028.130686] Pid: 12224, comm: dnsmasq Tainted: G           O 3.2.0-58-virtual #88-Ubuntu Bochs Bochs
Jan 24 19:59:19 localhost kernel: [ 1028.130686] RIP: 0010:[<ffffffff811a76d3>]  [<ffffffff811a76d3>] submit_bh+0x113/0x120
Jan 24 19:59:19 localhost kernel: [ 1028.130686] RSP: 0000:ffff88015b2df738  EFLAGS: 00010246
Jan 24 19:59:19 localhost kernel: [ 1028.130686] RAX: 0000000000000005 RBX: ffff880072d71f08 RCX: 00000000000b2a1e
Jan 24 19:59:19 localhost kernel: [ 1028.130686] RDX: 000000000000008d RSI: ffff880072d71f08 RDI: 0000000000000211
Jan 24 19:59:19 localhost kernel: [ 1028.130686] RBP: ffff88015b2df758 R08: ffffffff811190c7 R09: 0000000180150015
Jan 24 19:59:19 localhost kernel: [ 1028.130686] R10: ffff8801ec485180 R11: 0000000001000000 R12: 0000000000000211
Jan 24 19:59:19 localhost kernel: [ 1028.130686] R13: ffff880004a8c824 R14: 0000000000000001 R15: ffff880072d71f08
Jan 24 19:59:19 localhost kernel: [ 1028.130686] FS:  00007fd15c3aa700(0000) GS:ffff88021fc80000(0000) knlGS:0000000000000000
Jan 24 19:59:19 localhost kernel: [ 1028.130686] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 19:59:19 localhost kernel: [ 1028.130686] CR2: 00000000042c5560 CR3: 000000020f9b6000 CR4: 00000000000006e0
Jan 24 19:59:19 localhost kernel: [ 1028.130686] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 19:59:19 localhost kernel: [ 1028.130686] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 19:59:19 localhost kernel: [ 1028.130686] Process dnsmasq (pid: 12224, threadinfo ffff88015b2de000, task ffff8801a7178000)
Jan 24 19:59:19 localhost kernel: [ 1028.130686] Stack:
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  ffff8801ec485180 ffff880072d71f08 0000000000000211 ffff880004a8c824
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  ffff88015b2df788 ffffffff811a9442 ffff88015b2df7b8 ffff8801ec485180
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  ffff880004a8c800 ffff8800c3726c00 ffff88015b2df798 ffffffff811a94d3
Jan 24 19:59:19 localhost kernel: [ 1028.130686] Call Trace:
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff811a9442>] __sync_dirty_buffer+0x52/0xd0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff811a94d3>] sync_dirty_buffer+0x13/0x20
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8125c578>] journal_update_superblock+0x118/0x220
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff811ad575>] ? bio_fs_destructor+0x15/0x20
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff811ac6eb>] ? bio_put+0x2b/0x30
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff812f4e9d>] ? blkdev_issue_flush+0xbd/0xe0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81258463>] cleanup_journal_tail.part.6+0x113/0x210
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8125857d>] cleanup_journal_tail+0x1d/0x30
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81258d41>] log_do_checkpoint+0x21/0x1f0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8105053d>] ? ttwu_do_wakeup+0x3d/0x120
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8103dd05>] ? pvclock_clocksource_read+0x55/0xf0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81055db3>] ? update_curr+0x143/0x1f0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8103dc39>] ? default_spin_lock_flags+0x9/0x10
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81661680>] ? kprobe_flush_task+0xc0/0x150
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8105588f>] ? finish_task_switch+0xcf/0xf0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff816588cc>] ? __schedule+0x3cc/0x6f0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8103dc39>] ? default_spin_lock_flags+0x9/0x10
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8103dc39>] ? default_spin_lock_flags+0x9/0x10
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8108a6e7>] ? finish_wait+0x67/0x80
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8165ab3e>] ? _raw_spin_lock+0xe/0x20
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8125ab8a>] ? journal_kill_thread+0xfa/0x110
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8108a5f0>] ? add_wait_queue+0x60/0x60
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8125cc22>] journal_destroy+0x52/0x130
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81207557>] ext3_put_super+0x47/0x2a0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff811795f2>] generic_shutdown_super+0x62/0xe0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff811796a0>] kill_block_super+0x30/0x80
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81179d4c>] deactivate_locked_super+0x3c/0xa0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8117a5ce>] deactivate_super+0x4e/0x70
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81196abd>] mntput_no_expire+0x9d/0xf0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81196b2f>] mntput+0x1f/0x30
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8119760a>] release_mounts+0x7a/0x90
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81197866>] put_mnt_ns+0x66/0x80
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8108f3bf>] free_nsproxy+0x1f/0x90
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8108f5d8>] switch_task_namespaces+0x38/0x40
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8108f5f0>] exit_task_namespaces+0x10/0x20
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8106ad4f>] exit_notify+0x1f/0x110
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8106b5d2>] do_exit+0x1f2/0x450
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81079afa>] ? __dequeue_signal+0x6a/0xb0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8106b9d4>] do_group_exit+0x44/0xa0
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff8107c67c>] get_signal_to_deliver+0x21c/0x420
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81014825>] do_signal+0x45/0x130
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81014ad5>] do_notify_resume+0x65/0x80
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  [<ffffffff81663450>] int_signal+0x12/0x17
Jan 24 19:59:19 localhost kernel: [ 1028.130686] Code: 8d 14 00 49 8b 5d 18 4c 89 ef e8 09 50 00 00 48 83 c4 08 c1 e3 18 c1 fb 1f 83 e3 a1 89 d8 5b 41 5c 41 5d 5d c3 0f 0b 0f 0b 0f 0b <0f> 0b 0f 0b 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 
Jan 24 19:59:19 localhost kernel: [ 1028.130686] RIP  [<ffffffff811a76d3>] submit_bh+0x113/0x120
Jan 24 19:59:19 localhost kernel: [ 1028.130686]  RSP <ffff88015b2df738>
Jan 24 19:59:19 localhost kernel: [ 1028.182334] ---[ end trace dcd1b28a51f9e8d7 ]---

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1273386

Title:
  Neutron namespace metadata proxy triggers kernel crash on Ubuntu
  12.04/3.2 kernel

Status in OpenStack Neutron (virtual network service):
  New
Status in OpenStack Compute (Nova):
  New
Status in “linux” package in Ubuntu:
  Incomplete

Bug description:
  In the past 9 days we have been seeing very frequent occurences of
  this kernel crash: http://paste.openstack.org/show/61869/

  Even if the particular crash pasted here is triggered by dnsmasq, in
  almost all cases the crash is actually triggered by the neutron metada
  proxy.

  This also affects nova badly since this issue, which appears namespace
  related, results in a hang while mounting the ndb device for key
  injection.

  logstash query:
  http://logstash.openstack.org/#eyJzZWFyY2giOiJcImtlcm5lbCBCVUcgYXQgL2J1aWxkL2J1aWxkZC9saW51eC0zLjIuMC9mcy9idWZmZXIuYzoyOTE3XCIgYW5kIGZpbGVuYW1lOnN5c2xvZy50eHQiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6ImN1c3RvbSIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJmcm9tIjoiMjAxNC0wMS0xNlQxODo1MDo0OCswMDowMCIsInRvIjoiMjAxNC0wMS0yN1QxOToxNjoxMSswMDowMCIsInVzZXJfaW50ZXJ2YWwiOiIwIn0sInN0YW1wIjoxMzkwODUwMzI2ODY0fQ==

  We have seen about 398 hits since the bug started to manifest.
  Decreased hit rate in the past few days is due to less neutron patches being pushed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1273386/+subscriptions