← Back to team overview

kernel-packages team mailing list archive

[Bug 1452868] [NEW] "ip link delete" returns "RTNETLINK answers: Operation not supported" - OpenStack Juno is completely broken in Trusty + Linux 3.19.

 

Public bug reported:

Guys,

 I just upgraded my Trusty with "linux-generic-lts-vivid" that is
available in proposed repository.

 OpenStack is completely broken now. I'm using it with VXLAN +
OpenvSwitch. All Tenants are offline.

 I'm seeing the following error at Network Node (Neutron L3 Router):

---
==> /var/log/neutron/ovs-cleanup.log <==
2015-05-07 13:30:57.384 881 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 86, in _execute
2015-05-07 13:30:57.384 881 TRACE neutron     log_fail_as_error=log_fail_as_error)
2015-05-07 13:30:57.384 881 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 84, in execute
2015-05-07 13:30:57.384 881 TRACE neutron     raise RuntimeError(m)
2015-05-07 13:30:57.384 881 TRACE neutron RuntimeError: 
2015-05-07 13:30:57.384 881 TRACE neutron Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'delete', 'tap3da8f4fe-55']
2015-05-07 13:30:57.384 881 TRACE neutron Exit code: 2
2015-05-07 13:30:57.384 881 TRACE neutron Stdout: ''
2015-05-07 13:30:57.384 881 TRACE neutron Stderr: 'RTNETLINK answers: Operation not supported\n'
2015-05-07 13:30:57.384 881 TRACE neutron 
---

It also doesn't work manually:

---
root@network-1:~# ip link delete tap3da8f4fe-55
RTNETLINK answers: Operation not supported
---

dmesg shows this:

---
[  152.307822] Modules linked in: vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch libcrc32c snd_hda_codec_generic iosf_mbi snd_hda_intel crct10dif_pclmul crc32_pclmul snd_hda_controller ghash_clmulni_intel snd_hda_codec aesni_intel ppdev snd_hwdep snd_pcm aes_x86_64 qxl lrw snd_timer gf128mul snd ttm glue_helper drm_kms_helper soundcore i2c_piix4 ablk_helper drm serio_raw cryptd pvpanic 8250_fintek parport_pc mac_hid lp parport psmouse pata_acpi floppy
[  152.307893] CPU: 0 PID: 1067 Comm: ovs-vswitchd Not tainted 3.19.0-16-generic #16~14.04.2-Ubuntu
[  152.307901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[  152.307906] task: ffff880036aee220 ti: ffff880079eb4000 task.ti: ffff880079eb4000
[  152.307912] RIP: 0010:[<ffffffffc05fd901>]  [<ffffffffc05fd901>] ovs_vport_del+0x41/0x60 [openvswitch]
[  152.307923] RSP: 0018:ffff880079eb79a8  EFLAGS: 00010296
[  152.307929] RAX: 8000000000000025 RBX: ffff880079f04e40 RCX: 00000001810000ec
[  152.307934] RDX: 00000001810000ed RSI: ffffea0000d860c0 RDI: 0000000040000000
[  152.307938] RBP: ffff880079eb79b8 R08: ffff8800361834a0 R09: 00000001810000ec
[  152.307943] R10: ffffea0000d860c0 R11: ffffffff813a7f65 R12: ffff880079f04e40
[  152.307948] R13: ffff880079e97100 R14: 00000000ffffffea R15: ffff880079e30180
[  152.307955] FS:  00007fc5f96a5980(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
[  152.307961] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  152.307966] CR2: 00007f32854bd740 CR3: 000000007a032000 CR4: 00000000000406f0
[  152.307972] Stack:
[  152.307975]  ffff880079eb7a30 ffff880079eb7a30 ffff880079eb79c8 ffffffffc05f6380
[  152.307984]  ffff880079eb79f8 ffffffffc05f6455 ffffffffc05ff268 ffffffffc0602040
[  152.307992]  ffff88007a700814 ffff880079e97000 ffff880079eb7aa8 ffffffff816db0e5
[  152.308003] Call Trace:
[  152.308012]  [<ffffffffc05f6380>] ovs_dp_detach_port+0x30/0x40 [openvswitch]
[  152.308019]  [<ffffffffc05f6455>] ovs_vport_cmd_del+0xc5/0x110 [openvswitch]
[  152.308034]  [<ffffffff816db0e5>] genl_family_rcv_msg+0x1a5/0x3d0
[  152.308040]  [<ffffffff816db310>] ? genl_family_rcv_msg+0x3d0/0x3d0
[  152.308046]  [<ffffffff816db3a1>] genl_rcv_msg+0x91/0xd0
[  152.308051]  [<ffffffff816d9191>] netlink_rcv_skb+0xc1/0xe0
[  152.308056]  [<ffffffff816d983c>] genl_rcv+0x2c/0x40
[  152.308060]  [<ffffffff816d8816>] netlink_unicast+0xf6/0x200
[  152.308065]  [<ffffffff816d8c33>] netlink_sendmsg+0x313/0x690
[  152.308078]  [<ffffffff81350fe0>] ? aa_sk_perm.isra.4+0x70/0x150
[  152.308090]  [<ffffffff8168e34c>] do_sock_sendmsg+0x8c/0x100
[  152.308096]  [<ffffffff8168dd0e>] ? copy_msghdr_from_user+0x15e/0x1f0
[  152.308103]  [<ffffffff8168e963>] ___sys_sendmsg+0x313/0x320
[  152.308112]  [<ffffffff8120713c>] ? destroy_inode+0x3c/0x70
[  152.308118]  [<ffffffff81207287>] ? evict+0x117/0x190
[  152.308122]  [<ffffffff8120230f>] ? dentry_free+0x5f/0xa0
[  152.308127]  [<ffffffff81202c64>] ? __dentry_kill+0x154/0x1f0
[  152.308132]  [<ffffffff81202da5>] ? dput+0xa5/0x170
[  152.308138]  [<ffffffff8120bdc4>] ? mntput+0x24/0x40
[  152.308143]  [<ffffffff8168f052>] __sys_sendmsg+0x42/0x80
[  152.308148]  [<ffffffff8168f0a2>] SyS_sendmsg+0x12/0x20
[  152.308156]  [<ffffffff817b520d>] system_call_fastpath+0x16/0x1b
[  152.308208] Code: 8b 57 30 48 85 c0 48 89 02 74 04 48 89 50 08 48 b8 00 02 20 00 00 00 ad de 48 89 df 48 89 43 30 48 8b 43 48 ff 50 10 48 8b 43 48 <48> 8b 78 40 e8 d6 55 af c0 48 83 c4 08 5b 5d c3 66 66 66 66 66 
[  152.308375] RIP  [<ffffffffc05fd901>] ovs_vport_del+0x41/0x60 [openvswitch]
[  152.308433]  RSP <ffff880079eb79a8>
[  152.308504] ---[ end trace cc9561be41fd191a ]---
[  217.847530] init: neutron-ovs-cleanup pre-start process (871) terminated with status 1
[  217.913947] init: wait-for-state (neutron-dhcp-agentneutron-ovs-cleanup) main process (862) killed by TERM signal
---

Falling back to Linux 3.16.

Info:

root@network-1:~# cat /proc/version_signature > version.log
root@network-1:~# cat version.log 
Ubuntu 3.19.0-16.16~14.04.2-generic 3.19.3

Thanks,
Thiago

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: Incomplete


** Tags: trusty

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1452868

Title:
  "ip link delete" returns "RTNETLINK answers: Operation not supported"
  - OpenStack Juno is completely broken in Trusty + Linux 3.19.

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  Guys,

   I just upgraded my Trusty with "linux-generic-lts-vivid" that is
  available in proposed repository.

   OpenStack is completely broken now. I'm using it with VXLAN +
  OpenvSwitch. All Tenants are offline.

   I'm seeing the following error at Network Node (Neutron L3 Router):

  ---
  ==> /var/log/neutron/ovs-cleanup.log <==
  2015-05-07 13:30:57.384 881 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 86, in _execute
  2015-05-07 13:30:57.384 881 TRACE neutron     log_fail_as_error=log_fail_as_error)
  2015-05-07 13:30:57.384 881 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 84, in execute
  2015-05-07 13:30:57.384 881 TRACE neutron     raise RuntimeError(m)
  2015-05-07 13:30:57.384 881 TRACE neutron RuntimeError: 
  2015-05-07 13:30:57.384 881 TRACE neutron Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'delete', 'tap3da8f4fe-55']
  2015-05-07 13:30:57.384 881 TRACE neutron Exit code: 2
  2015-05-07 13:30:57.384 881 TRACE neutron Stdout: ''
  2015-05-07 13:30:57.384 881 TRACE neutron Stderr: 'RTNETLINK answers: Operation not supported\n'
  2015-05-07 13:30:57.384 881 TRACE neutron 
  ---

  It also doesn't work manually:

  ---
  root@network-1:~# ip link delete tap3da8f4fe-55
  RTNETLINK answers: Operation not supported
  ---

  dmesg shows this:

  ---
  [  152.307822] Modules linked in: vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch libcrc32c snd_hda_codec_generic iosf_mbi snd_hda_intel crct10dif_pclmul crc32_pclmul snd_hda_controller ghash_clmulni_intel snd_hda_codec aesni_intel ppdev snd_hwdep snd_pcm aes_x86_64 qxl lrw snd_timer gf128mul snd ttm glue_helper drm_kms_helper soundcore i2c_piix4 ablk_helper drm serio_raw cryptd pvpanic 8250_fintek parport_pc mac_hid lp parport psmouse pata_acpi floppy
  [  152.307893] CPU: 0 PID: 1067 Comm: ovs-vswitchd Not tainted 3.19.0-16-generic #16~14.04.2-Ubuntu
  [  152.307901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
  [  152.307906] task: ffff880036aee220 ti: ffff880079eb4000 task.ti: ffff880079eb4000
  [  152.307912] RIP: 0010:[<ffffffffc05fd901>]  [<ffffffffc05fd901>] ovs_vport_del+0x41/0x60 [openvswitch]
  [  152.307923] RSP: 0018:ffff880079eb79a8  EFLAGS: 00010296
  [  152.307929] RAX: 8000000000000025 RBX: ffff880079f04e40 RCX: 00000001810000ec
  [  152.307934] RDX: 00000001810000ed RSI: ffffea0000d860c0 RDI: 0000000040000000
  [  152.307938] RBP: ffff880079eb79b8 R08: ffff8800361834a0 R09: 00000001810000ec
  [  152.307943] R10: ffffea0000d860c0 R11: ffffffff813a7f65 R12: ffff880079f04e40
  [  152.307948] R13: ffff880079e97100 R14: 00000000ffffffea R15: ffff880079e30180
  [  152.307955] FS:  00007fc5f96a5980(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
  [  152.307961] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [  152.307966] CR2: 00007f32854bd740 CR3: 000000007a032000 CR4: 00000000000406f0
  [  152.307972] Stack:
  [  152.307975]  ffff880079eb7a30 ffff880079eb7a30 ffff880079eb79c8 ffffffffc05f6380
  [  152.307984]  ffff880079eb79f8 ffffffffc05f6455 ffffffffc05ff268 ffffffffc0602040
  [  152.307992]  ffff88007a700814 ffff880079e97000 ffff880079eb7aa8 ffffffff816db0e5
  [  152.308003] Call Trace:
  [  152.308012]  [<ffffffffc05f6380>] ovs_dp_detach_port+0x30/0x40 [openvswitch]
  [  152.308019]  [<ffffffffc05f6455>] ovs_vport_cmd_del+0xc5/0x110 [openvswitch]
  [  152.308034]  [<ffffffff816db0e5>] genl_family_rcv_msg+0x1a5/0x3d0
  [  152.308040]  [<ffffffff816db310>] ? genl_family_rcv_msg+0x3d0/0x3d0
  [  152.308046]  [<ffffffff816db3a1>] genl_rcv_msg+0x91/0xd0
  [  152.308051]  [<ffffffff816d9191>] netlink_rcv_skb+0xc1/0xe0
  [  152.308056]  [<ffffffff816d983c>] genl_rcv+0x2c/0x40
  [  152.308060]  [<ffffffff816d8816>] netlink_unicast+0xf6/0x200
  [  152.308065]  [<ffffffff816d8c33>] netlink_sendmsg+0x313/0x690
  [  152.308078]  [<ffffffff81350fe0>] ? aa_sk_perm.isra.4+0x70/0x150
  [  152.308090]  [<ffffffff8168e34c>] do_sock_sendmsg+0x8c/0x100
  [  152.308096]  [<ffffffff8168dd0e>] ? copy_msghdr_from_user+0x15e/0x1f0
  [  152.308103]  [<ffffffff8168e963>] ___sys_sendmsg+0x313/0x320
  [  152.308112]  [<ffffffff8120713c>] ? destroy_inode+0x3c/0x70
  [  152.308118]  [<ffffffff81207287>] ? evict+0x117/0x190
  [  152.308122]  [<ffffffff8120230f>] ? dentry_free+0x5f/0xa0
  [  152.308127]  [<ffffffff81202c64>] ? __dentry_kill+0x154/0x1f0
  [  152.308132]  [<ffffffff81202da5>] ? dput+0xa5/0x170
  [  152.308138]  [<ffffffff8120bdc4>] ? mntput+0x24/0x40
  [  152.308143]  [<ffffffff8168f052>] __sys_sendmsg+0x42/0x80
  [  152.308148]  [<ffffffff8168f0a2>] SyS_sendmsg+0x12/0x20
  [  152.308156]  [<ffffffff817b520d>] system_call_fastpath+0x16/0x1b
  [  152.308208] Code: 8b 57 30 48 85 c0 48 89 02 74 04 48 89 50 08 48 b8 00 02 20 00 00 00 ad de 48 89 df 48 89 43 30 48 8b 43 48 ff 50 10 48 8b 43 48 <48> 8b 78 40 e8 d6 55 af c0 48 83 c4 08 5b 5d c3 66 66 66 66 66 
  [  152.308375] RIP  [<ffffffffc05fd901>] ovs_vport_del+0x41/0x60 [openvswitch]
  [  152.308433]  RSP <ffff880079eb79a8>
  [  152.308504] ---[ end trace cc9561be41fd191a ]---
  [  217.847530] init: neutron-ovs-cleanup pre-start process (871) terminated with status 1
  [  217.913947] init: wait-for-state (neutron-dhcp-agentneutron-ovs-cleanup) main process (862) killed by TERM signal
  ---

  Falling back to Linux 3.16.

  Info:

  root@network-1:~# cat /proc/version_signature > version.log
  root@network-1:~# cat version.log 
  Ubuntu 3.19.0-16.16~14.04.2-generic 3.19.3

  Thanks,
  Thiago

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1452868/+subscriptions


Follow ups

References