kernel-packages team mailing list archive

Thread
Date
[Bug 1452868] Re: "ip link delete" returns "RTNETLINK answers: Operation not supported" - OpenStack Juno is completely broken in Trusty + Linux 3.19.

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Thiago Martins <thiagocmartinsc@xxxxxxxxx>
Date: Thu, 07 May 2015 21:40:51 -0000
Reply-to: Bug 1452868 <1452868@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Hello Chris,

 After a complete power cycle, Juno with Trusty + Linux 3.19 is working.
Tenants have connectivity again.

 So far, the only error that I'm still seeing, is the following:

---
==> /var/log/neutron/ovs-cleanup.log <==
2015-05-07 13:30:57.384 881 TRACE neutron Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'delete', 'tap3da8f4fe-55']
2015-05-07 13:30:57.384 881 TRACE neutron Exit code: 2
2015-05-07 13:30:57.384 881 TRACE neutron Stdout: ''
2015-05-07 13:30:57.384 881 TRACE neutron Stderr: 'RTNETLINK answers: Operation not supported\n'
---

 I'll clean up everything, bridges and Namespaces, to see if the problem
persists. I saw this problem in a lab as well, I'll reset the databases
and tenants to start over again, with Linux 3.19 from the beginning.

 If I do not manage to reproduce it until next Monday, I'll close this
BUG as "not reproducible".

 Sorry about the buzz!

Best,
Thiago

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1452868

Title:
  "ip link delete" returns "RTNETLINK answers: Operation not supported"
  - OpenStack Juno is completely broken in Trusty + Linux 3.19.

Status in linux package in Ubuntu:
  Confirmed
Status in linux source package in Vivid:
  New

Bug description:
  Guys,

   I just upgraded my Trusty with "linux-generic-lts-vivid" that is
  available in proposed repository.

   OpenStack is completely broken now. I'm using it with VXLAN +
  OpenvSwitch. All Tenants are offline.

   I'm seeing the following error at Network Node (Neutron L3 Router):

  ---
  ==> /var/log/neutron/ovs-cleanup.log <==
  2015-05-07 13:30:57.384 881 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 86, in _execute
  2015-05-07 13:30:57.384 881 TRACE neutron     log_fail_as_error=log_fail_as_error)
  2015-05-07 13:30:57.384 881 TRACE neutron   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 84, in execute
  2015-05-07 13:30:57.384 881 TRACE neutron     raise RuntimeError(m)
  2015-05-07 13:30:57.384 881 TRACE neutron RuntimeError: 
  2015-05-07 13:30:57.384 881 TRACE neutron Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'delete', 'tap3da8f4fe-55']
  2015-05-07 13:30:57.384 881 TRACE neutron Exit code: 2
  2015-05-07 13:30:57.384 881 TRACE neutron Stdout: ''
  2015-05-07 13:30:57.384 881 TRACE neutron Stderr: 'RTNETLINK answers: Operation not supported\n'
  2015-05-07 13:30:57.384 881 TRACE neutron 
  ---

  It also doesn't work manually:

  ---
  root@network-1:~# ip link delete tap3da8f4fe-55
  RTNETLINK answers: Operation not supported
  ---

  dmesg shows this:

  ---
  [  152.307822] Modules linked in: vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch libcrc32c snd_hda_codec_generic iosf_mbi snd_hda_intel crct10dif_pclmul crc32_pclmul snd_hda_controller ghash_clmulni_intel snd_hda_codec aesni_intel ppdev snd_hwdep snd_pcm aes_x86_64 qxl lrw snd_timer gf128mul snd ttm glue_helper drm_kms_helper soundcore i2c_piix4 ablk_helper drm serio_raw cryptd pvpanic 8250_fintek parport_pc mac_hid lp parport psmouse pata_acpi floppy
  [  152.307893] CPU: 0 PID: 1067 Comm: ovs-vswitchd Not tainted 3.19.0-16-generic #16~14.04.2-Ubuntu
  [  152.307901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
  [  152.307906] task: ffff880036aee220 ti: ffff880079eb4000 task.ti: ffff880079eb4000
  [  152.307912] RIP: 0010:[<ffffffffc05fd901>]  [<ffffffffc05fd901>] ovs_vport_del+0x41/0x60 [openvswitch]
  [  152.307923] RSP: 0018:ffff880079eb79a8  EFLAGS: 00010296
  [  152.307929] RAX: 8000000000000025 RBX: ffff880079f04e40 RCX: 00000001810000ec
  [  152.307934] RDX: 00000001810000ed RSI: ffffea0000d860c0 RDI: 0000000040000000
  [  152.307938] RBP: ffff880079eb79b8 R08: ffff8800361834a0 R09: 00000001810000ec
  [  152.307943] R10: ffffea0000d860c0 R11: ffffffff813a7f65 R12: ffff880079f04e40
  [  152.307948] R13: ffff880079e97100 R14: 00000000ffffffea R15: ffff880079e30180
  [  152.307955] FS:  00007fc5f96a5980(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
  [  152.307961] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [  152.307966] CR2: 00007f32854bd740 CR3: 000000007a032000 CR4: 00000000000406f0
  [  152.307972] Stack:
  [  152.307975]  ffff880079eb7a30 ffff880079eb7a30 ffff880079eb79c8 ffffffffc05f6380
  [  152.307984]  ffff880079eb79f8 ffffffffc05f6455 ffffffffc05ff268 ffffffffc0602040
  [  152.307992]  ffff88007a700814 ffff880079e97000 ffff880079eb7aa8 ffffffff816db0e5
  [  152.308003] Call Trace:
  [  152.308012]  [<ffffffffc05f6380>] ovs_dp_detach_port+0x30/0x40 [openvswitch]
  [  152.308019]  [<ffffffffc05f6455>] ovs_vport_cmd_del+0xc5/0x110 [openvswitch]
  [  152.308034]  [<ffffffff816db0e5>] genl_family_rcv_msg+0x1a5/0x3d0
  [  152.308040]  [<ffffffff816db310>] ? genl_family_rcv_msg+0x3d0/0x3d0
  [  152.308046]  [<ffffffff816db3a1>] genl_rcv_msg+0x91/0xd0
  [  152.308051]  [<ffffffff816d9191>] netlink_rcv_skb+0xc1/0xe0
  [  152.308056]  [<ffffffff816d983c>] genl_rcv+0x2c/0x40
  [  152.308060]  [<ffffffff816d8816>] netlink_unicast+0xf6/0x200
  [  152.308065]  [<ffffffff816d8c33>] netlink_sendmsg+0x313/0x690
  [  152.308078]  [<ffffffff81350fe0>] ? aa_sk_perm.isra.4+0x70/0x150
  [  152.308090]  [<ffffffff8168e34c>] do_sock_sendmsg+0x8c/0x100
  [  152.308096]  [<ffffffff8168dd0e>] ? copy_msghdr_from_user+0x15e/0x1f0
  [  152.308103]  [<ffffffff8168e963>] ___sys_sendmsg+0x313/0x320
  [  152.308112]  [<ffffffff8120713c>] ? destroy_inode+0x3c/0x70
  [  152.308118]  [<ffffffff81207287>] ? evict+0x117/0x190
  [  152.308122]  [<ffffffff8120230f>] ? dentry_free+0x5f/0xa0
  [  152.308127]  [<ffffffff81202c64>] ? __dentry_kill+0x154/0x1f0
  [  152.308132]  [<ffffffff81202da5>] ? dput+0xa5/0x170
  [  152.308138]  [<ffffffff8120bdc4>] ? mntput+0x24/0x40
  [  152.308143]  [<ffffffff8168f052>] __sys_sendmsg+0x42/0x80
  [  152.308148]  [<ffffffff8168f0a2>] SyS_sendmsg+0x12/0x20
  [  152.308156]  [<ffffffff817b520d>] system_call_fastpath+0x16/0x1b
  [  152.308208] Code: 8b 57 30 48 85 c0 48 89 02 74 04 48 89 50 08 48 b8 00 02 20 00 00 00 ad de 48 89 df 48 89 43 30 48 8b 43 48 ff 50 10 48 8b 43 48 <48> 8b 78 40 e8 d6 55 af c0 48 83 c4 08 5b 5d c3 66 66 66 66 66 
  [  152.308375] RIP  [<ffffffffc05fd901>] ovs_vport_del+0x41/0x60 [openvswitch]
  [  152.308433]  RSP <ffff880079eb79a8>
  [  152.308504] ---[ end trace cc9561be41fd191a ]---
  [  217.847530] init: neutron-ovs-cleanup pre-start process (871) terminated with status 1
  [  217.913947] init: wait-for-state (neutron-dhcp-agentneutron-ovs-cleanup) main process (862) killed by TERM signal
  ---

  Falling back to Linux 3.16.

  Info:

  root@network-1:~# cat /proc/version_signature > version.log
  root@network-1:~# cat version.log 
  Ubuntu 3.19.0-16.16~14.04.2-generic 3.19.3

  Thanks,
  Thiago
  --- 
  AlsaVersion: Advanced Linux Sound Architecture Driver Version k3.19.0-16-generic.
  AplayDevices: Error: [Errno 2] No such file or directory
  ApportVersion: 2.14.1-0ubuntu3.10
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/hwC0D0', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  Card0.Amixer.info: Error: [Errno 2] No such file or directory
  Card0.Amixer.values: Error: [Errno 2] No such file or directory
  DistroRelease: Ubuntu 14.04
  HibernationDevice: RESUME=/dev/mapper/vg01-swap_1
  InstallationDate: Installed on 2015-04-09 (27 days ago)
  InstallationMedia: Ubuntu-Server 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
  IwConfig: Error: [Errno 2] No such file or directory
  Lsusb:
   Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
   Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
   Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
   Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
  MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
  Package: linux (not installed)
  ProcFB: 0 qxldrmfb
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.19.0-16-generic root=/dev/mapper/vg01-root ro
  ProcVersionSignature: Ubuntu 3.19.0-16.16~14.04.2-generic 3.19.3
  RelatedPackageVersions:
   linux-restricted-modules-3.19.0-16-generic N/A
   linux-backports-modules-3.19.0-16-generic  N/A
   linux-firmware                             1.127.11
  RfKill: Error: [Errno 2] No such file or directory
  Tags:  trusty
  Uname: Linux 3.19.0-16-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups:
   
  _MarkForUpload: True
  dmi.bios.date: 01/01/2011
  dmi.bios.vendor: Bochs
  dmi.bios.version: Bochs
  dmi.chassis.type: 1
  dmi.chassis.vendor: Bochs
  dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-trusty:cvnBochs:ct1:cvr:
  dmi.product.name: Standard PC (i440FX + PIIX, 1996)
  dmi.product.version: pc-i440fx-trusty
  dmi.sys.vendor: QEMU

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1452868/+subscriptions
References

[Bug 1452868] [NEW] "ip link delete" returns "RTNETLINK answers: Operation not supported" - OpenStack Juno is completely broken in Trusty + Linux 3.19.
From: Thiago Martins, 2015-05-07