kernel-packages team mailing list archive

[Tero Marttila <terom@xxxxxxxx>]

Attached is a small script to setup dnsmasq-tftp and ipvs to reproduce
the dnsmasq-tftp stall. The strace at the end should start showing
sendto() = EPERM errors after running the curl tftp://.../ command given
as an example from a remote host.

The exact contents of the ipvs rules don't seem to matter.

The dnsmasq-tftp stall issue appears to happen with all the ubuntu-
installer/amd64/linux versions that I've tried, with the exact block
within the TFTP stream where it stalls changing betwen different
versions but remaining consistent between runs.

With the current netboot.tar.gz it appers to happen at block 600:

d4fc2ba26cad96f4360cb36f04c26b2f50dceee9413c206431148b87e4958909
/srv/tftp/ubuntu-installer/amd64/linux

With the original ubuntu-installer netboot.tar.gz version (from sometime
early June) I've been using it happens at block 311:

bdebc6c7bbd873bf5ad2480d90c2523da49cca8da5a0f4cb40adb7d1aeafb821
/srv/tftp/lp1349768/linux

Curiously, however, testing this setup on the 3.13.0-34-generic kernel
the dmesg error ("IPv6 header not found") does NOT show up... may be
some other facters at play as well there?

** Attachment added: "minimal testcase for LP#1349768 on Ubuntu 14.04"
   https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1349768/+attachment/4182427/+files/lp1349768.sh

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768

Title:
  kernel 3.13.0-32 ipvs "IPv6 header not found" related to UDP socket
  sendto() EPERM errors

Status in “linux” package in Ubuntu:
  Confirmed
Status in “linux” source package in Trusty:
  In Progress
Status in “linux” source package in Utopic:
  Confirmed

Bug description:
  I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
  loadbalancer and dnsmasq server for pxebooting servers.

  After updating linux-image 3.13.0-29.53 -> 3.13.0-32.57 I noticed that
  dnsmasq-tftp stopped working. pxeboot clients would hang on the
  "Loading ..../linux" TFTP transfer, with the transfer stalling roughly
  ~1000 blocks into the transfer:

  10:30:51.011728 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412
  10:30:51.011924 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4
  10:30:51.012012 IP 10.1.1.2.43540 > 10.1.12.1.49165: UDP, length 1412
  10:30:51.012183 IP 10.1.12.1.49165 > 10.1.1.2.43540: UDP, length 4

  stracing dnsmasq I noticed something very odd: sendto() on the
  socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
  persistently returning EPERM in mid-transfer, even when dnsmasq
  continued to periodically retry:

  select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249834})
  recvfrom(17, "\0\4\3\352", 4096, 0, NULL, NULL) = 4
  lseek(16, 1410816, SEEK_SET) = 1410816
  read(16, "\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v"..., 1408) = 1408
  sendto(17, "\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = 1412
  select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 1 (in [17], left {0, 249839})
  recvfrom(17, "\0\4\3\353", 4096, 0, NULL, NULL) = 4
  lseek(16, 1412224, SEEK_SET) = 1412224
  read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408
  sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted)
  select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout)
  select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout)
  select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout)
  select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout)
  select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout)
  select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout)
  select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout)
  select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 250000}) = 0 (Timeout)
  lseek(16, 1412224, SEEK_SET) = 1412224
  read(16, "*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277\221\\\307\372"..., 1408) = 1408
  sendto(17, "\0\3\3\354*\360 <C\363l\320:\256~\307\236\26P\323\274%\260\362\341&\232\r\243\370\224\277"..., 1412, 0, {sa_family=AF_INET, sin_port=htons(49165), sin_addr=inet_addr("10.1.11.3")}, 16) = -1 EPERM (Operation not permitted)

  This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
  apparmor profiles torn down.

  I also noticed the following dmesgs appearing at roughly similar times
  to the tftp transfers getting stuck (although not coinciding exactly
  with the stall):

  [70325.516724] IPv6 header not found

  The error pointed to ipvs (which I am using on the same host as an IPv4 NAT loadbalancer):
  http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
  http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614

  I then tore down the ipvs rules (service keepalived stop) and unloaded
  the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
  the stalled dnsmasq-tftp transfer resumed!

  This seems to be reproducible, i.e. modprobing ip_vs and starting
  keepalived will cause dnsmasq-tftp to stall again, and
  stopping/unloading will resume.

  This seems to happen reproducibly on boot with -32 and -30. This does NOT seem to happen with 3.13.0-29 which I was using up until now.
  --- 
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Jul 29 13:43 seq
   crw-rw---- 1 root audio 116, 33 Jul 29 13:43 timer
  AplayDevices: Error: [Errno 2] No such file or directory
  ApportVersion: 2.14.1-0ubuntu3.2
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: Error: [Errno 2] No such file or directory
  DistroRelease: Ubuntu 14.04
  HibernationDevice: RESUME=/dev/mapper/catcp2-swap
  InstallationDate: Installed on 2014-06-03 (56 days ago)
  InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.2)
  MachineType: Dell Inc. PowerEdge R410
  Package: linux-image-3.13.0-32-generic 3.13.0-32.57
  PackageArchitecture: amd64
  PciMultimedia:
   
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   LANG=C.UTF-8
   SHELL=/bin/bash
  ProcFB:
   
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-32-generic root=/dev/mapper/hostname-root ro console=ttyS1,115200n8 console=tty0 nomdmonddf nomdmonisw
  ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4
  RelatedPackageVersions:
   linux-restricted-modules-3.13.0-32-generic N/A
   linux-backports-modules-3.13.0-32-generic  N/A
   linux-firmware                             1.127.5
  RfKill: Error: [Errno 2] No such file or directory
  Tags:  trusty
  Uname: Linux 3.13.0-32-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups:
   
  _MarkForUpload: True
  dmi.bios.date: 07/30/2013
  dmi.bios.vendor: Dell Inc.
  dmi.bios.version: 1.12.0
  dmi.board.name: 01V648
  dmi.board.vendor: Dell Inc.
  dmi.board.version: A03
  dmi.chassis.type: 23
  dmi.chassis.vendor: Dell Inc.
  dmi.modalias: dmi:bvnDellInc.:bvr1.12.0:bd07/30/2013:svnDellInc.:pnPowerEdgeR410:pvr:rvnDellInc.:rn01V648:rvrA03:cvnDellInc.:ct23:cvr:
  dmi.product.name: PowerEdge R410
  dmi.sys.vendor: Dell Inc.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1349768/+subscriptions