touch-packages team mailing list archive

[Bug Watch Updater <1481388@xxxxxxxxxxxxxxxxxx>]

Launchpad has imported 8 comments from the remote bug at
http://bugs.ntp.org/show_bug.cgi?id=2224.

If you reply to an imported comment from within Launchpad, your comment
will be sent to the remote bug automatically. Read more about
Launchpad's inter-bugtracker facilities at
https://help.launchpad.net/InterBugTracking.

------------------------------------------------------------------------
On 2012-06-11T13:37:37+00:00 Ktamateas wrote:

Hello everyone, I want to file a bug.

Run command: ntpd --user=ntpd:ntpd --logfile=/var/log/ntpd.log

It runs for some seconds and then is segfaults.
Happens only when I use both the --user and --logfile parameters.
Happens only when I have configured it with --enable-clockctl alone.
If configured with both --enable-clockctl and --enable-linuxcaps it works OK.

/var/log/ntpd.log: -rw-r--r-- 1 ntpd ntpd 21957 Jun 11 14:49 /var/log/ntpd.log
/etc/passwd: ntpd:x:10:17:ntpd:/dev/null:/bin/false
/etc/group: ntpd:x:1008:
/etc/ntpd.conf is empty.

Strace gives in the end: http://pastebin.com/Bujn2MNn
With more advanced debugging I got: http://pastebin.com/YNWBrRJG
When runs in normal manner, strace gives: http://pastebin.com/2JpzK4jh

In my humble opinion, the error occurs when ntpd tries to do something
with the network interfaces.

My machine is a kernel 2.6.35.14 with glibc 2.14.1. 
ntpd - NTP daemon program - Ver. 4.2.6p5

Greetings.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/0

------------------------------------------------------------------------
On 2012-06-11T17:10:44+00:00 Dave Hart wrote:

Thanks for the report.  The additional debugging paste is short enough
to include directly in the comments:

==24767== Invalid read of size 8
==24767==    at 0x411048: input_handler (ntp_io.c:3621)
==24767==    by 0x414B84: ntpdmain (ntpd.c:1078)
==24767==    by 0x406448: main (ntpd.c:356)
==24767==  Address 0x5e897f0 is 0 bytes inside a block of size 32 free'd
==24767==    at 0x4C26649: free (in /lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==24767==    by 0x411072: input_handler (ntp_io.c:3619)
==24767==    by 0x414B84: ntpdmain (ntpd.c:1078)
==24767==    by 0x406448: main (ntpd.c:356)

The code in question is:


#ifdef HAS_ROUTING_SOCKET
	/*
	 * scan list of asyncio readers - currently only used for routing sockets
	 */
	asyncio_reader = asyncio_reader_list;

	while (asyncio_reader != NULL) {
		if (FD_ISSET(asyncio_reader->fd, &fds)) {
			++select_count;
			(asyncio_reader->receiver)(asyncio_reader); /*3619 */
		}
		asyncio_reader = asyncio_reader->link; /* 3621 */
	}
#endif /* HAS_ROUTING_SOCKET */

line 3619 is calling process_routing_msgs() which, after root is
dropped, is noticing a failed read or other error and removing the entry
from asyncio_reader_list and free()ing it, triggering the valgrind
catch.

I bet can be worked around by adding -U 0 to the command line to disable
dynamic interface updates, I suspect (I could be wrong, too).  To patch
it, we need to add a "next_asyncio_reader" local variable of the same
type as asyncio_reader, and assign to it asyncio_reader->link before if
(FD_ISSET(..., and change the asyncio_reader assignment to use the saved
next_asyncio_reader.  I will get that ready for ntp-dev, and am
requesting 4.2.6 blocking in case we do another release of that stable
version.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/1

------------------------------------------------------------------------
On 2012-06-11T18:12:06+00:00 Dave Hart wrote:

Ready in:

~hart/ntp-dev-2224

Reply at:
https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/2

------------------------------------------------------------------------
On 2012-06-12T06:23:04+00:00 Ktamateas wrote:

(In reply to comment #2)
> Ready in:
> 
> ~hart/ntp-dev-2224

How can I see the code difference?

Reply at:
https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/3

------------------------------------------------------------------------
On 2012-06-12T07:56:53+00:00 Dave Hart wrote:

Created attachment 883
pending patch for Bug 2224

With a bit of luck it'll be in 4.2.7p280 before too long.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/4

------------------------------------------------------------------------
On 2012-06-12T08:24:23+00:00 Ktamateas wrote:

(In reply to comment #4)
> Created attachment 883 [details]
> pending patch for Bug 2224
> 
> With a bit of luck it'll be in 4.2.7p280 before too long.

We consider 4.2.7p XXX stable revisions ?

Reply at:
https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/5

------------------------------------------------------------------------
On 2012-06-12T08:50:50+00:00 Dave Hart wrote:

(In reply to comment #5)
> (In reply to comment #4)
> > Created attachment 883 [details]
> > pending patch for Bug 2224
> > 
> > With a bit of luck it'll be in 4.2.7p280 before too long.
> 
> We consider 4.2.7p XXX stable revisions ?

Some of we do.  I've always preferred ntp-dev snapshots to -stable
releases as the -stable releases are years apart and a lot of
interesting changes happen in between.

Speaking of which, it's been 2.5 years since 4.2.7 forked from 4.2.6.
I'm anxious, as are others, to get 4.2.8 out as the next stable release,
rather than spend more time patching 4.2.6.  Given the relatively small
impact of this bug, I'm still inclined to prefer avoiding another 4.2.6
release.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/6

------------------------------------------------------------------------
On 2012-06-13T05:18:00+00:00 Stenn wrote:

nosebleed,

Thanks for the report.  Please check 4.2.7p280 and mark this bug as
VERIFIED or REOPENED, as appropriate.

If there is another 4.2.6pX release this patch will be backported.

Dave, thanks for your work on this.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/1481388/comments/7


** Changed in: ntp
       Status: Unknown => Fix Released

** Changed in: ntp
   Importance: Unknown => High

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to ntp in Ubuntu.
https://bugs.launchpad.net/bugs/1481388

Title:
  NTP : Use-after-free in routing socket code after dropping root

Status in NTP:
  Fix Released
Status in ntp package in Ubuntu:
  Fix Released
Status in ntp source package in Precise:
  In Progress
Status in ntp source package in Trusty:
  In Progress
Status in ntp source package in Vivid:
  In Progress
Status in ntp source package in Wily:
  Fix Released
Status in ntp package in Debian:
  Unknown

Bug description:
  [Impact]

   * User experienced repeated segfaults at the same instruction pointer

  i/o error on routing socket No buffer space available - disabling
  segfault at 31 ip 0000000000000031 sp 00007ffff9f11788 error 14 in libpthread-2.15.so[7f967a5d9000+18000] 

  The remove_ and delete_ functions remove the current element from the
  asyncio_reader_list, and free it, respectively.

  We then return back to the loop at the top, wherein the asyncio_reader variable still points at the now-freed element, whose contents are (in theory) now scrambled
  by having link pointers, etc, from internal malloc state overlaying the data.

  [Test Case]

  You can easily reproduce the bug by :

  - Lowering the sysctl value net.core.rmem_max

  $ sysctl -w net.core.wmem_max=<LOWER_VALUE>
  This sets the max OS send buffer size for all types of connections.

  - Adding multiple network interfaces and static routes.

  [Regression Potential]

  None expected since the fix is already available upstream
  (https://github.com/ntp-project/ntp.git) and Debian package.

  If after installing the patch, user are receiving this kind of message in /var/log/syslog : "routing socket reports: No buffer space available".
  The next step, would be to increase the "net.core.rmem_max" and "net.core.wmem_max" values equally until the "routing socket reports: No buffer space available" message no longer showed up.

  [Other Info]

  NTP upstream (https://github.com/ntp-project/ntp.git)
  [Bug 2224] Use-after-free in routing socket code after dropping root. - Commit: d6df9d3
  [Bug 2890] Ignore ENOBUFS on routing netlink socket. - Commit: db47bd4 

  The use-after-free bug has been fix in Debian release (closes: #795315)
  Will submit the ignore-ENOBUFS-on-routing-netlink-socket in Debian in the next days.

  [Original Description]

  We have 1 server (among hundreds) that its ntp service is crashing.

  A few minute/seconds after a start attempts we can see the following in syslog:
  ntpd[2729]: peers refreshed
  ntpd[2729]: Listening on routing socket on fd #49 for interface updates
  ntpd[2729]: i/o error on routing socket No buffer space available - disabling
  kernel: [157516.495224] ntpd[2729]: segfault at 31 ip 0000000000000031 sp 00007ffff9f11788 error 14 in libpthread-2.15.so[7f967a5d9000+18000]

  OS: Ubuntu 12.04.4 LTS
  Kernel: 3.11.0-19-generic

  I tried to compare it to other servers, and the only thing I could find that is different is that while it's up (before it crashes) I can see the following when running "lsof | grep ntp":
  ntpd 2729 ntp 49u sock 0,7 0t0 2473952565 can't identify protocol.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ntp/+bug/1481388/+subscriptions