kernel-packages team mailing list archive

[Rafael David Tinoco <rafael.tinoco@xxxxxxxxxxxxx>]

Okay, so, I had more time to dig a bit into this and, after some
analysis, I got:

Errors being reproduced:

[1668392.078137] audit: type=1400 audit(1459311786.129:1375455):
apparmor="DENIED" operation="sendmsg" info="Failed name lookup -
disconnected path" error=-13 profile="/usr/sbin/dnsmasq" name="dev/log"
pid=15735 comm="dnsmasq" requested_mask="w" denied_mask="w" fsuid=0
ouid=0

And apparmor dnsmasq profile:

#/usr/sbin/dnsmasq flags=(attach_disconnected) {
#/usr/sbin/dnsmasq flags=(complain) {
/usr/sbin/dnsmasq {

Without any flags.

And the command causing the apparmor errors:

root     16877  0.0  0.2  66416  3648 ?        S    13:23   0:00 sudo
/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec
qdhcp-37d013b6-f6fa-4652-8073-5e7d2c418a9d env
NEUTRON_NETWORK_ID=37d013b6-f6fa-4652-8073-5e7d2c418a9d dnsmasq --no-
hosts --no-resolv --strict-order --bind-interfaces --interface=ns-
aa95fe20-ff --except-interface=lo --pid-
file=/var/lib/neutron/dhcp/37d013b6-f6fa-4652-8073-5e7d2c418a9d/pid
--dhcp-
hostsfile=/var/lib/neutron/dhcp/37d013b6-f6fa-4652-8073-5e7d2c418a9d/host
--addn-
hosts=/var/lib/neutron/dhcp/37d013b6-f6fa-4652-8073-5e7d2c418a9d/addn_hosts
--dhcp-
optsfile=/var/lib/neutron/dhcp/37d013b6-f6fa-4652-8073-5e7d2c418a9d/opts
--dhcp-
leasefile=/var/lib/neutron/dhcp/37d013b6-f6fa-4652-8073-5e7d2c418a9d/leases
--dhcp-range=set:tag0,192.168.21.0,static,86400s --dhcp-lease-max=256
--conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal

It is a "sudo-like" approach from openstack (rootwrap) to execute
dnsmasq in a new network namespace with different privileges.

Ubuntu kernel 3.13.X has apparmor 3 alpha 6 code: https://pastebin.canonical.com/152812/
Ubuntu kernel 3.16 and 3.19 has apparmor 3 rc 1 code: https://pastebin.canonical.com/152813/

>From apparmor I could see that the error comes from "aa_path_name"
called by either:

- path_name *
- aa_remount
- aa_bind_mount
- aa_mount_change_type
- aa_move_mount
- aa_new_mount
- aa_unmount
- aa_pivotroot

So, since the job is being restarted by neutron (or at least it is
trying to re-start it, causing the apparmor to block the access), I
created a systemtap script to monitor path_name and check for dnsmasq
trying to open "log" (allegedly /dev/log) file.

probe kernel.function("path_name").call {
	funcname = execname();
	if (funcname == "dnsmasq") {
		filename = reverse_path_walk($path->dentry);
		if (filename == "log") {
			printf("(%s) %s\n", execname(), filename);
			print_backtrace();
		}
	}
}

And got the backtrace from the denials:

(dnsmasq) log

 0xffffffff8132deb0 : path_name+0x0/0x140 [kernel]
 0xffffffff8132e413 : aa_path_perm+0xa3/0x130 [kernel]
 0xffffffff81337e26 : aa_unix_peer_perm+0x536/0x990 [kernel]
 0xffffffff8132c653 : apparmor_unix_may_send+0x73/0x150 [kernel]
 0xffffffff812eb8a6 : security_unix_may_send+0x16/0x20 [kernel]
 0xffffffff817019db : unix_dgram_connect+0x23b/0x250 [kernel]
 0xffffffff8164a987 : SYSC_connect+0xe7/0x120 [kernel]
 0xffffffff8164b68e : sys_connect+0xe/0x10 [kernel]
 0xffffffff817700cd : system_call_fastpath+0x1a/0x1f [kernel]

 When trying to check if "log" could be converted to "fullpath" by using
systemtap function:

 	return task_dentry_path(task_current(),
		@cast(path,"path","kernel:nfs:kernel<linux/path.h>")->dentry,
		@cast(path,"path","kernel:nfs:kernel<linux/path.h>")->mnt)

I saw that I could resolve path for all other files but "/dev/log":

(dnsmasq) /usr/lib/x86_64-linux-gnu/libnfnetlink.so.0.2.0
(dnsmasq) /usr/lib/x86_64-linux-gnu/libmnl.so.0.1.0
(dnsmasq) /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
(dnsmasq) /etc/neutron/dnsmasq.conf
(dnsmasq) /etc/neutron/dnsmasq.conf
(dnsmasq) /etc/localtime
(dnsmasq) /etc/localtime
(dnsmasq) /etc/localtime
(dnsmasq) <unknown>

Because

    function task_dentry_path:string(task:long,dentry:long,vfsmnt:long)

Couldn't handle the udev (vfsmnt) path. With that, I thought apparmor
couldn't be handling udev path for different root vfsmnt (like a
different FS namespace).

Checking iproute2 ipnetns.c I saw that executing a command in a new
network namespace causes:

unshare(CLONE_NEWNS) AND
mount("", "/", "none", MS_SLAVE | MS_REC, NULL)

This second being possibly the "problem" - its intent is not to allow
mounts to propagate back to parent task - since it basically removes a
root vfsmnt from the execution taken in place, breaking apparmor (or
even dentry cache) path resolution logic. Here, the needed flag
"attach_disconnected" (present in apparmor path resolution logic) comes
in place, to allow "/" to be added in path names missing leading / due
to missing root vfsmnt from dentry cache.

To observe: 3.13 (apparmor 3 alpha 6) doesn't complain on disconnected
paths but 3.16 and beyond (rc1) does (needing the flag
"attach_disconnected").

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1373070

Title:
  full fix for disconnected path (paths)

Status in cups package in Ubuntu:
  Fix Released
Status in linux package in Ubuntu:
  Triaged
Status in rsyslog package in Ubuntu:
  New

Bug description:
  With the apparmor 3 RC1 upload, there is an incomplete bug fix for
  disconnected paths. This bug is to track that work.

  This denial may be related:
  Sep 23 10:10:50 localhost kernel: [40262.517799] audit: type=1400 audit(1411485050.722:2862): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/rsyslogd" name="dev/log" pid=7011 comm="logger" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

  This is related to bug 1375410

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cups/+bug/1373070/+subscriptions