maria-discuss team mailing list archive

["Scott A. Wozny" <sawozny@xxxxxxxxxxx>]

I’m switching over from Percona XtraDB Cluster 5.7 to MariaDB 10.3 with Galera support on CentOS 7.  During this process things have gone mostly smoothly, but I’ve found a lot of SELinux AVC failures once I started testing.  Some googling later and I found this article which describes the heart of my issue and it appears to center around a fix that was put into wsrep_sst_xtrabackup-v2 in the Percona repositories, but still exists in wsrep_sst_mariabackup (which, I guess, was ported AFTER Percona made their changes).

https://dzone.com/articles/lock-down-enforcing-selinux-with-percona-xtradb-cl

For the TL;DR set, the crux is this.  In the wsrep_sst_mariabackup script, the wait_for_listen function scans all the processes, which upsets SELinux who either forbids or logs the scan of every listening process on the system that mysqld_t doesn’t have rights to do a process getattr on.  Solutions to this include runing SELinux in permission mode or allowing a process getattr from mysqld_t to every type attached to a listening process, neither of which is appealing.

So where MariaDB 10.3 wsrep_sst_mariabackup calls wait_for_listen() like this:

wait_for_listen ${SST_PORT} ${ADDR} ${MODULE} &

and executes this function:

# waits ~10 seconds for nc to open the port and then reports ready
# (regardless of timeout)
wait_for_listen()
{
    local PORT=$1
    local ADDR=$2
    local MODULE=$3
    for i in {1..50}
    do
if [ "$OS" = "FreeBSD" ];then
            sockstat -46lp $PORT | grep -qE "^[^ ]* *(socat|nc) *[^ ]* *[^ ]* *[^ ]* *[^ ]*:$PORT" &&$
        else
            ss -p state listening "( sport = :$PORT )" | grep -qE 'socat|nc' && break
        fi
        sleep 0.2
    done
    echo "ready ${ADDR}/${MODULE}//$sst_ver"
}

Which causes SELinux to light up like a Christmas tree.

PXC 5.7’s wsrep_sst_xtrabackup-v2 calls wait_for_listen() like this:

    # Note: this is started as a background process
    # So it has to wait for processes that are started by THIS process
    wait_for_listen $$ ${WSREP_SST_OPT_HOST} ${WSREP_SST_OPT_PORT:-4444} ${MODULE} &

and executes this function:

# waits ~1 minute for nc/socat to open the port and then reports ready
# (regardless of timeout)
#
# Assumptions:
#   1. The socat/nc processes do not launch subprocesses to handle
#      the connections.  Note that socat can be configured to do so.
#   2. socat is not bound to a specific interface/address, so the
#      IP portion of the local address is all zeros (0000...000).
#
# Parameter 1: the pid of the wsrep_sst_xtrabackup-v2 process. We are looking
#              for the socat process that was started by this script.
# Parameter 2: the IP address of the SST host
# Parameter 3: the Port of the SST host
# Parameter 4: a descriptive name for what we are doing at this point
#
wait_for_listen()
{
    local parentpid=$1
    local host=$2
    local port=$3
    local module=$4

    #
    # Check to see if the OS supports /proc/<pid>/net/tcp
    #
    if [[ ! -r /proc/$$/net/tcp && ! -r /proc/$$/net/tcp6 ]]; then
        wsrep_log_debug "$LINENO: Using ss for socat/nc discovery"

        # Revert to using ss to check if socat/nc is listening
        wsrep_check_program ss
        if [[ $? -ne 0 ]]; then
            wsrep_log_error "******** FATAL ERROR *********************** "
            wsrep_log_error "* Could not find 'ss'.  Check that it is installed and in the path."
            wsrep_log_error "******************************************** "
            return 2
        fi

        for i in {1..300}
        do
            ss -p state listening "( sport = :${port} )" | grep -qE 'socat|nc' && break
            sleep 0.2
        done

        echo "ready ${host}:${port}/${module}//$sst_ver"
        return 0
    fi

    wsrep_log_debug "$LINENO: Using /proc/pid/net/tcp for socat/nc discovery"

    # Get the index for the 'local_address' column in /proc/xxxx/net/tcp
    # We expect this to be the same for IPv4 (net/tcp) and IPv6 (net/tcp6)
    local ip_index=0
    local header
    if [[ -r /proc/$$/net/tcp ]]; then
        read -ra header <<< $(head -n 1 /proc/$$/net/tcp)
    elif [[ -r /proc/$$/net/tcp6 ]]; then
        read -ra header <<< $(head -n 1 /proc/$$/net/tcp6)
    else
        wsrep_log_error "******** FATAL ERROR *********************** "
        wsrep_log_error "* Cannot find /proc/$$/net/tcp (or tcp6)"
        wsrep_log_error "******************************************** "
        exit 1
    fi
    for i in "${!header[@]}"; do
        if [[ ${header[$i]} = "local_address" ]]; then
            # Add one to the index since arrays are 0-based
            # but awk is 1-based
            ip_index=$(( i + 1 ))
            break
        fi
    done
    if [[ $ip_index -eq 0 ]]; then
        wsrep_log_error "******** FATAL ERROR *********************** "
        wsrep_log_error "* Unexpected /proc/xx/net/tcp layout: cannot find local_address"
        wsrep_log_error "******************************************** "
        exit 1
    fi

    wsrep_log_debug "$LINENO: local_address index is $ip_index"
    local port_in_hex
    port_in_hex=$(printf "%04X" $port)

    local user_id
    user_id=$(id -u)

    for i in {1..300}
    do
        # List all socat/nc processes started by the user of this script
        # Then look for processes that have the script pid as a parent prcoess
        # somewhere in the process tree

        wsrep_log_debug "$LINENO: Entering loop body : $i"

        # List only socat/nc processes started by this user to avoid triggering SELinux
        for pid in $(ps -u $user_id -o pid,comm | grep -E 'socat|nc' | awk '{ printf $1 " " }')
        do
            if [[ -z $pid || $pid = " " ]]; then
                continue
            fi

            wsrep_log_debug "$LINENO: Examining pid: $pid"

            # Now get the processtree for this pid
            # If the parentpid is NOT in the process tree, then ignore
            if ! echo "$(get_parent_pids $pid)" | grep -qw "$parentpid"; then
                wsrep_log_debug "$LINENO: $parentpid is not in the process tree: $(get_parent_pids $pid)"
                continue
            fi

            # get the sockets for the pid
            # Note: may not need to get the list of sockets, is it ok to
            # just look at the list of local addresses in tcp?
            local sockets
            sockets=$(ls -l /proc/$pid/fd | grep socket | cut -d'[' -f2 | cut -d ']' -f1 | tr '\n' '|')

            # remove the trailing '|'
            sockets=${sockets%|}
            wsrep_log_debug "$LINENO: sockets: $sockets"

            if [[ -n $sockets ]]; then
                # For the network addresses, we expect to be listening
                # on all interfaces, thus the address should be
                # 00..000:PORT (all zeros for the IP address).

                # Dumping the data in the lines
                #if [[ -n "$WSREP_LOG_DEBUG" ]]; then
                #    lines=$(grep -E "\s(${sockets})\s" /proc/$pid/net/tcp)
                #    if [[ -n $lines ]]; then
                #        while read -r line; do
                #            if [[ -z $line ]]; then
                #                continue
                #            fi
                #            wsrep_log_debug "$LINENO: $line"
                #        done <<< "$lines\n"
                #    fi
                #fi

                # Checking IPv4
                if grep -E "\s(${sockets})\s" /proc/$pid/net/tcp |
                        awk "{print \$${ip_index}}" |
                        grep -q "^00*:${port_in_hex}$"; then
                    wsrep_log_debug "$LINENO: found a match for pid: $pid"
                    break 2
                fi

                # Also check IPv6
                if grep -E "\s(${sockets})\s" /proc/$pid/net/tcp6 |
                        awk "{print \$${ip_index}}" |
                        grep -q "^00*:${port_in_hex}$"; then
                    break 2
                fi
            fi
        done

        sleep 0.2
    done

    wsrep_log_debug "$LINENO: wait_for_listen() exiting"
    echo "ready ${host}:${port}/${module}//$sst_ver"
    return 0
}

which does NOT light up SELinux like a Christmas tree (but DOES still require a few SELinux policy additions to run in enforcing mode).

Obviously the PXC function code isn’t directly compatible with the Mariabackup sst script since at the very least it’s not using the same parameters, but can I just change the calling line AND the function itself in my version of wsrep_sst_mariabackup and will it work, or is it more complicated than that?

There are also other SELinux AVC failures that I’m prepared to write custom policy for, but I’d like to “patch” the wait_for_listen function in wsrep_sst_mariabackup so I don’t have to remember every time I add a listening daemon on the server that I need to rebuild my custom SELinux policy.  If the author of the article is correct, the custom SELinux policy allows needed OUTSIDE the noisy listening port scan should be pretty static and only need a re-check on major version upgrades.

So, I know this was a lot and if you think I should take this to the devs list, please let me know.  I was hoping there might be a powerful enough user on this list who could suggest the code changes needed in wsrep_sst_mariabackup to get me around the most onerous part of this, OR whatever changes I might need to make outside of drag-and-drop.

Thanks,

Scott