maria-discuss team mailing list archive

["Scott A. Wozny" <sawozny@xxxxxxxxxxx>]

In case anyone comes across this issue in the future (although hopefully someone on the dev team sees this and just fixes it in future builds), I fixed the excessive AVC failures related to noisy ss scanning after some digging into the wsrep_sst scripts.  Here are my notes and steps taken.

My biggest problem was I didn’t know where or how the wsrep_sst code was called by Galera, so I could only make educated guesses at the values of the parameters used in both the “good” version of the wait_for_listen function and the “bad” version of the wait_for_listen function.  I downloaded the yum repo pakages from http://repo.percona.com/yum/release/7/RPMS/x86_64/Percona-XtraDB-Cluster-server-57-5.7.29-31.43.1.el7.x86_64.rpm and http://yum.mariadb.org/10.3/centos7-amd64/rpms/MariaDB-server-10.3.21-1.el7.centos.x86_64.rpm and looked at the wsrep_sst_common scripts in both and the wsrep_sst_xtrabackup-v2 and wsrep_sst_mariabackup in their respective packages.

I first looked at the calling parameters for wait_for_listen.

In the Percona version (with the version of the function that doesn’t enrage SELinux) the parameters are My Pid ($$ which I can use in anywhere) WSREP_SST_OPT_HOST (which is either explicitly provided with the --host parameter or populated in wsrep_sst_common script with the host portion of the --address parameter; in my cases these are probably going to be the same, unless I start poking the bear) , WSREP_SST_OPT_PORT (defaulting to TCP/4444, if not provided in the --address parameter) and MODULE.

In the MariaBackup version (that I want to make the Percona wait_for_listen code work in) they are SST_PORT (which is either passed in or set to 4444, by default in wsrep_sst_common), ADDR (which is set right before the call to the value of WSREP_SST_OPT_ADDR which is, in turn set by the --address parameter in the call, making it functionally like the use of HOST in the Percona version) and MODULE, same as the call in the Percona version.

So, I think I can replace the calling line AND the wait_for_listen function code in wsrep_sst_mariabackup with the Percona versions and it should work.  At least as far as inputs are concerned.

After assessing the gazintas, I needed to see if there was a significant change in the gazoutas between these functions.  The MaraiBackup version provides output in the form of a “echo "ready ${ADDR}/${MODULE}//$sst_ver"” and the Percona version does output in the form of a “echo "ready ${host}:${port}/${module}//$sst_ver"” PLUS a return 0 on success.  These look pretty close to each other, so my first instinct is to leave them alone, but not knowing what’s consuming this output (if anything is consuming it at all, it may just be there for aesthetic reasons), I can’t say for sure.  I’ll guess I’ll just have to see how it goes.

Now that the crunchy outside has been addressed, I needed to take a look at the soft, gooey center.  I went through the Percona wait_for_listen function line by line looking for anything that isn’t 1) general bash code, including common OS calls; 2) using a passed in variable (addressed above); 3) using a locally declared variable.  The first thing I noted was there were a variety of wsrep_log_debug function calls and while that’s a defined function in Percona’s version of wsrep_sst_common it is NOT in the MariaDB version.  This left me with 3 main choices.  1) Add a copy of Percona’s function to my copy of wsrep_sst_common, 2) change all the wsrep_log_debug calls to wsrep_log_info calls or 3) comment out / delete the calls to wsrep_log_debug in wait_for_listen.  I decided to do the third choice for simplicity’s sake.  Next was the call to wsrep_check_program defined in wsrep_sst_common.  While not identical, they seemed close enough so I elected to not touch anything.  Then I encountered the function get_parent_pids which exists in wsrep_sst_xtrabackup-v2 (right above the wait_for_listen function) but not in wsrep_sst_mariabackup.  Assessing this function by the same criteria as wait_for_listen I found no onward dependencies so I think I should be able to drag and drop it with the wait_for_listen function code.

So, now I had my battle plan.  I replaced the single wait_for_listen call in the main body of the wsrep_sst_mariabackup script with the one from Percona, I replaced the wait_for_listen function in wsrep_sst_mariabackup with both the get_parent_pids AND wait_for_listen function code from the Percona script and I deleted the calls to wsrep_debug_log in the copied code.

Lo and behold, it worked.  I’d like to think chance favours the prepared mind, but I just might have gotten lucky.  After clearing audit logs and forcing an SST with mariabackup, I got a MUCH more limited set of audit2allow output which I’ll add to a custom policy on my MariaDB servers.

#============= mysqld_t ==============
allow mysqld_t self:process setpgid;
allow mysqld_t self:unix_stream_socket connectto;
allow mysqld_t tmpfs_t:lnk_file read;

Anyway, if there are any devs watching this list, maybe this is an easy fix for wsrep_sst_mariabackup.  I’m going to set my systems management system to watch for changes in case other fixes (that aren’t this) are put in that I need to revise again later to keep SST working, should I need it.  Otherwise if someone finds themselves in this position in the future, this might make the fix a little more straightforward.

Thanks,

Scott

________________________________
From: Maria-discuss <maria-discuss-bounces+sawozny=hotmail.com@xxxxxxxxxxxxxxxxxxx> on behalf of Scott A. Wozny <sawozny@xxxxxxxxxxx>
Sent: May 20, 2020 7:25 PM
To: maria-discuss@xxxxxxxxxxxxxxxxxxx <maria-discuss@xxxxxxxxxxxxxxxxxxx>
Subject: [Maria-discuss] Making wsrep_sst_mariabackup more SELinux friendly

I’m switching over from Percona XtraDB Cluster 5.7 to MariaDB 10.3 with Galera support on CentOS 7.  During this process things have gone mostly smoothly, but I’ve found a lot of SELinux AVC failures once I started testing.  Some googling later and I found this article which describes the heart of my issue and it appears to center around a fix that was put into wsrep_sst_xtrabackup-v2 in the Percona repositories, but still exists in wsrep_sst_mariabackup (which, I guess, was ported AFTER Percona made their changes).

https://dzone.com/articles/lock-down-enforcing-selinux-with-percona-xtradb-cl

For the TL;DR set, the crux is this.  In the wsrep_sst_mariabackup script, the wait_for_listen function scans all the processes, which upsets SELinux who either forbids or logs the scan of every listening process on the system that mysqld_t doesn’t have rights to do a process getattr on.  Solutions to this include runing SELinux in permission mode or allowing a process getattr from mysqld_t to every type attached to a listening process, neither of which is appealing.

So where MariaDB 10.3 wsrep_sst_mariabackup calls wait_for_listen() like this:

wait_for_listen ${SST_PORT} ${ADDR} ${MODULE} &

and executes this function:

# waits ~10 seconds for nc to open the port and then reports ready
# (regardless of timeout)
wait_for_listen()
{
    local PORT=$1
    local ADDR=$2
    local MODULE=$3
    for i in {1..50}
    do
if [ "$OS" = "FreeBSD" ];then
            sockstat -46lp $PORT | grep -qE "^[^ ]* *(socat|nc) *[^ ]* *[^ ]* *[^ ]* *[^ ]*:$PORT" &&$
        else
            ss -p state listening "( sport = :$PORT )" | grep -qE 'socat|nc' && break
        fi
        sleep 0.2
    done
    echo "ready ${ADDR}/${MODULE}//$sst_ver"
}

Which causes SELinux to light up like a Christmas tree.

PXC 5.7’s wsrep_sst_xtrabackup-v2 calls wait_for_listen() like this:

    # Note: this is started as a background process
    # So it has to wait for processes that are started by THIS process
    wait_for_listen $$ ${WSREP_SST_OPT_HOST} ${WSREP_SST_OPT_PORT:-4444} ${MODULE} &

and executes this function:

# waits ~1 minute for nc/socat to open the port and then reports ready
# (regardless of timeout)
#
# Assumptions:
#   1. The socat/nc processes do not launch subprocesses to handle
#      the connections.  Note that socat can be configured to do so.
#   2. socat is not bound to a specific interface/address, so the
#      IP portion of the local address is all zeros (0000...000).
#
# Parameter 1: the pid of the wsrep_sst_xtrabackup-v2 process. We are looking
#              for the socat process that was started by this script.
# Parameter 2: the IP address of the SST host
# Parameter 3: the Port of the SST host
# Parameter 4: a descriptive name for what we are doing at this point
#
wait_for_listen()
{
    local parentpid=$1
    local host=$2
    local port=$3
    local module=$4

    #
    # Check to see if the OS supports /proc/<pid>/net/tcp
    #
    if [[ ! -r /proc/$$/net/tcp && ! -r /proc/$$/net/tcp6 ]]; then
        wsrep_log_debug "$LINENO: Using ss for socat/nc discovery"

        # Revert to using ss to check if socat/nc is listening
        wsrep_check_program ss
        if [[ $? -ne 0 ]]; then
            wsrep_log_error "******** FATAL ERROR *********************** "
            wsrep_log_error "* Could not find 'ss'.  Check that it is installed and in the path."
            wsrep_log_error "******************************************** "
            return 2
        fi

        for i in {1..300}
        do
            ss -p state listening "( sport = :${port} )" | grep -qE 'socat|nc' && break
            sleep 0.2
        done

        echo "ready ${host}:${port}/${module}//$sst_ver"
        return 0
    fi

    wsrep_log_debug "$LINENO: Using /proc/pid/net/tcp for socat/nc discovery"

    # Get the index for the 'local_address' column in /proc/xxxx/net/tcp
    # We expect this to be the same for IPv4 (net/tcp) and IPv6 (net/tcp6)
    local ip_index=0
    local header
    if [[ -r /proc/$$/net/tcp ]]; then
        read -ra header <<< $(head -n 1 /proc/$$/net/tcp)
    elif [[ -r /proc/$$/net/tcp6 ]]; then
        read -ra header <<< $(head -n 1 /proc/$$/net/tcp6)
    else
        wsrep_log_error "******** FATAL ERROR *********************** "
        wsrep_log_error "* Cannot find /proc/$$/net/tcp (or tcp6)"
        wsrep_log_error "******************************************** "
        exit 1
    fi
    for i in "${!header[@]}"; do
        if [[ ${header[$i]} = "local_address" ]]; then
            # Add one to the index since arrays are 0-based
            # but awk is 1-based
            ip_index=$(( i + 1 ))
            break
        fi
    done
    if [[ $ip_index -eq 0 ]]; then
        wsrep_log_error "******** FATAL ERROR *********************** "
        wsrep_log_error "* Unexpected /proc/xx/net/tcp layout: cannot find local_address"
        wsrep_log_error "******************************************** "
        exit 1
    fi

    wsrep_log_debug "$LINENO: local_address index is $ip_index"
    local port_in_hex
    port_in_hex=$(printf "%04X" $port)

    local user_id
    user_id=$(id -u)

    for i in {1..300}
    do
        # List all socat/nc processes started by the user of this script
        # Then look for processes that have the script pid as a parent prcoess
        # somewhere in the process tree

        wsrep_log_debug "$LINENO: Entering loop body : $i"

        # List only socat/nc processes started by this user to avoid triggering SELinux
        for pid in $(ps -u $user_id -o pid,comm | grep -E 'socat|nc' | awk '{ printf $1 " " }')
        do
            if [[ -z $pid || $pid = " " ]]; then
                continue
            fi

            wsrep_log_debug "$LINENO: Examining pid: $pid"

            # Now get the processtree for this pid
            # If the parentpid is NOT in the process tree, then ignore
            if ! echo "$(get_parent_pids $pid)" | grep -qw "$parentpid"; then
                wsrep_log_debug "$LINENO: $parentpid is not in the process tree: $(get_parent_pids $pid)"
                continue
            fi

            # get the sockets for the pid
            # Note: may not need to get the list of sockets, is it ok to
            # just look at the list of local addresses in tcp?
            local sockets
            sockets=$(ls -l /proc/$pid/fd | grep socket | cut -d'[' -f2 | cut -d ']' -f1 | tr '\n' '|')

            # remove the trailing '|'
            sockets=${sockets%|}
            wsrep_log_debug "$LINENO: sockets: $sockets"

            if [[ -n $sockets ]]; then
                # For the network addresses, we expect to be listening
                # on all interfaces, thus the address should be
                # 00..000:PORT (all zeros for the IP address).

                # Dumping the data in the lines
                #if [[ -n "$WSREP_LOG_DEBUG" ]]; then
                #    lines=$(grep -E "\s(${sockets})\s" /proc/$pid/net/tcp)
                #    if [[ -n $lines ]]; then
                #        while read -r line; do
                #            if [[ -z $line ]]; then
                #                continue
                #            fi
                #            wsrep_log_debug "$LINENO: $line"
                #        done <<< "$lines\n"
                #    fi
                #fi

                # Checking IPv4
                if grep -E "\s(${sockets})\s" /proc/$pid/net/tcp |
                        awk "{print \$${ip_index}}" |
                        grep -q "^00*:${port_in_hex}$"; then
                    wsrep_log_debug "$LINENO: found a match for pid: $pid"
                    break 2
                fi

                # Also check IPv6
                if grep -E "\s(${sockets})\s" /proc/$pid/net/tcp6 |
                        awk "{print \$${ip_index}}" |
                        grep -q "^00*:${port_in_hex}$"; then
                    break 2
                fi
            fi
        done

        sleep 0.2
    done

    wsrep_log_debug "$LINENO: wait_for_listen() exiting"
    echo "ready ${host}:${port}/${module}//$sst_ver"
    return 0
}

which does NOT light up SELinux like a Christmas tree (but DOES still require a few SELinux policy additions to run in enforcing mode).

Obviously the PXC function code isn’t directly compatible with the Mariabackup sst script since at the very least it’s not using the same parameters, but can I just change the calling line AND the function itself in my version of wsrep_sst_mariabackup and will it work, or is it more complicated than that?

There are also other SELinux AVC failures that I’m prepared to write custom policy for, but I’d like to “patch” the wait_for_listen function in wsrep_sst_mariabackup so I don’t have to remember every time I add a listening daemon on the server that I need to rebuild my custom SELinux policy.  If the author of the article is correct, the custom SELinux policy allows needed OUTSIDE the noisy listening port scan should be pretty static and only need a re-check on major version upgrades.

So, I know this was a lot and if you think I should take this to the devs list, please let me know.  I was hoping there might be a powerful enough user on this list who could suggest the code changes needed in wsrep_sst_mariabackup to get me around the most onerous part of this, OR whatever changes I might need to make outside of drag-and-drop.

Thanks,

Scott