maria-developers team mailing list archive

[Nirbhay Choubey <nirbhay@xxxxxxxxxxx>]

Hi Svoj!

On Wed, May 25, 2016 at 3:56 AM, Sergey Vojtovich <svoj@xxxxxxxxxxx> wrote:

> Hi Nirbhay,
>
> I believe this patch is acceptable. Still I didn't completely understand
> the
> need for all this complexity and your answer to Daniel's question.
>

wsrep recovery broadly involves following steps:

1.0 # Grab the last saved global transaction ID from the storage engine
(InnoDB) after performing SE recovery.
2,0 # Pass this ID to wsrep provider (galera).
3.0 # wsrep provider decides whether the node requires a state transfer.
3.1 # Perform state transfer (in case of rsync/xtrabackup SST methods,
receive files - this has
to be done before SE initialization).
3.2 # In case of rsync/xtrabackup : load the storage engine (InnoDB).

While 1.0 is done after loading InnoDB, 3.1 needs to be does before its
initialization. And since InnoDB cannot be reinitialized without a
restarting
the server, the entire recovery process requires a server restart.

1.0 : mysqld --wsrep-recover
2.0 - 3.2 : mysqld [all-other-options] --wsrep-start-position=xxx


> Some minor comments inline.
>
> On Fri, May 20, 2016 at 11:04:24PM -0400, Nirbhay Choubey wrote:
> > revision-id: 454a9dc53a5b02af316dd713c577082eb8cf79c4
> (mariadb-10.1.14-4-g454a9dc)
> > parent(s): 9a1c4e900b98fdb9940aab57c895753f175c2bd8
> > author: Nirbhay Choubey
> > committer: Nirbhay Choubey
> > timestamp: 2016-05-20 23:04:20 -0400
> > message:
> >
> > MDEV-10004: Galera's pc.recovery process fails in 10.1 with systemd
> >
> > Galera recovery process works in two phases. In the first
> > phase, mysqld is started as non-daemon with --wsrep-recover
> > to recover and fetch the last logged global transaction ID.
> > This ID is then used in second phase as the start position
> > (--wsrep-start-position=XX) to start mysqld as daemon.
> >
> > As this process was implemented in mysqld_safe script, the
> > recovery did not work when server was started using systemd.
> >
> > Fixed by introducing a shell script (wsrep_recovery.sh) that
> > mimics the first phase of the recovery process.
> >
> > ---
> >  cmake/systemd.cmake              |  3 +-
> >  scripts/galera_recovery.sh       | 94
> ++++++++++++++++++++++++++++++++++++++++
> >  support-files/mariadb.service.in | 11 ++++-
> >  3 files changed, 106 insertions(+), 2 deletions(-)
> >
> > diff --git a/cmake/systemd.cmake b/cmake/systemd.cmake
> > index b0161cf..17f44f7 100644
> > --- a/cmake/systemd.cmake
> > +++ b/cmake/systemd.cmake
> > @@ -55,9 +55,10 @@ MACRO(CHECK_SYSTEMD)
> >        IF(HAVE_SYSTEMD AND HAVE_SYSTEMD_SD_DAEMON_H AND
> HAVE_SYSTEMD_SD_LISTEN_FDS
> >           AND HAVE_SYSTEMD_SD_NOTIFY AND HAVE_SYSTEMD_SD_NOTIFYF)
> >          ADD_DEFINITIONS(-DHAVE_SYSTEMD)
> > -        SET(SYSTEMD_SCRIPTS mariadb-service-convert galera_new_cluster)
> > +        SET(SYSTEMD_SCRIPTS mariadb-service-convert galera_new_cluster
> galera_recovery)
> >          SET(SYSTEMD_DEB_FILES "usr/bin/mariadb-service-convert
> >                                 usr/bin/galera_new_cluster
> > +                               usr/bin/galera_recovery
> >
>  ${INSTALL_SYSTEMD_UNITDIR}/mariadb.service
> >                                 ${INSTALL_SYSTEMD_UNITDIR}/mariadb@
> .service
> >
>  ${INSTALL_SYSTEMD_UNITDIR}/mariadb@bootstrap.service.d
> /use_galera_new_cluster.conf")
> > diff --git a/scripts/galera_recovery.sh b/scripts/galera_recovery.sh
> > new file mode 100755
> > index 0000000..6d4f1d5
> > --- /dev/null
> > +++ b/scripts/galera_recovery.sh
> > @@ -0,0 +1,94 @@
> > +#!/bin/sh
> > +
> > +# Copyright (c) 2016 MariaDB Corporation
> > +#
> > +# This program is free software; you can redistribute it and/or modify
> > +# it under the terms of the GNU General Public License as published by
> > +# the Free Software Foundation; version 2 of the License.
> > +#
> > +# This program is distributed in the hope that it will be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write to the Free Software
> > +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301
> USA */
> > +
> > +
> > +# This script is intended to be executed by systemd. It starts mysqld
> with
> > +# --wsrep-recover to recover from a non-graceful shutdown, determines
> the
> > +# last stored global transaction ID and echoes it in
> --wsrep-start-position=XX
> > +# format. The output is then captured and used by systemd to start
> mysqld.
> > +# If the server was configured to start without wsrep, nothing is
> echoed.
> > +
> > +user='@MYSQLD_USER@'
> > +log_file=$(mktemp /tmp/wsrep_recovery.XXXXXX)
> > +euid=$(id -u)
> > +recovered_pos=""
> > +skipped=""
> > +start_pos=""
> > +start_pos_opt=""
> > +ret=0
> > +
> > +log ()
> > +{
> > +  local msg="$1"
> > +  # Print all messages to stderr as we reserve stdout for printing
> > +  # --wsrep-start-position=XXXX.
> > +  echo "$msg" >&2
> > +}
> > +
> > +finish()
> > +{
> > +  rm -f "$log_file"
> > +}
> > +
> > +trap finish EXIT
> > +
> > +# Safety checks
> > +if [ -z "$log_file" ]; then
> > +  log "WSREP: mktemp failed"
> > +  exit 1
> > +fi
> > +
> > +if [ -f "$log_file" ]; then
> > +  [ "$euid" = "0" ] && chown $user $log_file
> > +    chmod 600 $log_file
> > +else
> > +  log "WSREP: Failed to change the permissions of $log_file."
> I'd say it's rather mktemp failed than change permissions failed.
>

Right. I have merged the above 2 check.


>
> > +  exit 1
> > +fi
> > +
> > +# Redirect server's error log to the log file.
> > +eval /usr/sbin/mysqld --user=$user --wsrep_recover 2> "$log_file"
> > +ret=$?
> > +if [ $ret -ne 0 ]; then
> > +  # Something went wrong, let us also print the error log so that it
> > +  # shows up in systemctl status output as a hint to the user.
> > +  log "WSREP: Failed to start mysqld for wsrep recovery: '`cat
> $log_file`'"
> > +  exit 1
> > +fi
> Since this script is executed unconditionally, what shall happen if wsrep
> is
> not enabled at compile time (--wsrep_recover is not known)? Fail the whole
> service startup?
>

I have modified this script to use my_print_defaults to first parse the
configs
and perform this step iff --wsrep-on is set (wsrep-on being one of the
mandatory
setting to enable wsrep). This logic has been taken from mysqld_safe script.

Now, if mysqld is built without wsrep, having wsrep-on in configs will fail
anyway.


> > +
> > +# Parse server's error log for recovered position. The server prints
> > +# "..skipping position recovery.." if started without wsrep.
> > +
> > +recovered_pos="$(grep 'WSREP: Recovered position:' $log_file)"
> > +
> > +if [ -z "$recovered_pos" ]; then
> > +  skipped="$(grep WSREP $log_file | grep 'skipping position recovery')"
> > +  if [ -z "$skipped" ]; then
> > +    log "WSREP: Failed to recover position: '`cat $log_file`'"
> > +    exit 1
> Can we come here with wsrep disabled?
>

No.


>
> > +  else
> > +    log "WSREP: Position recovery skipped."
> > +  fi
> > +else
> > +  start_pos="$(echo $recovered_pos | sed 's/.*WSREP\:\ Recovered\
> position://' \
> > +                  | sed 's/^[ \t]*//')"
> > +  log "WSREP: Recovered position $start_pos"
> > +  start_pos_opt="--wsrep_start_position=$start_pos"
> > +fi
> > +
> > +echo "$start_pos_opt"
> > +
> > diff --git a/support-files/mariadb.service.in b/support-files/
> mariadb.service.in
> > index b18674b..00162d4 100644
> > --- a/support-files/mariadb.service.in
> > +++ b/support-files/mariadb.service.in
> > @@ -48,6 +48,12 @@ CapabilityBoundingSet=CAP_IPC_LOCK
> >  # Execute pre and post scripts as root, otherwise it does it as User=
> >  PermissionsStartOnly=true
> >
> > +# Perform automatic wsrep recovery. When server is started without
> wsrep,
> > +# galera_recovery simply returns an empty string. In any case, however,
> > +# the script is not expected to return with a non-zero status.
> > +ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]
> && \
> > + systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1"
> I believe _WSREP_START_POSITION won't work with multi instance version. Not
> sure if it works but we may need to use something like
> _WSREP_START_POSITION%I
> here.
>

I have updated and included this command in the multi-instance file
(mariadb@xxxxxxxxxxx)
with instance name placeholder (%I).


> > +
> >  # Needed to create system tables etc.
> >  # ExecStartPre=/usr/bin/mysql_install_db -u mysql
> >
> > @@ -57,9 +63,12 @@ PermissionsStartOnly=true
> >  # This isn't a replacement for my.cnf.
> >  # _WSREP_NEW_CLUSTER is for the exclusive use of the script
> galera_new_cluster
> >  @SYSTEMD_EXECSTARTPRE@
> > -ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER
> > +ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER
> $_WSREP_START_POSITION
> >  @SYSTEMD_EXECSTARTPOST@
> >
> > +# Unset _WSREP_START_POSITION environment variable.
> > +ExecStartPost=/bin/sh -c "systemctl unset-environment
> _WSREP_START_POSITION"
> > +
> >  KillMode=process
> >  KillSignal=SIGTERM
> >
> > _______________________________________________
> > commits mailing list
> > commits@xxxxxxxxxxx
> > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits
>
>
Thanks!

-- Nirbhay


>
> Regards,
> Sergey
> _______________________________________________
> commits mailing list
> commits@xxxxxxxxxxx
> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits