Manual SST of Galera Cluster Node With Percona XtraBackup - Source

Revision: 95421
User: Geoff Montee
Date: 2020-05-04 12:37

<<toc>>

<<style class="redbox">>
Mariabackup should be used instead of XtraBackup on all supported releases. See [[manual-sst-of-galera-cluster-node-with-mariabackup|manual SST with Mariabackup]].
<</style>>

<<include slug="xtrabackup_warning">>

Sometimes it can be helpful to perform a "manual SST" when Galera's [[introduction-to-state-snapshot-transfers-ssts|normal SSTs]] fail. This can be especially useful when the cluster's ##[[server-system-variables/#datadir|datadir]]## is very large, since a normal SST can take a long time to fail in that case.

A manual SST essentially consists of taking a backup of the donor, loading the backup on the joiner, and then manually editing the cluster state on the joiner node. This page will show how to perform this process with [[backup-restore-and-import-clients-percona-xtrabackup|Percona XtraBackup]].

== Process

* Check that the donor and joiner nodes have the same XtraBackup version.

<<code>>
innobackupex --version
<</code>>

* Create backup directory on donor.

<<code>>
MYSQL_BACKUP_DIR=/mysql_backup
mkdir $MYSQL_BACKUP_DIR
<</code>>

* Take a [[https://www.percona.com/doc/percona-xtrabackup/2.4/howtos/recipes_ibkx_local.html|full backup]] the of the donor node with ##innobackupex##. The ##[[https://www.percona.com/doc/percona-xtrabackup/2.4/innobackupex/innobackupex_option_reference.html#cmdoption-innobackupex-galera-info|--galera-info]]## option should also be provided, so that the node's cluster state is also backed up.

<<code>>
DB_USER=sstuser
DB_USER_PASS=password
innobackupex --user=$DB_USER --password=$DB_USER_PASS --galera-info --no-timestamp $MYSQL_BACKUP_DIR
<</code>>

* Verify that the MariaDB Server process is stopped on the joiner node. This will depend on your [[starting-and-stopping-mariadb-starting-and-stopping-mariadb|service manager]].

For example, on [[systemd|systemd]] systems, you can execute::

<<code>>
systemctl status mariadb
<</code>>

* Create the backup directory on the joiner node.

<<code>>
MYSQL_BACKUP_DIR=/mysql_backup
mkdir $MYSQL_BACKUP_DIR
<</code>>

* Copy the backup from the donor node to the joiner node.

<<code>>
OS_USER=dba
JOINER_HOST=dbserver2.mariadb.com
rsync -av $MYSQL_BACKUP_DIR/* ${OS_USER}@${JOINER_HOST}:${MYSQL_BACKUP_DIR}
<</code>>

* [[https://www.percona.com/doc/percona-xtrabackup/2.4/howtos/recipes_ibkx_local.html#prepare-the-backup|Prepare the backup]] on the joiner node.

<<code>>
innobackupex --apply-log $MYSQL_BACKUP_DIR
<</code>>

* Get the Galera Cluster version ID from the donor node's ##grastate.dat## file.

<<code>>
MYSQL_DATADIR=/var/lib/mysql
cat $MYSQL_DATADIR/grastate.dat | grep version
<</code>>

For example, a very common version number is "2.1".

* Get the node's cluster state from the ##[[https://www.percona.com/doc/percona-xtrabackup/2.4/innobackupex/innobackupex_option_reference.html#cmdoption-innobackupex-galera-info|--xtrabackup_galera_info]]## file in the backup that was copied to the joiner node.

<<code>>
cat $MYSQL_BACKUP_DIR/xtrabackup_galera_info
<</code>>

Example output:

<<code>>
d38587ce-246c-11e5-bcce-6bbd0831cc0f:1352215
<</code>>

This output is in the format:

<<code>>
uuid:seqno
<</code>>

* Create a ##grastate.dat## file in the backup directory of the joiner node. The Galera Cluster version ID, the cluster uuid, and the seqno from previous steps will be used to fill in the relevant fields.

For example, with the example values from the last two steps, we could do:

<<code>>
sudo tee $MYSQL_BACKUP_DIR/grastate.dat <<EOF
# GALERA saved state
version: 2.1
uuid:    d38587ce-246c-11e5-bcce-6bbd0831cc0f
seqno:   1352215
safe_to_bootstrap: 0
EOF
<</code>>

* Remove the existing contents of the ##[[server-system-variables/#datadir|datadir]]## on the joiner node.

<<code>>
MYSQL_DATADIR=/var/lib/mysql
rm -Rf $MYSQL_DATADIR/*
<</code>>

* Copy the contents of the backup directory to the ##[[server-system-variables/#datadir|datadir]]## the on joiner node.

<<code>>
cp -R $MYSQL_BACKUP_DIR/* $MYSQL_DATADIR/
<</code>>

* Make sure the permissions of the ##[[server-system-variables/#datadir|datadir]]## are correct on the joiner node.

<<code>>
chown -R mysql:mysql $MYSQL_DATADIR/
<</code>>

* Start the MariaDB Server process on the joiner node. This will depend on your [[starting-and-stopping-mariadb-starting-and-stopping-mariadb|service manager]].

For example, on [[systemd|systemd]] systems, you can execute::

<<code>>
systemctl start mariadb
<</code>>

* Watch the MariaDB [[error-log|error log]] on the joiner node and verify that the node does not need to perform a [[introduction-to-state-snapshot-transfers-ssts|normal SSTs]] due to the manual SST.

<<code>>
tail -f /var/log/mysql/mysqld.log
<</code>>

Content reproduced on this site is the property of its respective owners, and this content is not reviewed in advance by MariaDB. The views, information and opinions expressed by this content do not necessarily represent those of MariaDB or any other party.

Manual SST of Galera Cluster Node With Percona XtraBackup - Source

Products

Services

Pricing

Resources

About Us

Download MariaDB