Introduction to State Snapshot Transfers (SSTs)

In a State Snapshot Transfer (SST), the cluster provisions nodes by transferring a full data copy from one node to another. When a new node joins the cluster, the new node initiates a State Snapshot Transfer to synchronize its data with a node that is already part of the cluster.

Types of SSTs

There are two conceptually different ways to transfer a state from one MariaDB server to another:

1. Logical

The only SST method of this type uses mysqldump. This SST method requires the joiner node to be fully initialized and ready to accept connections before the transfer. This method is, by definition, blocking, in that it blocks the donor node from modifying its own state for the duration of the transfer. It is also the slowest of all, and that might be an issue in a loaded cluster.

2. Physical

SST methods of this type physically copy the data files from the donor node to the joiner node. This requires that the joiner node is initialized after the transfer. mariabackup, and other a few other SST methods fall into this category. These SST methods are much faster than mysqldump, but they have certain limitations. For example, they can be used only on server startup and the joiner node must be configured very similarly to the donor node (e.g. innodb_file_per_table should be the same and so on). Some of the SST methods in this category are non-blocking on the donor node, meaning that the donor node is still able to process queries while donating the SST (e.g. mariabackup allow this).

SST Methods

SST methods are supported via a scriptable interface. New SST methods could potentially be developed by creating new SST scripts.

MariaDB Galera Cluster comes with the following SST methods:

mariabackup

This SST method uses the MariaDB Backup utility for performing SSTs. One of the two non-locking methods. Note that if you use the mariabackup SST method, you also need to have socat installed on the server. This is needed to stream the backup from the donor to the joiner. This is a limitation inherited from the xtrabackup-v2 SST method.

This SST method is available from MariaDB 10.1.26 and MariaDB 10.2.10.

mysqldump

This SST method runs mysqldump on the donor node and pipes the output to the mysql client connected to the joiner node. mysqldump needs a username/password pair set in the wsrep_sst_auth variable in order to get the dump. The donor node is blocked with a read lock during the SST.

rsync

This is the default method. This method uses the rsync utility to create a snapshot of the donor node. rsync should be available by default on all modern linux distributions. The donor node is blocked with a read lock during the SST. This is the fastest SST method, especially for large datasets since it copies binary data. It is the recommended SST method, if you do not need to allow the donor node to execute queries during the SST.

As of MariaDB 10.1.36, MariaDB 10.2.18, and MariaDB 10.3.10, stunnel can be used to encrypt data over the wire. Be sure to have it installed, and to set the tkey and tcert options, referring to the Galera Cluster documentation. The certificate directory will have to be hashed: openssl rehash /path/to/certs/.

xtrabackup-v2

This SST method uses the Percona XtraBackup utility for performing SSTs. One of the two non-locking methods. Note that if you use the mariabackup SST method, you also need to have socat installed on the server. However, since it is a third party product, it requires an additional installation some additional configuration. Please refer to the xtrabackup SST documentation for more information.

xtrabackup

This SST method is an older SST method that uses the Percona XtraBackup utility for performing SSTs. The xtrabackup-v2 SST method should be used instead of the xtrabackup SST method starting from MariaDB 5.5.33.

SSTs and Systemd

MariaDB's systemd unit file has a fairly low timeout value by default. If your SST takes longer than about 2 minutes, this low default timeout can cause systemd to assume that mysqld startup has failed, which causes it to kill the mysqld process. To work around this, you can reconfigure the MariaDB systemd unit to have an infinite timeout, such as by executing one of the following commands:

If you are using systemd 228 or older, then you can execute the following to set an infinite timeout:

sudo tee /etc/systemd/system/mariadb.service.d/timeoutstartsec.conf <<EOF
[Service]

TimeoutStartSec=0
EOF
sudo systemctl daemon-reload

Systemd 229 added the infinity option, so if you are using systemd 229 or later, then you can execute the following to set an infinite timeout:

sudo tee /etc/systemd/system/mariadb.service.d/timeoutstartsec.conf <<EOF
[Service]

TimeoutStartSec=infinity
EOF
sudo systemctl daemon-reload

See customizing systemd for more details.

Note that systemd 236 added the EXTEND_TIMEOUT_USEC environment variable that allows services to extend the startup timeout during long-running processes. MariaDB uses that functionality during long SSTs to extend the timeout on systems with systemd versions that support it. Therefore, if you are using systemd 236 or later, then you should not need to manually override TimeoutStartSec, even if your SSTs run for longer than the configured value. See MDEV-15607 for more information.

SST Failure

An SST failure generally renders the joiner node unusable. Therefore, when an SST failure is detected, the joiner node will abort.

Restarting a node after a mysqldump SST failure may require manual restoration of the administrative tables.

Minimal Cluster Size

In order to avoid a split-brain condition, the minimum recommended number of nodes in a cluster is 3.

When using an SST method that blocks the donor, there is yet another reason to require a minimum of 3 nodes. In a 3-node cluster, if one node is acting as an SST joiner and one other node is acting as an SST donor, then there is still one more node to continue executing queries.

Manual SSTs

In some cases, if Galera Cluster's automatic SSTs repeatedly fail, then it can be helpful to perform a "manual SST". See the following pages on how to do that:

• Manual SST of Galera Cluster node with Mariabackup
• Manual SST of Galera Cluster node with Percona XtraBackup