Introduction to State Snapshot Transfers (SSTs)

In a State Snapshot Transfer (SST), the cluster provisions nodes by transferring a full data copy from one node to another. When a new node joins the cluster, the new node initiates a State Snapshot Transfer to synchronize its data with a node that is already part of the cluster.

Types of SSTs

There are two conceptually different ways to transfer a state from one MariaDB server to another:

1. Logical: The only SST method of this type is the mysqldump SST method, which uses the utility to get a logical dump of the donor. This SST method requires the joiner node to be fully initialized and ready to accept connections before the transfer. This method is, by definition, blocking, in that it blocks the donor node from modifying its state for the duration of the transfer. It is also the slowest of all, and that might be an issue in a cluster with a lot of loads.

2. Physical: SST methods of this type physically copy the data files from the donor node to the joiner node. This requires that the joiner node be initialized after the transfer. The SST method and a few other SST methods fall into this category. These SST methods are much faster than the mysqldump SST method, but they have certain limitations. For example, they can be used only on server startup, and the joiner node must be configured very similarly to the donor node (e.g., should be the same, and so on). Some of the SST methods in this category are non-blocking on the donor node, meaning that the donor node is still able to process queries while donating the SST (e.g. the SST method is non-blocking).

SST Methods

SST methods are supported via a scriptable interface. New SST methods could potentially be developed by creating new SST scripts. The scripts usually have names of the form wsrep_sst_<method> where <method> is one of the SST methods listed below.

You can choose your SST method by setting the system variable. It can be changed dynamically with on the node that you intend to be an SST donor. For example:

It can also be set in a server in an prior to starting up a node:

For an SST to work properly, the donor and joiner node must use the same SST method. Therefore, it is recommended to set to the same value on all nodes, since any node will usually be a donor or joiner node at some point.

MariaDB Galera Cluster comes with the following built-in SST methods:

mariadb-backup

This SST method uses the utility for performing SSTs. It is one of the two non-locking methods. This is the recommended SST method if you require the ability to run queries on the donor node during the SST. Note that if you use the mariadb-backup SST method, then you also need to have socat installed on the server. This is needed to stream the backup from the donor to the joiner. This is a limitation inherited from the xtrabackup-v2 SST method.

• This SST method supports

• This SST method supports .

• This SST method is available from and .

With this SST method, it is impossible to upgrade the cluster between some major versions; see .

See for more information.

rsync / rsync_wan

rsync is the default method. This method uses the utility to create a snapshot of the donor node. rsync should be available by default on all modern Linux distributions. The donor node is blocked with a read lock during the SST. This is the fastest SST method, especially for large datasets since it copies binary data. Because of that, this is the recommended SST method if you do not need to allow the donor node to execute queries during the SST.

The rsync method runs rsync in --whole-file mode, assuming that nodes are connected by fast local network links so that the default delta transfer mode would consume more processing time than it may save on data transfer bandwidth. When having a distributed cluster with slow links between nodes, the rsync_wan method runs rsync in the default delta transfer mode, which may reduce data transfer time substantially when an older datadir state is already present on the joiner node. Both methods are actually implemented by the same script, wsrep_sst_rsync_wan is just a symlink to the wsrep_sst_rsync script and the actual rsync mode to use is determined by the name the script was called by.

• This SST method supports

• This SST method supports .

The rsync SST method does not support tables created with the clause. Use the as an alternative to support this feature.

mysqldump

This SST method runs on the donor node and pipes the output to the client connected to the joiner node. The mysqldump SST method needs a username/password pair set in the variable in order to get the dump. The donor node is blocked with a read lock during the SST. This is the slowest SST method.

• This SST method supports .

• This SST method supports .

xtrabackup-v2

Percona XtraBackup is not supported in MariaDB. is the recommended backup method to use instead of Percona XtraBackup. See for more information.

This SST method uses the utility for performing SSTs. It is one of the two non-blocking methods. Note that if you use the xtrabackup-v2 SST method, you also need to have socat installed on the server. Since Percona XtraBackup is a third-party product, this SST method requires an additional installation and some additional configuration. Please refer to for information from the vendor.

• This SST method does not support

• This SST method does not support .

This SST method is available from MariaDB Galera Cluster 5.5.37 and MariaDB Galera Cluster 10.0.10.

See xtrabackup-v2 SST method for more information.

xtrabackup

Percona XtraBackup is not supported in MariaDB. is the recommended backup method to use instead of Percona XtraBackup. See for more information.

This SST method is an older SST method that uses the utility for performing SSTs. The xtrabackup-v2 SST method should be used instead of the xtrabackup SST method starting from .

• This SST method does not support

• This SST method does not support .

Authentication

All SST methods except rsync require authentication via username and password. You can tell the client what username and password to use by setting the system variable. It can be changed dynamically with on the node that you intend to be a SST donor. For example:

It can also be set in a server in an prior to starting up a node:

Some do not require a password. For example, the and authentication plugins do not require a password. If you are using a user account that does not require a password in order to log in, then you can just leave the password component of empty. For example:

See the relevant description or page for each SST method to find out what privileges need to be to the user and whether the privileges are needed on the donor node or joiner node for that method.

SSTs and Systemd

MariaDB's unit file has a default startup timeout of about 90 seconds on most systems. If an SST takes longer than this default startup timeout on a joiner node, then systemd will assume that mysqld has failed to startup, which causes systemd to kill the mysqld process on the joiner node. To work around this, you can reconfigure the MariaDB systemd unit to have an infinite timeout, such as by executing one of the following commands:

If you are using systemd 228 or older, then you can execute the following to set an infinite timeout:

, so if you are using systemd 229 or later, then you can execute the following to set an infinite timeout:

See for more details.

Note that that allows services to extend the startup timeout during long-running processes. Starting with , , and , on systems with systemd versions that support it, MariaDB uses this feature to extend the startup timeout during long SSTs. Therefore, if you are using systemd 236 or later, then you should not need to manually override TimeoutStartSec, even if your SSTs run for longer than the configured value. See for more information.

SST Failure

An SST failure generally renders the joiner node unusable. Therefore, when an SST failure is detected, the joiner node will abort.

Restarting a node after a mysqldump SST failure may require manual restoration of the administrative tables.

SSTs and Data at Rest Encryption

Look at the description of each SST method to determine which methods support .

For logical SST methods like mysqldump, each node should be able to have different . For physical SST methods, all nodes need to have the same , since the donor node will copy encrypted data files to the joiner node, and the joiner node will need to be able to decrypt them.

Minimal Cluster Size

In order to avoid a split-brain condition, the minimum recommended number of nodes in a cluster is 3.

When using an SST method that blocks the donor, there is yet another reason to require a minimum of 3 nodes. In a 3-node cluster, if one node is acting as an SST joiner and one other node is acting as an SST donor, then there is still one more node to continue executing queries.

Manual SSTs

In some cases, if Galera Cluster's automatic SSTs repeatedly fail, then it can be helpful to perform a "manual SST". See the following pages on how to do that:

Known Issues

mysqld_multi

SST scripts can't currently read the mysqld<#> in an that are read by instances managed by .

See for more information.

See Also

_{This page is licensed: CC BY-SA / Gnu FDL}