Sometimes it can be helpful to perform a "manual SST" when Galera's normal SSTs fail. This can be especially useful when the cluster's is very large, since a normal SST can take a long time to fail in that case.

A manual SST essentially consists of taking a backup of the donor, loading the backup on the joiner, and then manually editing the cluster state on the joiner node. This page will show how to perform this process with .

Process

For example, a very common version number is "2.1".

In a State Snapshot Transfer (SST), the cluster provisions nodes by transferring a full data copy from one node to another. When a new node joins the cluster, the new node initiates a State Snapshot Transfer to synchronize its data with a node that is already part of the cluster.

Types of SSTs

There are two conceptually different ways to transfer a state from one MariaDB server to another:

1. Logical: The only SST method of this type is the mysqldump

The mariabackup SST method uses the utility for performing SSTs. It is one of the methods that does not block the donor node. mariadb-backup was originally forked from , and similarly, the mariabackup SST method was originally forked from the xtrabackup-v2 SST method.

Choosing mariadb-backup for SSTs

To use the mariadb-backup SST method, you must set the on both the donor and joiner node. It can be changed dynamically with on the node that you intend to be an SST donor. For example:

It can be set in a server in an prior to starting up a node:

For an SST to work properly, the donor and joiner node must use the same SST method. Therefore, it is recommended to set to the same value on all nodes, since any node will usually be a donor or joiner node at some point.

Major Version Upgrades

The default method wsrep_sst_method=rsync works for major-version upgrades; see .

Configuration Options

The mariabackup SST method is configured by placing options in the [sst] section of a MariaDB configuration file (e.g., /etc/my.cnf.d/server.cnf). These settings are parsed by the wsrep_sst_mariabackup and wsrep_sst_common scripts.

Primary Transfer and Format Options

These options control the core data transfer mechanism.

Compression Options

These options configure on-the-fly compression to reduce network bandwidth.

Authentication and Security (TLS)

These options manage user authentication and stream encryption.

Logging and Miscellaneous Options

Pass-through mariadb-backup Options

This feature allows mariadb-backup specific options to be passed through the SST script.

Example: Using Native Encryption and Threading

Authentication and Privileges

To use the mariadb-backup SST method, the utility must be able to authenticate locally on the donor node to create a backup stream. There are two ways to manage this authentication:

Automatic User Account Management (ES 11.4+)

Starting with MariaDB Enterprise Server 11.4, the cluster can automatically manage the SST user account. This method is more secure and requires less configuration because it avoids storing plain-text passwords in configuration files.

When this feature is used:

1. The donor node automatically creates a temporary internal user (e.g., 'wsrep.sst. <timestamp>_<node_id>'@localhost) with a generated password when the SST process begins.

2. The necessary privileges (RELOAD, PROCESS, LOCK TABLES, etc.) are automatically granted to this temporary user.

3. Once the SST process completes, the donor node automatically drops the user.

To enable automatic user management:

Ensure that the system variable is not set (or is left blank) in your configuration file.

Manual User Configuration

For versions prior to 11.4, or if you prefer to manage the user manually, you must create a user and provide the credentials to the server.

You can tell the donor node what username and password to use by setting the system variable. It can be changed dynamically with SET GLOBAL on the node that you intend to be an SST donor:

It can also be set in a server in an prior to starting up a node:

Some do not require a password. For example, the unix_socket and gssapi authentication plugins do not require a password. If you are using a user account that does not require a password in order to log in, then you can just leave the password component of empty. For example:

The user account that performs the backup for the SST needs to have the same privileges as , which are the RELOAD, PROCESS, LOCK TABLES and BINLOG MONITOR, REPLICA MONITOR . To be safe, ensure that these privileges are set on each node in your cluster. mariadb-backup connects locally on the donor node to perform the backup, so the following user should be sufficient:

Passwordless Authentication - Unix Socket

It is possible to use the authentication plugin for the user account that performs SSTs. This would provide the benefit of not needing to configure a plain-text password in .

The user account would have to have the same name as the operating system user account that is running the mysqld process. On many systems, this is the user account configured as the user option, and it tends to default to mysql.

For example, if the authentication plugin is already installed, then you could execute the following to create the user account:

To configure , set the following in a server in an prior to starting up a node:

Passwordless Authentication - GSSAPI

It is possible to use the authentication plugin for the user account that performs SSTs. This would provide the benefit of not needing to configure a plain-text password in .

The following steps would need to be done beforehand:

• You need a KDC running or .

• You will need to for the MariaDB server.

• You will need to containing the authentication plugin.

• You will need to in MariaDB, so that the authentication plugin is available to use.

For example, you could execute the following to create the user account in MariaDB:

To configure , set the following in a server in an prior to starting up a node:

Choosing a Donor Node

When mariadb-backup is used to create the backup for the SST on the donor node, mariadb-backup briefly requires a system-wide lock at the end of the backup. This is done with .

If a specific node in your cluster is acting as the primary node by receiving all of the application's write traffic, then this node should not usually be used as the donor node, because the system-wide lock could interfere with the application. In this case, you can define one or more preferred donor nodes by setting the system variable.

For example, let's say that we have a 5-node cluster with the nodes node1, node2, node3, node4, and node5, and let's say that node1 is acting as the primary node. The preferred donor nodes for node2 could be configured by setting the following in a server in an prior to starting up a node:

The trailing comma tells the server to allow any other node as donor when the preferred donors are not available. Therefore, if node1 is the only node left in the cluster, the trailing comma allows it to be used as the donor node.

Socat Dependency

During the SST process, the donor node uses socat to stream the backup to the joiner node. Then the joiner node prepares the backup before restoring it. The socat utility must be installed on both the donor node and the joiner node in order for this to work. Otherwise, the MariaDB error log will contain an error like:

Installing Socat on RHEL/CentOS

On RHEL/CentOS, socat can be installed from the repository.

TLS

This SST method supports three different TLS methods. The specific method can be selected by setting the encrypt option in the [sst] section of the MariaDB configuration file. The options are:

• TLS using OpenSSL encryption built into socat (encrypt=2)

• TLS using OpenSSL encryption with Galera-compatible certificates and keys (encrypt=3)

• TLS using OpenSSL encryption with standard MySQL/MariaDB SSL certificates (encrypt=4)

TLS Using OpenSSL Encryption Built into Socat

To generate keys compatible with this encryption method, follow .

First, generate the keys and certificates:

On some systems, you may also have to add dhparams to the certificate:

Next, copy the certificate and keys to all nodes in the cluster.

When done, configure the following on all nodes in the cluster:

TLS Using OpenSSL Encryption With Galera-Compatible Certificates and Keys

To generate keys compatible with this encryption method, follow .

First, generate the keys and certificates:

Next, copy the certificate and keys to all nodes in the cluster.

When done, configure the following on all nodes in the cluster:

Logs

The mariadb-backup SST method has its own logging outside of the MariaDB Server logging.

Logging to SST Logs

Logging for mariadb-backup SSTs works the following way.

By default, on the donor node, it logs to mariadb-backup.backup.log. This log file is located in the .

By default, on the joiner node, it logs to mariadb-backup.prepare.log and mariadb-backup.move.log These log files are also located in the datadir.

By default, before a new SST is started, existing mariadb-backup SST log files are compressed and moved to /tmp/sst_log_archive. This behavior can be disabled by setting sst-log-archive=0 in the [sst] in an . Similarly, the archive directory can be changed by setting sst-log-archive-dir:

See for more information.

Logging to Syslog

Redirect the SST logs to the syslog instead, by setting the following in the [sst] in an :

You can also redirect the SST logs to the syslog by setting the following in the [mysqld_safe] in an :

Performing SSTs With IPv6 Addresses

If you are performing SSTs with IPv6 addresses, then the socat utility needs to be passed the pf=ip6 option. This can be done by setting the sockopt option in the [sst] in an :

See for more information.

Manual SST With mariadb-backup

If Galera Cluster's automatic SSTs repeatedly fail, it can be helpful to perform a "manual SST"; see:

See Also