While a client application can connect directly to any node in a MariaDB Galera Cluster, this is not a practical approach for a production environment. A direct connection creates a single point of failure and does not allow the application to take advantage of the cluster's high availability and read-scaling capabilities.

A load balancer or database proxy is an essential component that sits between your application and the cluster. Its primary responsibilities are:

• Provide a Single Endpoint: Your application connects to the load balancer's virtual IP address, not to the individual database nodes.

• Health Checks: The load balancer constantly monitors the health of each cluster node (e.g., is it Synced? is it up or down?).

• Traffic Routing: It intelligently distributes incoming client connections and queries among the healthy nodes in the cluster.

• Automatic Failover: If a node fails, the load balancer automatically stops sending traffic to it, providing seamless failover for your application.

Recommended Load Balancer: MariaDB MaxScale

For MariaDB Galera Cluster, the recommended load balancer is MariaDB MaxScale. Unlike a generic TCP proxy, MaxScale is a database-aware proxy that understands the Galera Cluster protocol. This allows it to make intelligent routing decisions based on the real-time state of the cluster nodes.

Common Routing Strategies

A database-aware proxy like MaxScale can be configured to use several different routing strategies.

Read-Write Splitting (Recommended)

This is the most common and highly recommended strategy for general-purpose workloads.

• How it Works: The load balancer is configured to send all write operations (INSERT, UPDATE, DELETE) to a single, designated primary node. All read operations (SELECT) are then distributed across the remaining available nodes.

• Advantages:

Read Connection Load Balancing

In this simpler strategy, the load balancer distributes all connections evenly across all available nodes.

• How it Works: Each new connection is sent to the next available node in a round-robin fashion.

• Disadvantages: This approach can easily lead to transaction conflicts if your application sends writes to multiple nodes simultaneously. It is generally only suitable for applications that are almost exclusively read-only.

Other Load Balancing Solutions

While MariaDB MaxScale is the recommended solution, other proxies and load balancers can also be used with Galera Cluster, including:

• ProxySQL: Another popular open-source, database-aware proxy.

• HAProxy: A very common and reliable TCP load balancer. When used with Galera, HAProxy is typically configured with a simple TCP health check or a custom script to determine node availability.

• Cloud Load Balancers: Cloud providers like AWS (ELB/NLB), Google Cloud, and Azure offer native load balancing services that can be used to distribute traffic across a Galera Cluster.

From a , you can check the status of throughout the cluster using standard queries. Status variables that relate to write-set replication have the prefix wsrep_, meaning that you can display them all using the following query:

Understanding Quorum and Cluster Integrity

The most fundamental aspect of a healthy cluster is . Quorum is a mechanism that ensures data consistency by requiring a majority of nodes to be online and in communication to form a Primary Component. Only the Primary Component will process transactions. This prevents

In a MariaDB Galera Cluster, an individual node is considered to have "failed" when it loses communication with the cluster's Primary Component. This can happen for many reasons, including hardware failure, a software crash, loss of network connectivity, or a critical error during a state transfer.

From the perspective of the cluster, a node has failed when the other members can no longer see it. From the perspective of the failed node itself (assuming it hasn't crashed), it has simply lost its connection to the Primary Component and will enter a non-operational state to protect data integrity.

The EVS Protocol

Node failure detection is handled automatically by Galera's group communication system, which uses an Extended Virtual Synchrony (EVS) protocol. This process is controlled by several evs.* in your configuration file.

The cluster determines a node's health based on the last time it received a network packet from that node. The process is as follows:

1. The cluster periodically checks for inactive nodes, controlled by .

2. If a node hasn't sent a packet within the , other nodes begin sending heartbeat beacons to it.

3. If the node remains silent for the duration of , the other nodes will mark it as "suspect."

The Availability vs. Partition Tolerance Trade-off

Within the context of the CAP Theorem (Consistency, Availability, Partition Tolerance), Galera Cluster strongly prioritizes Consistency. This leads to a direct trade-off when configuring the failure detection timeouts, especially on unstable networks like a .

Low Timeouts

Setting low values for evs.suspect_timeout allows the cluster to detect a genuinely failed node very quickly, minimizing downtime. However, on an unstable network, this can lead to "false positives," where a temporarily slow node is incorrectly evicted.

High Timeouts

Setting higher values makes the cluster more tolerant of and slow nodes. However, if a node truly fails, the cluster will remain unavailable for a longer period while it waits for the timeout to expire.

Recovering a Single Failed Node

Recovery from a single node failure is typically automatic. If one node in a cluster with three or more members fails, the rest of the cluster and continues to operate. When the failed node comes back online, it will automatically connect to the cluster and initiate a to synchronize its data. No data is lost in a single node failure.

Recovering the Primary Component After a Full Cluster Outage

A full cluster outage occurs when all nodes shut down or when , leaving no Primary Component. In this scenario, you must manually intervene to safely restart the cluster.

Manual Bootstrap (Using grastate.dat)

This is the traditional recovery method. You must manually identify the node with the most recent data and force it to become the first node in a new cluster.

1. Stop all nodes in the cluster.

2. Identify the most advanced node by checking the seqno value in the in each node's data directory. The node with the highest seqno is the correct one to start from.

3. the new Primary Component by starting the MariaDB service on that single advanced node using a special command (e.g., galera_new_cluster

Automatic Recovery with pc.recovery

Modern versions of Galera Cluster enable the pc.recovery parameter by default. This feature attempts to automate the recovery of the Primary Component.

When pc.recovery is enabled, nodes that were part of the last known Primary Component will save the state of that component to a file on disk called gvwstate.dat. If the entire cluster goes down, it can automatically recover its state once all the nodes from that last saved component achieve connectivity with each other.

Understanding the gvwstate.dat file

The gvwstate.dat file is created in the data directory of a node when it is part of a Primary Component and is deleted upon graceful shutdown. It contains the node's own UUID and its view of the other members of the component. An example:

• my_uuid: The UUID of the node that owns this file.

• view_id: An identifier for the specific cluster view.

• member: The UUIDs of all nodes that were part of this saved Primary Component.

Advanced Procedure: Modifying the Saved State

In the rare case that you need to force a specific set of nodes to form a new Primary Component, you can manually edit the gawtate.dat file on each of those nodes. By ensuring that each node's file lists itself and all other desired members in the member fields, you can force them to recognize each other and form a new component when you start them.

Failures During State Transfers

A node failure can also occur if a is interrupted. This will cause the receiving node (the "joiner") to abort its startup process. To recover, simply restart the MariaDB service on the failed joiner node.

_{This page is licensed: CC BY-SA / Gnu FDL}

Quorum is essential for maintaining data consistency in a MariaDB Galera Cluster by safeguarding against network partitions or node failures. It ensures that the cluster processes database queries and transactions only when a majority of nodes are operational, healthy, and in communication.

Primary Component

This majority group is known as the Primary Component. Nodes not in this group switch to a non-primary state, halting queries and entering a read-only "safe mode" to prevent data discrepancies. The primary function of Quorum is to avoid "split-brain" scenarios, which occur when network partitions lead to parts of the cluster operating independently and accepting writes. By ensuring only the partition with a majority of nodes becomes the Primary Component, Quorum effectively prevents these inconsistencies.

Quorum Calculation

Quorum is achieved when more than 50% of the total nodes in the last known membership are in communication.

• Odd Number of Nodes (Recommended): In a 3-node cluster, a majority is 2. The cluster can tolerate the failure of 1 node and remain operational.

• Even Number of Nodes: In a 2-node cluster, a majority is also 2. If one node fails, the remaining node represents only 50% of the cluster, which is not a majority, and it will lose Quorum. This is why a 2-node cluster has no fault tolerance without an external .

The Galera Arbitrator (garbd)

The Galera Arbitrator (garbd) is the standard solution for clusters with an even number of nodes. It is a lightweight, stateless daemon that acts as a voting member in the cluster without being a full database node. It participates in Quorum calculations, effectively turning an even-numbered cluster into an odd-numbered one. In the diagram, in a 2-node cluster, adding garbd makes the total number of voting members 3, allowing the cluster to maintain Quorum if one database node fails.

Understanding and Recovering from a Split-Brain

A split-brain occurs when a network partition divides the cluster and no resulting group of nodes has a majority (e.g., a 4-node cluster splitting into two groups of 2). By design, both halves of the cluster will fail to achieve a majority, and all nodes will enter a non-Primary state.

If you need to restore service before the network issue is fixed, you must manually intervene:

1. Choose ONE side of the partition to become the new Primary Component.

2. On a single node within that chosen group, execute the following command:

The nodes in this group will now form a new Primary Component. When network connectivity is restored, the nodes from the other partition will automatically rejoin.

Advanced Quorum Control

As a more advanced alternative to garbd for fine-grained control, nodes can also be assigned a specific voting weight.

Monitoring Quorum and Cluster Membership

You can check the health of the cluster and its Quorum status at any time by querying the following .

Recovering from a Full Cluster Shutdown

If the entire cluster loses Quorum (e.g., from a simultaneous crash or shutdown), you must a new Primary Component to restore service. This must be done from the node that contains the most recent data to avoid any data loss.

Identifying the Most Advanced Node

MariaDB Galera Cluster provides a safe_to_bootstrap flag in the /var/lib/mysql/grastate.dat file to make this process safer and easier.

After a Graceful Shutdown

The last node to shut down will be the most up-to-date and will have safe_to_bootstrap: 1 set in its . You should always look for and bootstrap from this node.

After a Cluster-wide Crash

If all nodes crashed, they will all likely have safe_to_bootstrap: 0. In this case, you must manually determine the most advanced node by finding the one with the highest seqno in its grastate.dat file or by using the --wsrep-recover utility.

Bootstrapping and Restarting

Once you have identified the correct node, you will start the MariaDB service on that node only using a special bootstrap command (e.g., galera_new_cluster). After it comes online and forms a new Primary Component, you can start the other nodes normally, and they will .

For detailed, step-by-step instructions on this critical procedure, see

_{This page is licensed: CC BY-SA / Gnu FDL}

When a network failure or a crash affects over half of your cluster nodes, the cluster might lose its Primary Component. In such cases, the remaining nodes may return an Unknown command error for many queries. This behavior is a safeguard to prevent data inconsistency.

You can confirm this by checking the wsrep_cluster_status variable on all nodes:

SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';

If none of your nodes return a value of Primary, you must manually intervene to reset the Quorum and bootstrap a new Primary Component.

Find the Most Advanced Node

Before you can reset the Quorum, you must identify the most advanced node in the cluster. This is the node whose local database committed the last . Starting the cluster from any other can result in data loss.

The "Safe-to-Bootstrap" Feature

To facilitate a safe restart and prevent an administrator from choosing the wrong node, modern versions of Galera Cluster include a .

When a cluster is shut down gracefully, the last node to be stopped will be the most up-to-date. Galera tracks this and marks only that last node as safe to bootstrap from by setting a flag in its state file. If you attempt to bootstrap from a node marked as unsafe, Galera will refuse and show a message in the logs. In the case of a sudden, simultaneous crash, all nodes will be considered unsafe, requiring manual intervention.

Procedure for Selecting the Right Node

The procedure to select the right node depends on how the cluster was stopped.

Orderly Cluster Shutdown

In the case of a planned, orderly shutdown, you only need to follow the recommendation of the "Safe-to-Bootstrap" feature. On each node, inspect the /var/lib/mysql/grastate.dat and look for the one where safe_to_bootstrap: 1 is set.

Use this node for the bootstrap.

Full Cluster Crash

In the case of a hard crash, all nodes will likely have safe_to_bootstrap: 0. You must therefore manually determine which node is the most advanced.

1. On each node, run the mysqld daemon with the --wsrep-recover option. This will read the InnoDB storage engine logs and report the last known transaction position in the MariaDB error log.

2. Inspect the error log for a line similar to this:

3. Compare the sequence number (the number after the colon) from all nodes. The node with the highest sequence number is the most advanced.

Bootstrap the New Primary Component

Once you have identified the most advanced node, there are two methods to bootstrap the new Primary Component from it.

Automatic Bootstrap (Recommended)

This method is recommended if the mysqld process is still running on the most advanced node. It is non-destructive and can preserve the , increasing the chance of a fast for the other nodes.

To perform an automatic bootstrap, connect to the most advanced node with a and execute:

This node will now form a new Primary Component by itself.

Manual Bootstrap

This method involves a and a special startup of the most advanced node.

1. Ensure the mysqld service is stopped on all nodes in the cluster.

2. On the most advanced node only, start the cluster using the :

Start the Remaining Nodes

After the first node is successfully running and has formed the new Primary Component, start the MariaDB service normally on all of the other nodes.

They will detect the existing Primary Component, connect to it, and automatically initiate a to synchronize their data and rejoin the cluster.

_{This page is licensed: CC BY-SA / Gnu FDL}

Hybrid replication leverages standard, asynchronous MariaDB Replication to copy data from a synchronous MariaDB Galera Cluster to an external server or another cluster. This configuration establishes a one-way data flow, where the entire Galera Cluster serves as the source (primary) for one or more asynchronous replicas. This advanced setup combines the strengths of both replication methods: synchronous replication ensures high availability within the primary site, while asynchronous replication caters to specific use cases, allowing for flexible data distribution.

Common Use Cases

Implementing a hybrid replication setup is a powerful technique for solving several common business needs:

Key Challenges and Considerations

Before implementing a hybrid setup, it is critical to understand the technical challenges:

_{This page is licensed: CC BY-SA / Gnu FDL}

Sometimes it can be helpful to perform a "manual SST" when Galera's fail. This can be especially useful when the cluster's is very large, since a normal SST can take a long time to fail in that case.

A manual SST essentially consists of taking a backup of the donor, loading the backup on the joiner, and then manually editing the cluster state on the joiner node. This page will show how to perform this process with .

Process

Galera Load Balancer (glbd) is a simple, multi-threaded TCP connection balancer, optimized for database workloads.

It was inspired by pen, but unlike it, GLB focuses only on balancing generic TCP connections.

Features

Installation

GLB must be built from source. There are no pre-built packages.

This installs:

• glbd (daemon) → /usr/sbin

• libglb (shared library)

Running as a Service

To run as a service:

Manage with:

Configuration

GLB can be configured either via command-line options or via a configuration file.

Command-Line Options {#command-line-options}

Configuration File (glbd.cfg)

Example:

Destination Selection Policies

GLB supports five policies:

-T | --top option: restricts balancing to servers with the highest weight.

Runtime Management

GLB can be managed at runtime via:

• FIFO file

• Control socket (-c <addr:port>)

Commands

Performance Statistics

Example:

Watchdog

The watchdog module performs asynchronous health checks beyond simple TCP reachability.

Enable with:

• Runs mysql.sh with host:port as first argument.

• Exit code 0 = healthy, non-zero = failure.

• Use -i

libglb (Shared Library)

libglb enables transparent load balancing by intercepting the connect() system call.

Basic Example

Environment Variables

Operational Limits

• Limited by system open files (ulimit -n)

• With default 1024 → ~493 connections

• With 4096 (typical unprivileged user) → ~2029 connections

See Also

• – recommended modern proxy

_{This page is licensed: CC BY-SA / Gnu FDL}

MariaDB's are very useful when used with , which is primarily what that feature was developed for. Galera Cluster, on the other hand, was developed by Codership for all MySQL and MariaDB variants, and the initial development of the technology pre-dated MariaDB's implementation. As a side effect, MariaDB Galera Cluster (at least until ) only partially supports MariaDB's implementation.

GTID Support for Write Sets Replicated by Galera Cluster

Galera Cluster has its own certification-based replication method that is substantially different from . However, it would still be beneficial if MariaDB Galera Cluster was able to associate a Galera Cluster write set with a that is globally unique but that is also consistent for that write set on each cluster node.

Wsrep GTID Mode

MariaDB supports .

MariaDB has a feature called wsrep GTID mode. When this mode is enabled, MariaDB uses some tricks to try to associate each Galera Cluster write set with a that is globally unique, but that is also consistent for that write set on each cluster node. These tricks work in some cases, but can still become inconsistent among cluster nodes.

Enabling Wsrep GTID Mode

Several things need to be configured for wsrep GTID mode to work, such as

• needs to be set on all nodes in the cluster.

• needs to be set to the same value on all nodes in a given cluster, so that each cluster node uses the same domain when assigning for Galera Cluster's write sets. When replicating between two clusters, each cluster should have this set to a different value, so that each cluster uses different domains when assigning for their write sets.

• needs to be enabled on all nodes in the cluster. See .

And as an extra safety measure:

• should be set to a different value on all nodes in a given cluster, and each of these values should be different than the configured value. This is to prevent a node from using the same domain used for Galera Cluster's write sets when assigning for non-Galera transactions, such as DDL executed with set or DML executed with set.

If you want to avoid writes accidentally local GTIDS, you can avoid it with by setting this:

In this case you get an error:

You can overwrite it temporarily with:

For information on setting , see .

GTIDs for Transactions Applied by Slave Thread

If a Galera Cluster node is also a , then that node's will be applying transactions that it replicates from its replication master. If the node has set, then each transaction that the applies will also generate a Galera Cluster write set that is replicated to the rest of the nodes in the cluster.

The node acting as slave includes the transaction's original Gtid_Log_Event in the replicated write set, so all nodes should associate the write set with its original GTID. See .

_{This page is licensed: CC BY-SA / Gnu FDL}

Incremental State Transfer (IST)

An Incremental State Transfer (IST) is the fast and efficient process where a joining node receives only the missing

and can be used together. However, there are some things that have to be taken into account.

Tutorials

If you want to use and together, then the following tutorials may be useful:

In a State Snapshot Transfer (SST), the cluster provisions nodes by transferring a full data copy from one node to another. When a new node joins the cluster, the new node initiates a State Snapshot Transfer to synchronize its data with a node that is already part of the cluster.

Types of SSTs

There are two conceptually different ways to transfer a state from one MariaDB server to another:

1. Logical: The only SST method of this type is the mysqldump SST method, which uses the utility to get a logical dump of the donor. This SST method requires the joiner node to be fully initialized and ready to accept connections before the transfer. This method is, by definition, blocking, in that it blocks the donor node from modifying its state for the duration of the transfer. It is also the slowest of all, and that might be an issue in a cluster with a lot of loads.

2. Physical: SST methods of this type physically copy the data files from the donor node to the joiner node. This requires that the joiner node be initialized after the transfer. The SST method and a few other SST methods fall into this category. These SST methods are much faster than the mysqldump SST method, but they have certain limitations. For example, they can be used only on server startup, and the joiner node must be configured very similarly to the donor node (e.g., should be the same, and so on). Some of the SST methods in this category are non-blocking on the donor node, meaning that the donor node is still able to process queries while donating the SST (e.g. the SST method is non-blocking).

SST Methods

SST methods are supported via a scriptable interface. New SST methods could potentially be developed by creating new SST scripts. The scripts usually have names of the form wsrep_sst_<method> where <method> is one of the SST methods listed below.

You can choose your SST method by setting the system variable. It can be changed dynamically with on the node that you intend to be an SST donor. For example:

It can also be set in a server in an prior to starting up a node:

For an SST to work properly, the donor and joiner node must use the same SST method. Therefore, it is recommended to set to the same value on all nodes, since any node will usually be a donor or joiner node at some point.

MariaDB Galera Cluster comes with the following built-in SST methods:

mariadb-backup

This SST method uses the utility for performing SSTs. It is one of the two non-locking methods. This is the recommended SST method if you require the ability to run queries on the donor node during the SST. Note that if you use the mariadb-backup SST method, then you also need to have socat installed on the server. This is needed to stream the backup from the donor to the joiner. This is a limitation inherited from the xtrabackup-v2 SST method.

• This SST method supports

• This SST method supports .

• This SST method is available from and .

With this SST method, it is impossible to upgrade the cluster between some major versions; see .

See for more information.

rsync / rsync_wan

rsync is the default method. This method uses the utility to create a snapshot of the donor node. rsync should be available by default on all modern Linux distributions. The donor node is blocked with a read lock during the SST. This is the fastest SST method, especially for large datasets since it copies binary data. Because of that, this is the recommended SST method if you do not need to allow the donor node to execute queries during the SST.

The rsync method runs rsync in --whole-file mode, assuming that nodes are connected by fast local network links so that the default delta transfer mode would consume more processing time than it may save on data transfer bandwidth. When having a distributed cluster with slow links between nodes, the rsync_wan method runs rsync in the default delta transfer mode, which may reduce data transfer time substantially when an older datadir state is already present on the joiner node. Both methods are actually implemented by the same script, wsrep_sst_rsync_wan is just a symlink to the wsrep_sst_rsync script and the actual rsync mode to use is determined by the name the script was called by.

• This SST method supports

• This SST method supports .

The rsync SST method does not support tables created with the clause. Use the as an alternative to support this feature.

mysqldump

This SST method runs on the donor node and pipes the output to the client connected to the joiner node. The mysqldump SST method needs a username/password pair set in the variable in order to get the dump. The donor node is blocked with a read lock during the SST. This is the slowest SST method.

• This SST method supports .

• This SST method supports .

xtrabackup-v2

Percona XtraBackup is not supported in MariaDB. is the recommended backup method to use instead of Percona XtraBackup. See for more information.

This SST method uses the utility for performing SSTs. It is one of the two non-blocking methods. Note that if you use the xtrabackup-v2 SST method, you also need to have socat installed on the server. Since Percona XtraBackup is a third-party product, this SST method requires an additional installation and some additional configuration. Please refer to for information from the vendor.

• This SST method does not support

• This SST method does not support .

This SST method is available from MariaDB Galera Cluster 5.5.37 and MariaDB Galera Cluster 10.0.10.

See xtrabackup-v2 SST method for more information.

xtrabackup

Percona XtraBackup is not supported in MariaDB. is the recommended backup method to use instead of Percona XtraBackup. See for more information.

This SST method is an older SST method that uses the utility for performing SSTs. The xtrabackup-v2 SST method should be used instead of the xtrabackup SST method starting from .

• This SST method does not support

• This SST method does not support .

Authentication

All SST methods except rsync require authentication via username and password. You can tell the client what username and password to use by setting the system variable. It can be changed dynamically with on the node that you intend to be a SST donor. For example:

It can also be set in a server in an prior to starting up a node:

Some do not require a password. For example, the and authentication plugins do not require a password. If you are using a user account that does not require a password in order to log in, then you can just leave the password component of empty. For example:

See the relevant description or page for each SST method to find out what privileges need to be to the user and whether the privileges are needed on the donor node or joiner node for that method.

SSTs and Systemd

MariaDB's unit file has a default startup timeout of about 90 seconds on most systems. If an SST takes longer than this default startup timeout on a joiner node, then systemd will assume that mysqld has failed to startup, which causes systemd to kill the mysqld process on the joiner node. To work around this, you can reconfigure the MariaDB systemd unit to have an infinite timeout, such as by executing one of the following commands:

If you are using systemd 228 or older, then you can execute the following to set an infinite timeout:

, so if you are using systemd 229 or later, then you can execute the following to set an infinite timeout:

See for more details.

Note that that allows services to extend the startup timeout during long-running processes. Starting with , , and , on systems with systemd versions that support it, MariaDB uses this feature to extend the startup timeout during long SSTs. Therefore, if you are using systemd 236 or later, then you should not need to manually override TimeoutStartSec, even if your SSTs run for longer than the configured value. See for more information.

SST Failure

An SST failure generally renders the joiner node unusable. Therefore, when an SST failure is detected, the joiner node will abort.

Restarting a node after a mysqldump SST failure may require manual restoration of the administrative tables.

SSTs and Data at Rest Encryption

Look at the description of each SST method to determine which methods support .

For logical SST methods like mysqldump, each node should be able to have different . For physical SST methods, all nodes need to have the same , since the donor node will copy encrypted data files to the joiner node, and the joiner node will need to be able to decrypt them.

Minimal Cluster Size

In order to avoid a split-brain condition, the minimum recommended number of nodes in a cluster is 3.

When using an SST method that blocks the donor, there is yet another reason to require a minimum of 3 nodes. In a 3-node cluster, if one node is acting as an SST joiner and one other node is acting as an SST donor, then there is still one more node to continue executing queries.

Manual SSTs

In some cases, if Galera Cluster's automatic SSTs repeatedly fail, then it can be helpful to perform a "manual SST". See the following pages on how to do that:

Known Issues

mysqld_multi

SST scripts can't currently read the mysqld<#> in an that are read by instances managed by .

See for more information.

See Also

_{This page is licensed: CC BY-SA / Gnu FDL}

can be used to replicate between and MariaDB Server. This article will discuss how to do that.

Configuring the Cluster

Before we set up replication, we need to ensure that the cluster is configured properly. This involves the following steps:

can be used for replication between two . This article will discuss how to do that.

Configuring the Clusters

Before we set up replication, we need to ensure that the clusters are configured properly. This involves the following steps:

The mariabackup SST method uses the utility for performing SSTs. It is one of the methods that does not block the donor node. mariadb-backup was originally forked from , and similarly, the mariabackup SST method was originally forked from the xtrabackup-v2 SST method.

Choosing mariadb-backup for SSTs

To use the mariadb-backup SST method, you must set the on both the donor and joiner node. It can be changed dynamically with on the node that you intend to be an SST donor. For example:

It can be set in a server in an prior to starting up a node:

For an SST to work properly, the donor and joiner node must use the same SST method. Therefore, it is recommended to set to the same value on all nodes, since any node will usually be a donor or joiner node at some point.

Major Version Upgrades

The default method wsrep_sst_method=rsync works for major-version upgrades; see .

Configuration Options

The mariabackup SST method is configured by placing options in the [sst] section of a MariaDB configuration file (e.g., /etc/my.cnf.d/server.cnf). These settings are parsed by the wsrep_sst_mariabackup and wsrep_sst_common scripts.

Primary Transfer and Format Options

These options control the core data transfer mechanism.

Compression Options

These options configure on-the-fly compression to reduce network bandwidth.

Authentication and Security (TLS)

These options manage user authentication and stream encryption.

Logging and Miscellaneous Options

Pass-through mariadb-backup Options

This feature allows mariadb-backup specific options to be passed through the SST script.

Example: Using Native Encryption and Threading

Authentication and Privileges

To use the mariadb-backup SST method, the utility must be able to authenticate locally on the donor node to create a backup stream. There are two ways to manage this authentication:

Automatic User Account Management (ES 11.4+)

Starting with MariaDB Enterprise Server 11.4, the cluster can automatically manage the SST user account. This method is more secure and requires less configuration because it avoids storing plain-text passwords in configuration files.

When this feature is used:

1. The donor node automatically creates a temporary internal user (e.g., 'wsrep.sst. <timestamp>_<node_id>'@localhost) with a generated password when the SST process begins.

2. The necessary privileges (RELOAD, PROCESS, LOCK TABLES, etc.) are automatically granted to this temporary user.

To enable automatic user management:

Ensure that the system variable is not set (or is left blank) in your configuration file.

Manual User Configuration

For versions prior to 11.4, or if you prefer to manage the user manually, you must create a user and provide the credentials to the server.

You can tell the donor node what username and password to use by setting the system variable. It can be changed dynamically with SET GLOBAL on the node that you intend to be an SST donor:

It can also be set in a server in an prior to starting up a node:

Some do not require a password. For example, the unix_socket and gssapi authentication plugins do not require a password. If you are using a user account that does not require a password in order to log in, then you can just leave the password component of empty. For example:

The user account that performs the backup for the SST needs to have the same privileges as , which are the RELOAD, PROCESS, LOCK TABLES and BINLOG MONITOR, REPLICA MONITOR . To be safe, ensure that these privileges are set on each node in your cluster. mariadb-backup connects locally on the donor node to perform the backup, so the following user should be sufficient:

Passwordless Authentication - Unix Socket

It is possible to use the authentication plugin for the user account that performs SSTs. This would provide the benefit of not needing to configure a plain-text password in .

The user account would have to have the same name as the operating system user account that is running the mysqld process. On many systems, this is the user account configured as the user option, and it tends to default to mysql.

For example, if the authentication plugin is already installed, then you could execute the following to create the user account:

To configure , set the following in a server in an prior to starting up a node:

Passwordless Authentication - GSSAPI

It is possible to use the authentication plugin for the user account that performs SSTs. This would provide the benefit of not needing to configure a plain-text password in .

The following steps would need to be done beforehand:

• You need a KDC running or .

• You will need to for the MariaDB server.

• You will need to containing the authentication plugin.

For example, you could execute the following to create the user account in MariaDB:

To configure , set the following in a server in an prior to starting up a node:

Choosing a Donor Node

When mariadb-backup is used to create the backup for the SST on the donor node, mariadb-backup briefly requires a system-wide lock at the end of the backup. This is done with .

If a specific node in your cluster is acting as the primary node by receiving all of the application's write traffic, then this node should not usually be used as the donor node, because the system-wide lock could interfere with the application. In this case, you can define one or more preferred donor nodes by setting the system variable.

For example, let's say that we have a 5-node cluster with the nodes node1, node2, node3, node4, and node5, and let's say that node1 is acting as the primary node. The preferred donor nodes for node2 could be configured by setting the following in a server in an prior to starting up a node:

The trailing comma tells the server to allow any other node as donor when the preferred donors are not available. Therefore, if node1 is the only node left in the cluster, the trailing comma allows it to be used as the donor node.

Socat Dependency

During the SST process, the donor node uses socat to stream the backup to the joiner node. Then the joiner node prepares the backup before restoring it. The socat utility must be installed on both the donor node and the joiner node in order for this to work. Otherwise, the MariaDB error log will contain an error like:

Installing Socat on RHEL/CentOS

On RHEL/CentOS, socat can be installed from the repository.

TLS

This SST method supports three different TLS methods. The specific method can be selected by setting the encrypt option in the [sst] section of the MariaDB configuration file. The options are:

• TLS using OpenSSL encryption built into socat (encrypt=2)

• TLS using OpenSSL encryption with Galera-compatible certificates and keys (encrypt=3)

• TLS using OpenSSL encryption with standard MySQL/MariaDB SSL certificates (encrypt=4

TLS Using OpenSSL Encryption Built into Socat

To generate keys compatible with this encryption method, follow .

First, generate the keys and certificates:

On some systems, you may also have to add dhparams to the certificate:

Next, copy the certificate and keys to all nodes in the cluster.

When done, configure the following on all nodes in the cluster:

TLS Using OpenSSL Encryption With Galera-Compatible Certificates and Keys

To generate keys compatible with this encryption method, follow .

First, generate the keys and certificates:

Next, copy the certificate and keys to all nodes in the cluster.

When done, configure the following on all nodes in the cluster:

Logs

The mariadb-backup SST method has its own logging outside of the MariaDB Server logging.

Logging to SST Logs

Logging for mariadb-backup SSTs works the following way.

By default, on the donor node, it logs to mariadb-backup.backup.log. This log file is located in the .

By default, on the joiner node, it logs to mariadb-backup.prepare.log and mariadb-backup.move.log These log files are also located in the datadir.

By default, before a new SST is started, existing mariadb-backup SST log files are compressed and moved to /tmp/sst_log_archive. This behavior can be disabled by setting sst-log-archive=0 in the [sst] in an . Similarly, the archive directory can be changed by setting sst-log-archive-dir:

See for more information.

Logging to Syslog

Redirect the SST logs to the syslog instead, by setting the following in the [sst] in an :

You can also redirect the SST logs to the syslog by setting the following in the [mysqld_safe] in an :

Performing SSTs With IPv6 Addresses

If you are performing SSTs with IPv6 addresses, then the socat utility needs to be passed the pf=ip6 option. This can be done by setting the sockopt option in the [sst] in an :

See for more information.

Manual SST With mariadb-backup

If Galera Cluster's automatic SSTs repeatedly fail, it can be helpful to perform a "manual SST"; see:

See Also

_{This page is licensed: CC BY-SA / Gnu FDL}