From a , you can check the status of write-set replication throughout the cluster using standard queries. Status variables that relate to write-set replication have the prefix wsrep_, meaning that you can display them all using the following query:

Understanding Quorum and Cluster Integrity

The most fundamental aspect of a healthy cluster is Quorum. Quorum is a mechanism that ensures data consistency by requiring a majority of nodes to be online and in communication to form a Primary Component. Only the Primary Component will process transactions. This prevents "split-brain" scenarios where a network partition could otherwise lead to data conflicts.

You can check the cluster's integrity and Quorum status using these key variables. For a healthy cluster, the values for these variables must be identical on every node.

Checking Individual Node Status

You can monitor the status of individual nodes to ensure they are in working order and able to receive write-sets.

Understanding Galera Node States

The value of wsrep_local_state_comment tells you exactly what a node is doing. The most common states include:

Checking Replication Health

These can help identify performance issues and bottlenecks.

Recovering a Cluster After a Full Outage

If the entire cluster shuts down or , you must manually re-establish a Primary Component by bootstrapping from the most advanced node.

_{This page is licensed: CC BY-SA / Gnu FDL}

While a client application can connect directly to any node in a MariaDB Galera Cluster, this is not a practical approach for a production environment. A direct connection creates a single point of failure and does not allow the application to take advantage of the cluster's high availability and read-scaling capabilities.

A load balancer or database proxy is an essential component that sits between your application and the cluster. Its primary responsibilities are:

• Provide a Single Endpoint: Your application connects to the load balancer's virtual IP address, not to the individual database nodes.

• Health Checks: The load balancer constantly monitors the health of each cluster node (e.g., is it Synced? is it up or down?).

• Traffic Routing: It intelligently distributes incoming client connections and queries among the healthy nodes in the cluster.

• Automatic Failover: If a node fails, the load balancer automatically stops sending traffic to it, providing seamless failover for your application.

Recommended Load Balancer: MariaDB MaxScale

For MariaDB Galera Cluster, the recommended load balancer is MariaDB MaxScale. Unlike a generic TCP proxy, MaxScale is a database-aware proxy that understands the Galera Cluster protocol. This allows it to make intelligent routing decisions based on the real-time state of the cluster nodes.

Common Routing Strategies

A database-aware proxy like MaxScale can be configured to use several different routing strategies.

Read-Write Splitting (Recommended)

This is the most common and highly recommended strategy for general-purpose workloads.

• How it Works: The load balancer is configured to send all write operations (INSERT, UPDATE, DELETE) to a single, designated primary node. All read operations (SELECT) are then distributed across the remaining available nodes.

• Advantages:

  • Minimizes Transaction Conflicts: By directing all writes to one node, you significantly reduce the chance of two nodes trying to modify the same row at the same time, which would lead to deadlocks and transaction rollbacks.

Read Connection Load Balancing

In this simpler strategy, the load balancer distributes all connections evenly across all available nodes.

• How it Works: Each new connection is sent to the next available node in a round-robin fashion.

• Disadvantages: This approach can easily lead to transaction conflicts if your application sends writes to multiple nodes simultaneously. It is generally only suitable for applications that are almost exclusively read-only.

Other Load Balancing Solutions

While MariaDB MaxScale is the recommended solution, other proxies and load balancers can also be used with Galera Cluster, including:

• ProxySQL: Another popular open-source, database-aware proxy.

• HAProxy: A very common and reliable TCP load balancer. When used with Galera, HAProxy is typically configured with a simple TCP health check or a custom script to determine node availability.

• Cloud Load Balancers: Cloud providers like AWS (ELB/NLB), Google Cloud, and Azure offer native load balancing services that can be used to distribute traffic across a Galera Cluster.

When a network failure or a crash affects over half of your cluster nodes, the cluster might lose its . In such cases, the remaining nodes may return an Unknown command error for many queries. This behavior is a safeguard to prevent data inconsistency.

You can confirm this by checking the wsrep_cluster_status on all nodes:

If none of your nodes return a value of Primary, you must manually intervene to reset the Quorum and bootstrap a new Primary Component.

Quorum is essential for maintaining data consistency in a MariaDB Galera Cluster by safeguarding against network partitions or node failures. It ensures that the cluster processes database queries and transactions only when a majority of nodes are operational, healthy, and in communication.

Primary Component

This majority group is known as the Primary Component. Nodes not in this group switch to a non-primary state, halting queries and entering a read-only "safe mode" to prevent data discrepancies. The primary function of Quorum is to avoid "split-brain" scenarios, which occur when network partitions lead to parts of the cluster operating independently and accepting writes. By ensuring only the partition with a majority of nodes becomes the Primary Component, Quorum effectively prevents these inconsistencies.

Quorum Calculation

Quorum is achieved when more than 50% of the total nodes in the last known membership are in communication.

• Odd Number of Nodes (Recommended): In a 3-node cluster, a majority is 2. The cluster can tolerate the failure of 1 node and remain operational.

• Even Number of Nodes: In a 2-node cluster, a majority is also 2. If one node fails, the remaining node represents only 50% of the cluster, which is not a majority, and it will lose Quorum. This is why a 2-node cluster has no fault tolerance without an external .

The Galera Arbitrator (garbd)

The Galera Arbitrator (garbd) is the standard solution for clusters with an even number of nodes. It is a lightweight, stateless daemon that acts as a voting member in the cluster without being a full database node. It participates in Quorum calculations, effectively turning an even-numbered cluster into an odd-numbered one. In the diagram, in a 2-node cluster, adding garbd makes the total number of voting members 3, allowing the cluster to maintain Quorum if one database node fails.

Understanding and Recovering from a Split-Brain

A split-brain occurs when a network partition divides the cluster and no resulting group of nodes has a majority (e.g., a 4-node cluster splitting into two groups of 2). By design, both halves of the cluster will fail to achieve a majority, and all nodes will enter a non-Primary state.

If you need to restore service before the network issue is fixed, you must manually intervene:

1. Choose ONE side of the partition to become the new Primary Component.

2. On a single node within that chosen group, execute the following command:

The nodes in this group will now form a new Primary Component. When network connectivity is restored, the nodes from the other partition will automatically rejoin.

Advanced Quorum Control

As a more advanced alternative to garbd for fine-grained control, nodes can also be assigned a specific voting weight.

Monitoring Quorum and Cluster Membership

You can check the health of the cluster and its Quorum status at any time by querying the following .

Recovering from a Full Cluster Shutdown

If the entire cluster loses Quorum (e.g., from a simultaneous crash or shutdown), you must a new Primary Component to restore service. This must be done from the node that contains the most recent data to avoid any data loss.

Identifying the Most Advanced Node

MariaDB Galera Cluster provides a safe_to_bootstrap flag in the /var/lib/mysql/grastate.dat file to make this process safer and easier.

After a Graceful Shutdown

The last node to shut down will be the most up-to-date and will have safe_to_bootstrap: 1 set in its . You should always look for and bootstrap from this node.

After a Cluster-wide Crash

If all nodes crashed, they will all likely have safe_to_bootstrap: 0. In this case, you must manually determine the most advanced node by finding the one with the highest seqno in its grastate.dat file or by using the --wsrep-recover utility.

Bootstrapping and Restarting

Once you have identified the correct node, you will start the MariaDB service on that node only using a special bootstrap command (e.g., galera_new_cluster). After it comes online and forms a new Primary Component, you can start the other nodes normally, and they will .

For detailed, step-by-step instructions on this critical procedure, see

_{This page is licensed: CC BY-SA / Gnu FDL}

MariaDB's are very useful when used with , which is primarily what that feature was developed for. Galera Cluster, on the other hand, was developed by Codership for all MySQL and MariaDB variants, and the initial development of the technology pre-dated MariaDB's implementation. As a side effect, MariaDB Galera Cluster (at least until ) only partially supports MariaDB's implementation.

GTID Support for Write Sets Replicated by Galera Cluster

Galera Cluster has its own certification-based replication method that is substantially different from . However, it would still be beneficial if MariaDB Galera Cluster was able to associate a Galera Cluster write set with a that is globally unique but that is also consistent for that write set on each cluster node.

Wsrep GTID Mode

MariaDB supports .

MariaDB has a feature called wsrep GTID mode. When this mode is enabled, MariaDB uses some tricks to try to associate each Galera Cluster write set with a that is globally unique, but that is also consistent for that write set on each cluster node. These tricks work in some cases, but can still become inconsistent among cluster nodes.

Enabling Wsrep GTID Mode

Several things need to be configured for wsrep GTID mode to work, such as

• needs to be set on all nodes in the cluster.

• needs to be set to the same value on all nodes in a given cluster, so that each cluster node uses the same domain when assigning for Galera Cluster's write sets. When replicating between two clusters, each cluster should have this set to a different value, so that each cluster uses different domains when assigning for their write sets.

• needs to be enabled on all nodes in the cluster. See .

And as an extra safety measure:

• should be set to a different value on all nodes in a given cluster, and each of these values should be different than the configured value. This is to prevent a node from using the same domain used for Galera Cluster's write sets when assigning for non-Galera transactions, such as DDL executed with set or DML executed with set.

If you want to avoid writes accidentally local GTIDS, you can avoid it with by setting this:

In this case you get an error:

You can overwrite it temporarily with:

For information on setting , see .

GTIDs for Transactions Applied by Slave Thread

If a Galera Cluster node is also a , then that node's will be applying transactions that it replicates from its replication master. If the node has set, then each transaction that the applies will also generate a Galera Cluster write set that is replicated to the rest of the nodes in the cluster.

The node acting as slave includes the transaction's original Gtid_Log_Event in the replicated write set, so all nodes should associate the write set with its original GTID. See .

_{This page is licensed: CC BY-SA / Gnu FDL}

Hybrid replication leverages standard, asynchronous MariaDB Replication to copy data from a synchronous MariaDB Galera Cluster to an external server or another cluster. This configuration establishes a one-way data flow, where the entire Galera Cluster serves as the source (primary) for one or more asynchronous replicas. This advanced setup combines the strengths of both replication methods: synchronous replication ensures high availability within the primary site, while asynchronous replication caters to specific use cases, allowing for flexible data distribution.

Common Use Cases

Implementing a hybrid replication setup is a powerful technique for solving several common business needs:

Key Challenges and Considerations

Before implementing a hybrid setup, it is critical to understand the technical challenges:

_{This page is licensed: CC BY-SA / Gnu FDL}

Incremental State Transfer (IST)

An Incremental State Transfer (IST) is the fast and efficient process where a joining node receives only the missing transactions it needs to catch up with the cluster, rather than receiving a full copy of the entire database.

This is the preferred provisioning method because it is:

• Fast: Transferring only the missing changes is significantly faster than copying the entire dataset.

• Non-Blocking: The donor node can continue to serve read and write traffic while an IST is in progress.

Conditions for IST

IST is an automatic process, but it is only possible if the following conditions are met:

• The joining node has previously been a member of the cluster (its matches the cluster's).

• All of the that the joiner is missing are still available in the donor node's Write-set Cache (GCache).

If these conditions are not met, the cluster automatically falls back to performing a full State Snapshot Transfer (SST).

Skipping Foreign Key Checks

Appliers need to verify foreign key constraints during normal operation in multi-active topologies. Therefore, appliers are configured to enable FK checking.

However, during node joining, in IST and latter catch-up period, the node is still idle (from local connections), and the only source for incoming transactions is the cluster sending certified write sets for applying. IST happens with parallel applying — there is a possibility that foreign key check cause lock conflicts between appliers accessing FK child and parent tables. Also, excessive FK checking slows down the IST process.

To address that issue, you can relax FK checks for appliers during IST and catch-up periods. The relaxed FK check mode is configurable by :

When this operation mode is set, and the node is processing IST or catch-up, appliers skip FK checking.

The Write-Set Cache (GCache)

The GCache is a special cache on each node whose primary purpose is to store recent write-sets specifically to facilitate Incremental State Transfers. The size and configuration of the GCache are therefore critical for the cluster's recovery speed and .

How the GCache Enables IST

When a node attempts to rejoin the cluster, it reports the of the last transaction it successfully applied. The potential donor node then checks its GCache for the very next seqno in that sequence.

Configuring the GCache

You can control the GCache behavior with several in the [galera] section of your configuration file (my.cnf).

Tuning gcache.size

The gcache.size parameter is the most critical setting for ensuring nodes can use IST. A GCache that is too small is the most common reason for a cluster falling back to a full SST.

The ideal size depends on your cluster's write rate and the amount of downtime you want to tolerate for a node before forcing an SST. For instance, do you want a node that is down for 1 hour for maintenance to recover instantly (IST), or can you afford a full SST?

Calculating Size Based on Write Rate

The most accurate way to size your GCache is to base it on your cluster's write rate.

A General Heuristic for Sizing

If you cannot calculate the write rate, you can use a simpler heuristic based on your data directory size as a starting point.

1. Start with the size of your data directory.

2. Subtract the size of the GCache's ring buffer file itself (default: galera.cache).

3. Consider your SST method:

   • If you use mysqldump

_{This page is licensed: CC BY-SA / Gnu FDL}

and MariaDB Galera Cluster can be used together. However, there are some things that have to be taken into account.

Tutorials

If you want to use and MariaDB Galera Cluster together, then the following tutorials may be useful:

• Configuring MariaDB Replication between MariaDB Galera Cluster and MariaDB Server

Configuring a Cluster Node as a Replication Master

If a Galera Cluster node is also a , then some additional configuration may be needed.

Like with , write sets that are received by a node with are not written to the by default.

If the node is a replication master, then its replication slaves only replicate transactions that are in the binary log, so this means that the transactions that correspond to Galera Cluster write-sets would not be replicated by any replication slaves by default. If you would like a node to write its replicated write sets to the , then you will have to set . If the node has any replication slaves, then this would also allow those slaves to replicate the transactions that corresponded to those write sets.

See for more information.

Configuring a Cluster Node as a Replication Slave

If a Galera Cluster node is also a , then some additional configuration may be needed.

If the node is a replication slave, then the node's will be applying transactions that it replicates from its replication master. Transactions applied by the slave SQL thread will only generate Galera Cluster write-sets if the node has set. Therefore, in order to replicate these transactions to the rest of the nodes in the cluster, must be set.

If the node is a replication slave, then it is probably also a good idea to enable . When this is enabled, the node will restart its whenever it rejoins the cluster.

Parallel Replication Support

Historically, Galera Cluster nodes acting as asynchronous replication slaves were restricted to single-threaded execution (slave_parallel_threads=0). Enabling parallel replication often resulted in deadlocks due to conflicts between ordering and Galera's internal pre-commit ordering.

As of MariaDB 12.1.1, this limitation has been resolved.

On supported versions, you can safely configure slave_parallel_threads to a value greater than 0 to improve the performance of incoming replication streams.

Recommended Configuration (MariaDB 12.1.1+):

Replication Filters

Both and support , so extra caution must be taken when using all of these features together. See for more details on how MariaDB Galera Cluster interprets replication filters.

Setting server_id on Cluster Nodes

Setting the Same server_id on Each Cluster Node

It is most common to set to the same value on each node in a given cluster. Since uses a , all nodes should have the same data, so in a logical sense, a cluster can be considered in many cases a single logical server for purposes related to . The of each cluster node might even contain roughly the same transactions and if is set and if is enabled and if non-Galera transactions are not being executed on any nodes.

Setting a Different server_id on Each Cluster Node

There are cases when it might make sense to set a different value on each node in a given cluster. For example, if is set and if another cluster or a standard MariaDB Server is using to replicate transactions from each cluster node individually, then it would be required to set a different value on each node for this to work.

Keep in mind that if replication is set up in a scenario where each cluster node has a different value, and if the replication topology is set up in such a way that a cluster node can replicate the same transactions through Galera and through MariaDB replication, then you may need to configure the cluster node to ignore these transactions when setting up MariaDB replication. You can do so by setting to the server IDs of all nodes in the same cluster when executing . For example, this might be required when circular replication is set up between two separate clusters, and each cluster node has a different value, and each cluster has set.

_{This page is licensed: CC BY-SA / Gnu FDL}

Sometimes it can be helpful to perform a "manual SST" when Galera's normal SSTs fail. This can be especially useful when the cluster's is very large, since a normal SST can take a long time to fail in that case.

A manual SST essentially consists of taking a backup of the donor, loading the backup on the joiner, and then manually editing the cluster state on the joiner node. This page will show how to perform this process with .

Process

For example, a very common version number is "2.1".

Galera Load Balancer (glbd) is a simple, multi-threaded TCP connection balancer, optimized for database workloads.

It was inspired by pen, but unlike it, GLB focuses only on balancing generic TCP connections.

Features

Installation

GLB must be built from source. There are no pre-built packages.

This installs:

• glbd (daemon) → /usr/sbin

• libglb (shared library)

Running as a Service

To run as a service:

Manage with:

Configuration

GLB can be configured either via command-line options or via a configuration file.

Command-Line Options {#command-line-options}

Configuration File (glbd.cfg)

Example:

Destination Selection Policies

GLB supports five policies:

-T | --top option: restricts balancing to servers with the highest weight.

Runtime Management

GLB can be managed at runtime via:

• FIFO file

• Control socket (-c <addr:port>)

Commands

Performance Statistics

Example:

Watchdog

The watchdog module performs asynchronous health checks beyond simple TCP reachability.

Enable with:

• Runs mysql.sh with host:port as first argument.

• Exit code 0 = healthy, non-zero = failure.

• Use -i to set check interval.

libglb (Shared Library)

libglb enables transparent load balancing by intercepting the connect() system call.

Basic Example

Environment Variables

Operational Limits

• Limited by system open files (ulimit -n)

• With default 1024 → ~493 connections

• With 4096 (typical unprivileged user) → ~2029 connections

See Also

• – recommended modern proxy

_{This page is licensed: CC BY-SA / Gnu FDL}

can be used to replicate between MariaDB Galera Cluster and MariaDB Server. This article will discuss how to do that.

Configuring the Cluster

Before we set up replication, we need to ensure that the cluster is configured properly. This involves the following steps:

• Set on all nodes in the cluster. See Configuring MariaDB Galera Cluster: Writing Replicated Write Sets to the Binary Log and Using MariaDB Replication with MariaDB Galera Cluster: Configuring a Cluster Node as a Replication Master for more information on why this is important. It is also needed to .

• Set to the same value on all nodes in the cluster. See for more information on what this means.

Configuring Wsrep GTID Mode

If you want to use replication, then you also need to configure some things to . For example:

• needs to be set on all nodes in the cluster.

• needs to be set to the same value on all nodes in the cluster so that each cluster node uses the same domain when assigning for Galera Cluster's write sets.

• needs to be enabled on all nodes in the cluster. See about that.

• needs to be set to the same path on all nodes in the cluster. See

And as an extra safety measure:

• should be set to a different value on all nodes in a given cluster, and each of these values should be different than the configured value. This is to prevent a node from using the same domain used for Galera Cluster's write sets when assigning for non-Galera transactions, such as DDL executed with set or DML executed with set.

Configuring the Replica

Before we set up replication, we also need to ensure that the MariaDB Server replica is configured properly. This involves the following steps:

• Set to a different value than the one that the cluster nodes are using.

• Set to a value that is different than the and values that the cluster nodes are using.

• Set and if you want the replica to log the transactions that it replicates.

Setting up Replication

Our process to set up replication is going to be similar to the process described at , but it will be modified a bit to work in this context.

Start the cluster

The very first step is to start the nodes in the first cluster. The first node will have to be . The other nodes can be started normally.

Once the nodes are started, you need to pick a specific node that will act as the replication primary for the MariaDB Server.

Start the New Replica

Now that the backup has been restored to the MariaDB Server replica, you can start the MariaDB Server process.

Setting up Circular Replication

You can also set up between the cluster and MariaDB Server, which means that the MariaDB Server replicates from the cluster, and the cluster also replicates from the MariaDB Server.

_{This page is licensed: CC BY-SA / Gnu FDL}

In a State Snapshot Transfer (SST), the cluster provisions nodes by transferring a full data copy from one node to another. When a new node joins the cluster, the new node initiates a State Snapshot Transfer to synchronize its data with a node that is already part of the cluster.

Types of SSTs

There are two conceptually different ways to transfer a state from one MariaDB server to another:

1. Logical: The only SST method of this type is the mysqldump

can be used for replication between two MariaDB Galera Clusters. This article will discuss how to do that.

Configuring the Clusters

Before we set up replication, we need to ensure that the clusters are configured properly. This involves the following steps:

• Set on all nodes in both clusters. See Configuring MariaDB Galera Cluster: Writing Replicated Write Sets to the Binary Log and Using MariaDB Replication with MariaDB Galera Cluster: Configuring a Cluster Node as a Replication Master for more information on why this is important. This is also needed to .

• Set to the same value on all nodes in a given cluster, but be sure to use a different value in each cluster. See for more information on what this means.

Configuring Wsrep GTID Mode

If you want to use replication, then you also need to configure some things to . For example:

• needs to be set on all nodes in each cluster.

• needs to be set to the same value on all nodes in a given cluster so that each cluster node uses the same domain when assigning for Galera Cluster's write sets. Each cluster should have this set to a different value so that each cluster uses different domains when assigning for their write sets.

• needs to be enabled on all nodes in the cluster. See about that.

And as an extra safety measure:

• should be set to a different value on all nodes in a given cluster, and each of these values should be different than the configured value. This is to prevent a node from using the same domain used for Galera Cluster's write sets when assigning for non-Galera transactions, such as DDL executed with set or DML executed with set.

Configuring Parallel Replication

To improve the performance of the replication stream between clusters, it is recommended to enable on the nodes in the destination cluster (the cluster acting as the replica).

Setting up Replication

Our process to set up replication is going to be similar to the process described at , but it will be modified a bit to work in this context.

Setting up Circular Replication

You can also set up between the two clusters, which means that the second cluster replicates from the first cluster, and the first cluster also replicates from the second cluster.

The mariabackup SST method uses the utility for performing SSTs. It is one of the methods that does not block the donor node. mariadb-backup was originally forked from , and similarly, the mariabackup SST method was originally forked from the xtrabackup-v2 SST method.

Choosing mariadb-backup for SSTs

To use the mariadb-backup SST method, you must set the on both the donor and joiner node. It can be changed dynamically with on the node that you intend to be an SST donor. For example:

It can be set in a server in an prior to starting up a node:

For an SST to work properly, the donor and joiner node must use the same SST method. Therefore, it is recommended to set to the same value on all nodes, since any node will usually be a donor or joiner node at some point.

Major Version Upgrades

The default method wsrep_sst_method=rsync works for major-version upgrades; see .

Configuration Options

The mariabackup SST method is configured by placing options in the [sst] section of a MariaDB configuration file (e.g., /etc/my.cnf.d/server.cnf). These settings are parsed by the wsrep_sst_mariabackup and wsrep_sst_common scripts.

Primary Transfer and Format Options

These options control the core data transfer mechanism.

Compression Options

These options configure on-the-fly compression to reduce network bandwidth.

Authentication and Security (TLS)

These options manage user authentication and stream encryption.

Logging and Miscellaneous Options

Pass-through mariadb-backup Options

This feature allows mariadb-backup specific options to be passed through the SST script.

Example: Using Native Encryption and Threading

Authentication and Privileges

To use the mariadb-backup SST method, the utility must be able to authenticate locally on the donor node to create a backup stream. There are two ways to manage this authentication:

Automatic User Account Management (ES 11.4+)

Starting with MariaDB Enterprise Server 11.4, the cluster can automatically manage the SST user account. This method is more secure and requires less configuration because it avoids storing plain-text passwords in configuration files.

When this feature is used:

1. The donor node automatically creates a temporary internal user (e.g., 'wsrep.sst. <timestamp>_<node_id>'@localhost) with a generated password when the SST process begins.

2. The necessary privileges (RELOAD, PROCESS, LOCK TABLES, etc.) are automatically granted to this temporary user.

3. Once the SST process completes, the donor node automatically drops the user.

To enable automatic user management:

Ensure that the system variable is not set (or is left blank) in your configuration file.

Manual User Configuration

For versions prior to 11.4, or if you prefer to manage the user manually, you must create a user and provide the credentials to the server.

You can tell the donor node what username and password to use by setting the system variable. It can be changed dynamically with SET GLOBAL on the node that you intend to be an SST donor:

It can also be set in a server in an prior to starting up a node:

Some do not require a password. For example, the unix_socket and gssapi authentication plugins do not require a password. If you are using a user account that does not require a password in order to log in, then you can just leave the password component of empty. For example:

The user account that performs the backup for the SST needs to have the same privileges as , which are the RELOAD, PROCESS, LOCK TABLES and BINLOG MONITOR, REPLICA MONITOR . To be safe, ensure that these privileges are set on each node in your cluster. mariadb-backup connects locally on the donor node to perform the backup, so the following user should be sufficient:

Passwordless Authentication - Unix Socket

It is possible to use the authentication plugin for the user account that performs SSTs. This would provide the benefit of not needing to configure a plain-text password in .

The user account would have to have the same name as the operating system user account that is running the mysqld process. On many systems, this is the user account configured as the user option, and it tends to default to mysql.

For example, if the authentication plugin is already installed, then you could execute the following to create the user account:

To configure , set the following in a server in an prior to starting up a node:

Passwordless Authentication - GSSAPI

It is possible to use the authentication plugin for the user account that performs SSTs. This would provide the benefit of not needing to configure a plain-text password in .

The following steps would need to be done beforehand:

• You need a KDC running or .

• You will need to for the MariaDB server.

• You will need to containing the authentication plugin.

• You will need to in MariaDB, so that the authentication plugin is available to use.

For example, you could execute the following to create the user account in MariaDB:

To configure , set the following in a server in an prior to starting up a node:

Choosing a Donor Node

When mariadb-backup is used to create the backup for the SST on the donor node, mariadb-backup briefly requires a system-wide lock at the end of the backup. This is done with .

If a specific node in your cluster is acting as the primary node by receiving all of the application's write traffic, then this node should not usually be used as the donor node, because the system-wide lock could interfere with the application. In this case, you can define one or more preferred donor nodes by setting the system variable.

For example, let's say that we have a 5-node cluster with the nodes node1, node2, node3, node4, and node5, and let's say that node1 is acting as the primary node. The preferred donor nodes for node2 could be configured by setting the following in a server in an prior to starting up a node:

The trailing comma tells the server to allow any other node as donor when the preferred donors are not available. Therefore, if node1 is the only node left in the cluster, the trailing comma allows it to be used as the donor node.

Socat Dependency

During the SST process, the donor node uses socat to stream the backup to the joiner node. Then the joiner node prepares the backup before restoring it. The socat utility must be installed on both the donor node and the joiner node in order for this to work. Otherwise, the MariaDB error log will contain an error like:

Installing Socat on RHEL/CentOS

On RHEL/CentOS, socat can be installed from the repository.

TLS

This SST method supports three different TLS methods. The specific method can be selected by setting the encrypt option in the [sst] section of the MariaDB configuration file. The options are:

• TLS using OpenSSL encryption built into socat (encrypt=2)

• TLS using OpenSSL encryption with Galera-compatible certificates and keys (encrypt=3)

• TLS using OpenSSL encryption with standard MySQL/MariaDB SSL certificates (encrypt=4)

TLS Using OpenSSL Encryption Built into Socat

To generate keys compatible with this encryption method, follow .

First, generate the keys and certificates:

On some systems, you may also have to add dhparams to the certificate:

Next, copy the certificate and keys to all nodes in the cluster.

When done, configure the following on all nodes in the cluster:

TLS Using OpenSSL Encryption With Galera-Compatible Certificates and Keys

To generate keys compatible with this encryption method, follow .

First, generate the keys and certificates:

Next, copy the certificate and keys to all nodes in the cluster.

When done, configure the following on all nodes in the cluster:

Logs

The mariadb-backup SST method has its own logging outside of the MariaDB Server logging.

Logging to SST Logs

Logging for mariadb-backup SSTs works the following way.

By default, on the donor node, it logs to mariadb-backup.backup.log. This log file is located in the .

By default, on the joiner node, it logs to mariadb-backup.prepare.log and mariadb-backup.move.log These log files are also located in the datadir.

By default, before a new SST is started, existing mariadb-backup SST log files are compressed and moved to /tmp/sst_log_archive. This behavior can be disabled by setting sst-log-archive=0 in the [sst] in an . Similarly, the archive directory can be changed by setting sst-log-archive-dir:

See for more information.

Logging to Syslog

Redirect the SST logs to the syslog instead, by setting the following in the [sst] in an :

You can also redirect the SST logs to the syslog by setting the following in the [mysqld_safe] in an :

Performing SSTs With IPv6 Addresses

If you are performing SSTs with IPv6 addresses, then the socat utility needs to be passed the pf=ip6 option. This can be done by setting the sockopt option in the [sst] in an :

See for more information.

Manual SST With mariadb-backup

If Galera Cluster's automatic SSTs repeatedly fail, it can be helpful to perform a "manual SST"; see:

See Also

In a MariaDB Galera Cluster, an individual node is considered to have "failed" when it loses communication with the cluster's Primary Component. This can happen for many reasons, including hardware failure, a software crash, loss of network connectivity, or a critical error during a state transfer.

From the perspective of the cluster, a node has failed when the other members can no longer see it. From the perspective of the failed node itself (assuming it hasn't crashed), it has simply lost its connection to the Primary Component and will enter a non-operational state to protect data integrity.

The EVS Protocol

Node failure detection is handled automatically by Galera's group communication system, which uses an Extended Virtual Synchrony (EVS) protocol. This process is controlled by several evs.* parameters in your configuration file.

The cluster determines a node's health based on the last time it received a network packet from that node. The process is as follows:

1. The cluster periodically checks for inactive nodes, controlled by .

2. If a node hasn't sent a packet within the , other nodes begin sending heartbeat beacons to it.

3. If the node remains silent for the duration of , the other nodes will mark it as "suspect."

4. Once all members of the Primary Component agree that a node is suspect, it is declared inactive and .

The Availability vs. Partition Tolerance Trade-off

Within the context of the CAP Theorem (Consistency, Availability, Partition Tolerance), Galera Cluster strongly prioritizes Consistency. This leads to a direct trade-off when configuring the failure detection timeouts, especially on unstable networks like a .

Low Timeouts

Setting low values for evs.suspect_timeout allows the cluster to detect a genuinely failed node very quickly, minimizing downtime. However, on an unstable network, this can lead to "false positives," where a temporarily slow node is incorrectly evicted.

High Timeouts

Setting higher values makes the cluster more tolerant of and slow nodes. However, if a node truly fails, the cluster will remain unavailable for a longer period while it waits for the timeout to expire.

Recovering a Single Failed Node

Recovery from a single node failure is typically automatic. If one node in a cluster with three or more members fails, the rest of the cluster and continues to operate. When the failed node comes back online, it will automatically connect to the cluster and initiate a to synchronize its data. No data is lost in a single node failure.

Recovering the Primary Component After a Full Cluster Outage

A full cluster outage occurs when all nodes shut down or when , leaving no Primary Component. In this scenario, you must manually intervene to safely restart the cluster.

Manual Bootstrap (Using grastate.dat)

This is the traditional recovery method. You must manually identify the node with the most recent data and force it to become the first node in a new cluster.

1. Stop all nodes in the cluster.

2. Identify the most advanced node by checking the seqno value in the in each node's data directory. The node with the highest seqno is the correct one to start from.

3. the new Primary Component by starting the MariaDB service on that single advanced node using a special command (e.g., galera_new_cluster).

Automatic Recovery with pc.recovery

Modern versions of Galera Cluster enable the pc.recovery parameter by default. This feature attempts to automate the recovery of the Primary Component.

When pc.recovery is enabled, nodes that were part of the last known Primary Component will save the state of that component to a file on disk called gvwstate.dat. If the entire cluster goes down, it can automatically recover its state once all the nodes from that last saved component achieve connectivity with each other.

Understanding the gvwstate.dat file

The gvwstate.dat file is created in the data directory of a node when it is part of a Primary Component and is deleted upon graceful shutdown. It contains the node's own UUID and its view of the other members of the component. An example:

• my_uuid: The UUID of the node that owns this file.

• view_id: An identifier for the specific cluster view.

• member: The UUIDs of all nodes that were part of this saved Primary Component.

Advanced Procedure: Modifying the Saved State

In the rare case that you need to force a specific set of nodes to form a new Primary Component, you can manually edit the gawtate.dat file on each of those nodes. By ensuring that each node's file lists itself and all other desired members in the member fields, you can force them to recognize each other and form a new component when you start them.

Failures During State Transfers

A node failure can also occur if a is interrupted. This will cause the receiving node (the "joiner") to abort its startup process. To recover, simply restart the MariaDB service on the failed joiner node.

_{This page is licensed: CC BY-SA / Gnu FDL}