URLs in Galera take a particular format:

Schema

• gcomm⁣- This is the option to use for a working implementation.

• dummy⁣- Used for running tests and profiling, does not do any actual replication, and all following parameters are ignored.

Cluster address

• The cluster address shouldn't be empty like gcomm://. This should never be hardcoded into any configuration files.

• To connect the node to an existing cluster, the cluster address should contain the address of any member of the cluster you want to join.

• The cluster address can also contain a comma-separated list of multiple members of the cluster. It is good practice to list all possible members of the cluster, for example. ⁣gcomm:<node1 name or ip>,<node2 name or ip2>,<node3 name or ip> Alternately, if multicast is used, put the multicast address instead of the list of nodes. Each member address or multicast address can specify <node name or ip>:<port>

Option list

• The variable is used to set a . These parameters can also be provided (and overridden) as part of the URL. Unlike options provided in a configuration file, they will not endure and need to be resubmitted with each connection.

A useful option to set is pc.wait_prim=noto ensure the server will start running even if it can't determine a primary node. This is useful if all members go down at the same time.

Port

By default, gcomm listens on all interfaces. The port is either provided in the cluster address or will default to 4567 if not set.

_{This page is licensed: CC BY-SA / Gnu FDL}

While a client application can connect directly to any node in a MariaDB Galera Cluster, this is not a practical approach for a production environment. A direct connection creates a single point of failure and does not allow the application to take advantage of the cluster's high availability and read-scaling capabilities.

A load balancer or database proxy is an essential component that sits between your application and the cluster. Its primary responsibilities are:

• Provide a Single Endpoint: Your application connects to the load balancer's virtual IP address, not to the individual database nodes.

• Health Checks: The load balancer constantly monitors the health of each cluster node (e.g., is it Synced? is it up or down?).

• Traffic Routing: It intelligently distributes incoming client connections and queries among the healthy nodes in the cluster.

• Automatic Failover: If a node fails, the load balancer automatically stops sending traffic to it, providing seamless failover for your application.

Recommended Load Balancer: MariaDB MaxScale

For MariaDB Galera Cluster, the recommended load balancer is MariaDB MaxScale. Unlike a generic TCP proxy, MaxScale is a database-aware proxy that understands the Galera Cluster protocol. This allows it to make intelligent routing decisions based on the real-time state of the cluster nodes.

Common Routing Strategies

A database-aware proxy like MaxScale can be configured to use several different routing strategies.

Read-Write Splitting (Recommended)

This is the most common and highly recommended strategy for general-purpose workloads.

• How it Works: The load balancer is configured to send all write operations (INSERT, UPDATE, DELETE) to a single, designated primary node. All read operations (SELECT) are then distributed across the remaining available nodes.

• Advantages:

Read Connection Load Balancing

In this simpler strategy, the load balancer distributes all connections evenly across all available nodes.

• How it Works: Each new connection is sent to the next available node in a round-robin fashion.

• Disadvantages: This approach can easily lead to transaction conflicts if your application sends writes to multiple nodes simultaneously. It is generally only suitable for applications that are almost exclusively read-only.

Other Load Balancing Solutions

While MariaDB MaxScale is the recommended solution, other proxies and load balancers can also be used with Galera Cluster, including:

• ProxySQL: Another popular open-source, database-aware proxy.

• HAProxy: A very common and reliable TCP load balancer. When used with Galera, HAProxy is typically configured with a simple TCP health check or a custom script to determine node availability.

• Cloud Load Balancers: Cloud providers like AWS (ELB/NLB), Google Cloud, and Azure offer native load balancing services that can be used to distribute traffic across a Galera Cluster.

Hybrid replication leverages standard, asynchronous MariaDB Replication to copy data from a synchronous MariaDB Galera Cluster to an external server or another cluster. This configuration establishes a one-way data flow, where the entire Galera Cluster serves as the source (primary) for one or more asynchronous replicas. This advanced setup combines the strengths of both replication methods: synchronous replication ensures high availability within the primary site, while asynchronous replication caters to specific use cases, allowing for flexible data distribution.

Common Use Cases

Implementing a hybrid replication setup is a powerful technique for solving several common business needs:

Key Challenges and Considerations

Before implementing a hybrid setup, it is critical to understand the technical challenges:

_{This page is licensed: CC BY-SA / Gnu FDL}

Quickstart Guide: About Galera Replication

Galera Replication is a core technology enabling MariaDB Galera Cluster to provide a highly available and scalable database solution. It is characterized by its virtually synchronous replication, ensuring strong data consistency across all cluster nodes.

To facilitate development and QA, we have created some test repositories for the Galera wsrep provider.

Galera Test Repositories for YUM

The instructions on this page were used to create the galera package on the Ubuntu and Debian Linux distributions. This package contains the wsrep provider for .

The version of the wsrep provider is 25.3.5. We also provide 25.2.9 for those that need or want it. Prior to that, the wsrep version was 23.2.7.

1. Install prerequisites:

1. Clone from and checkout mariadb-3.x banch:

The recommended strategy for creating a full, consistent backup of a MariaDB Galera Cluster is to perform the backup on a single . Because all nodes in a contain the same data, a complete backup from one node represents a snapshot of the entire cluster at a specific point in time.

The preferred tool for this is . mariadb-backup creates a "hot" backup without blocking the node from serving traffic for an extended period.

The Challenge of Consistency in a Live Cluster

While taking a backup, the donor node is still receiving and applying transactions from the rest of the cluster. If the backup process is long, it's possible for the data at the end of the backup to be newer than the data at the

Certification-based replication uses and transaction ordering techniques to achieve synchronous replication.

Transactions execute optimistically in a , or replica, and then at commit time, they run a coordinated certification process to enforce global consistency. It achieves global coordination with the help of a broadcast service that establishes a global total order among concurrent transactions.

Requirements for Certification-Based Replication

It is not possible to implement certification-based replication for all database systems. It requires certain features of the database in order to work:

Overview

Name for the cluster.

Details

This system variable specifies the logical name of the cluster. Every Cluster Node that connects to each other must

Overview

State snapshot transfer method.

DETAILS

Auto-Eviction enhances cluster stability by automatically removing non-responsive or "unhealthy" nodes in MariaDB Galera Cluster. This prevents a single problematic node from degrading the entire cluster's performance. In a Galera Cluster, each node monitors the network response times of other nodes. If a node becomes unresponsive due to reasons like memory swapping, network congestion, or a hung process, it can delay and potentially disrupt cluster operations. Auto-Eviction provides a deterministic method to isolate these misbehaving nodes effectively.

Auto-Eviction Process

The Auto-Eviction process is based on a consensus mechanism among the healthy cluster members.

1. Monitoring and Delay List: Each node in the cluster monitors the response times from all its peers. If a given node fails to respond within the expected timeframes, the other nodes will add an entry for it to their internal "delayed list."

2. Eviction Trigger: If a of the cluster nodes independently add the same peer to their delayed lists, it triggers the Auto-Eviction protocol.

3. Eviction: The cluster evicts the unresponsive node, removing it from the . The evicted node will enter a non-primary state and must be restarted to

The sensitivity of this process is determined by the evs.auto_evict parameter.

Configuration

Auto-Eviction is configured by passing the evs.auto_evict within the wsrep_provider_options in your MariaDB configuration file (my.cnf).

The value of evs.auto_evict determines the threshold for eviction. It defines how many times a peer can be placed on the delayed list before the node votes to evict it.

In the above example example, if a node registers that a peer has been delayed 5 times, it will vote to have that peer evicted from the cluster.

To disable Auto-Eviction, you can set the value to 0:

Even when disabled, the node will continue to monitor response times and log information about delayed peers; it just won't vote to evict them.

Related Parameters for Failure Detection

The Auto-Eviction feature is directly related to the that control how the cluster detects unresponsive nodes in the first place. These parameters define what it means for a node to be "delayed."

Tuning these values in conjunction with evs.auto_evict allows you to define how aggressively the cluster will fence off struggling nodes.

_{This page is licensed: CC BY-SA / Gnu FDL}

By default, Galera Cluster replicates data between each node without encrypting it. This is generally acceptable when the cluster nodes runs on the same host or in networks where security is guaranteed through other means. However, in cases where the cluster nodes exist on separate networks or they are in a high-risk network, the lack of encryption does introduce security concerns as a malicious actor could potentially eavesdrop on the traffic or get a complete copy of the data by triggering an SST.

To mitigate this concern, Galera Cluster allows you to encrypt data in transit as it is replicated between each cluster node using the Transport Layer Security (TLS) protocol. TLS was formerly known as Secure Socket Layer (SSL), but strictly speaking the SSL protocol is a predecessor to TLS and, that version of the protocol is now considered insecure. The documentation still uses the term SSL often and for compatibility reasons TLS-related server system and status variables still use the prefix ssl_, but internally, MariaDB only supports its secure successors.

In order to secure connections between the cluster nodes, you need to ensure that all servers were compiled with TLS support. See to determine how to check whether a server was compiled with TLS support.

For each cluster node, you also need a certificate, private key, and the Certificate Authority (CA) chain to verify the certificate. If you want to use self-signed certificates that are created with OpenSSL, then see for information on how to create those.

Securing Galera Cluster Replication Traffic

In order to enable TLS for Galera Cluster's replication traffic, there are a number of that you need to set, such as:

• You need to set the path to the server's certificate by setting the wsrep_provider_option.

• You need to set the path to the server's private key by setting the wsrep_provider_option.

• You need to set the path to the certificate authority (CA) chain that can verify the server's certificate by setting the wsrep_provider_option.

• If you want to restrict the server to certain ciphers, then you also need to set the wsrep_provider_option.

It is also a good idea to set MariaDB Server's regular TLS-related system variables, so that TLS will be enabled for regular client connections as well. See for information on how to do that.

For example, to set these variables for the server, add the system variables to a relevant server in an :

And then restart the server to make the changes persistent.

By setting both MariaDB Server's TLS-related system variables and Galera Cluster's TLS-related wsrep_provider_options, the server can secure both external client connections and Galera Cluster's replication traffic.

Securing State Snapshot Transfers

The method that you would use to enable TLS for would depend on the value of .

mariadb-backup

See for more information.

xtrabackup-v2

See : TLS for more information.

mysqldump

This SST method simply uses the (previously mysqldump) utility, so TLS would be enabled by following the guide at

rsync

This SST method supports encryption in transit via . See for more information.

_{This page is licensed: CC BY-SA / Gnu FDL}

Overview

Print warning about certificate expiration if the X509 certificate used for wsrep connections is about to expire in hours given as an argument. If the value is 0, warnings are not printed.

Usage

The wsrep_certificate_expiration_hours_warning system variable can be set in a configuration file:

The global value of the wsrep_certificate_expiration_hours_warning system variable can also be set dynamically at runtime by executing :

When the wsrep_certificate_expiration_hours_warning system variable is set dynamically at runtime, its value will be reset the next time the server restarts. To make the value persist on restart, set it in a configuration file too.

Details

The wsrep_certificate_expiration_hours_warning system variable can be used to configure certificate expiration warnings for MariaDB Enterprise Cluster, powered by Galera:

• When the wsrep_certificate_expiration_hours_warning system variable is set to 0, certificate expiration warnings are not printed to the .

• When the wsrep_certificate_expiration_hours_warning system variable is set to a value N, which is greater than 0, certificate expiration warnings are printed to the MariaDB Error Log when the node's certificate expires in N hours or less.

Parameters

MariaDB Galera Cluster provides a powerful automation feature through the wsrep_notify_cmd . When this variable is configured, the MariaDB server will automatically execute a specified command or script in response to changes in the cluster's membership or the local node's state.

This is extremely useful for integrating the cluster with external systems:

MariaDB Galera Cluster supports a weighted quorum, where each node can be assigned a weight in the range of 0 to 255, with which it will participate in quorum calculations. This provides fine-grained control over which nodes are most critical for forming a , especially in complex or .

By default, every has a weight of 1. You can customize a node's weight during runtime by setting the pc.weight

MariaDB Galera Cluster is flexible and can be deployed in several different topologies to meet various business needs, from within a single data center to geographically distributed disaster recovery. The primary deployment patterns are designed for Local Area Networks (LAN) and Wide Area Networks (WAN).

Standard LAN Cluster (Single Data Center)

This is the most common deployment pattern for achieving high availability and read scaling within a single data center.

When a network failure or a crash affects over half of your cluster nodes, the cluster might lose its . In such cases, the remaining nodes may return an Unknown command error for many queries. This behavior is a safeguard to prevent data inconsistency.

You can confirm this by checking the wsrep_cluster_status on all nodes:

If none of your nodes return a value of Primary

and can be used together. However, there are some things that have to be taken into account.

Tutorials

If you want to use and together, then the following tutorials may be useful:

wsrep_sst_common Variables

The wsrep_sst_common script provides shared functionality used by various State Snapshot Transfer (SST) methods in Galera Cluster. It centralizes the handling of common configurations such as authentication credentials, SSL/TLS encryption parameters, and other security-related settings. This ensures consistent and secure communication between cluster nodes during the SST process.

The wsrep_sst_common script parses the following options:

There are binary installation packages available for RPM and Debian-based distributions, which will pull in all required Galera dependencies.

If these are not available, you will need to build Galera from source.

The wsrep API for Galera Cluster is included by default. Follow the usual instructions

Preparation

make cannot manage dependencies for the build process, so the following packages need to be installed first:

RPM-based:

Debian-based:

If running on an alternative system, or the commands are available, the following packages are required. You will need to check the repositories for the correct package names on your distribution—these may differ between distributions or require additional packages:

MariaDB Database Server with wsrep API

• Git, CMake (on Fedora, both cmake and cmake-fedora are required), GCC and GCC-C++, Automake, Autoconf, and Bison, as well as development releases of libaio and ncurses.

Building

You can use Git to download the source code, as MariaDB source code is available through GitHub. 1. Clone the repository:

1. Check out the branch (e.g., 10.5-galera or 11.1-galera), for example:

Building the Database Server

The standard and Galera Cluster database servers are the same, except that for Galera Cluster, the wsrep API patch is included. Enable the patch with the CMake configuration options. WITH_WSREP and WITH_INNODB_DISALLOW_WRITES. To build the database server, run the following commands:

There are also some build scripts in the *BUILD/* directory, which may be more convenient to use. For example, the following pre-configures the build options discussed above:

There are several others as well, so you can select the most convenient.

<>

Besides the server with the Galera support, you will also need a Galera provider.

Preparation

make cannot manage dependencies itself, so the following packages need to be installed first:

If running on an alternative system, or the commands are available, the following packages are required. You will need to check the repositories for the correct package names on your distribution - these may differ between distributions, or require additional packages:

Galera Replication Plugin

• SCons, as well as development releases of Boost (libboost_program_options, libboost_headers1), Check and OpenSSL.

Building

Run:

After this, the source files for the Galera provider will be in the galera directory.

Building the Galera Provider

The Galera Replication Plugin both implements the wsrep API and operates as the database server's wsrep Provider. To build, cd into the galera/ directory and do:

The path to libgalera_smm.so needs to be defined in the my.cnf configuration file.

Building Galera Replication Plugin from source on FreeBSD runs into issues due to Linux dependencies. To overcome these, either install the binary package: pkg install galera, or use the ports build available at /usr/ports/databases/galera.

Configuration

After building, a number of other steps are necessary:

1. Create the database server user and group:

1. Install the database (the path may be different if you specified CMAKE_INSTALL_PREFIX):

1. If you want to install the database in a location other than /usr/local/mysql/data , use the --basedir or --datadir options.

2. Change the user and group permissions for the base directory.

1. Create a system unit for the database server.

1. Galera Cluster can now be started using the service command and is set to start at boot.

_{This page is licensed: CC BY-SA / Gnu FDL}

Galera Cluster ships with the MariaDB Server. Upgrading a Galera Cluster node is very similar to upgrading a server from MariaDB 10.4 to MariaDB 10.5. For more information on that process as well as incompatibilities between versions, see the .

Performing a Rolling Upgrade

The following steps can be used to perform a rolling upgrade from MariaDB 10.4 to MariaDB 10.5 when using Galera Cluster. In a rolling upgrade, each node is upgraded individually, so the cluster is always operational. There is no downtime from the application's perspective.

First, before you get started:

1. First, take a look at to see what has changed between the major versions.

2. Check whether any system variables or options have been changed or removed. Make sure that your server's configuration is compatible with the new MariaDB version before upgrading.

3. Check whether replication has changed in the new MariaDB version in any way that could cause issues while the cluster contains upgraded and non-upgraded nodes.

4. Check whether any new features have been added to the new MariaDB version. If a new feature in the new MariaDB version cannot be replicated to the old MariaDB version, then do not use that feature until all cluster nodes have been upgrades to the new MariaDB version.

5. Next, make sure that the Galera version numbers are compatible.

6. If you are upgrading from the most recent release to , then the versions will be compatible.

7. See ?: s for information on which MariaDB releases uses which Galera wsrep provider versions.

8. You want to have a large enough gcache to avoid a during the rolling upgrade. The gcache size can be configured by setting For example:wsrep_provider_options="gcache.size=2G"

Before you upgrade, it would be best to take a backup of your database. This is always a good idea to do before an upgrade. We would recommend .

Then, for each node, perform the following steps:

When this process is done for one node, move onto the next node.

_{This page is licensed: CC BY-SA / Gnu FDL}

Galera Cluster ships with the MariaDB Server. Upgrading a Galera Cluster node is very similar to upgrading a server from to . For more information on that process as well as incompatibilities between versions, see the .

Performing a Rolling Upgrade

The following steps can be used to perform a rolling upgrade from to when using Galera Cluster. In a rolling upgrade, each node is upgraded individually, so the cluster is always operational. There is no downtime from the application's perspective.

First, before you get started:

1. First, take a look at to see what has changed between the major versions.

2. Check whether any system variables or options have been changed or removed. Make sure that your server's configuration is compatible with the new MariaDB version before upgrading.

3. Check whether replication has changed in the new MariaDB version in any way that could cause issues while the cluster contains upgraded and non-upgraded nodes.

4. Check whether any new features have been added to the new MariaDB version. If a new feature in the new MariaDB version cannot be replicated to the old MariaDB version, then do not use that feature until all cluster nodes have been upgrades to the new MariaDB version.

5. Next, make sure that the Galera version numbers are compatible.

6. If you are upgrading from the most recent release to , then the versions will be compatible.

7. See ?: for information on which MariaDB releases uses which Galera wsrep provider versions.

8. You want to have a large enough gcache to avoid a during the rolling upgrade. The gcache size can be configured by setting for example:wsrep_provider_options="gcache.size=2G"

Before you upgrade, it would be best to take a backup of your database. This is always a good idea to do before an upgrade. We would recommend .

Then, for each node, perform the following steps:

When this process is done for one node, move onto the next node.

_{This page is licensed: CC BY-SA / Gnu FDL}

Flow Control is a key feature in MariaDB Galera Cluster that ensures nodes remain synchronized. In synchronous replication, no node should lag significantly in processing transactions.

Flow Control manages this by aligning all nodes' replication processes:

Preventing Memory Overflow

Without Flow Control, a slow node's replication queue can grow unchecked, consuming all server memory and potentially crashing the MariaDB process due to an Out-Of-Memory (OOM) error.

Maintaining Synchronization

It maintains synchronization across the cluster, ensuring all nodes have nearly identical database states at all times.

Flow Control Sequence

The Flow Control process is an automatic feedback loop triggered by the state of a node's replication queue.

1. Queue Growth: A node (the "slow node") begins receiving from its peers faster than it can apply them. This causes its local receive queue, measured by the wsrep_local_recv_queue , to grow.

2. Upper Limit Trigger: When the receive queue size exceeds the configured upper limit, defined by the gcs.fc_limit , the slow node triggers Flow Control.

3. Pause Message: The node broadcasts a "Flow Control PAUSE" message to all other nodes in the cluster.

Monitoring Flow Control

As an administrator, observing Flow Control is a key indicator of a performance bottleneck in your cluster. You can monitor it using the following global :

Troubleshooting Flow Control Issues

If you observe frequent Flow Control pauses, it is essential to identify and address the underlying cause.

Key Configuration Parameters

These my.cnf control the sensitivity of Flow Control:

Common Causes and Solutions

_{This page is licensed: CC BY-SA / Gnu FDL}

Streaming Replication optimizes replication of large or long-running transactions in MariaDB Galera Cluster. Typically, a node executes a transaction fully and replicates the complete write-set to other nodes at time. Although efficient for most workloads, this approach can be challenging for very large or lengthy transactions.

With Streaming Replication, the initiating node divides the transaction into smaller fragments. These fragments are certified and replicated to other nodes while the transaction is ongoing. Once a fragment is certified and applied to the replicas, it becomes immune to abortion by conflicting transactions, thus improving the chances of the entire transaction succeeding. This method also supports processing of transaction write-sets over two Gigabytes.

When to Use Streaming Replication

In most cases, the standard replication method is sufficient. Streaming Replication is a specialized tool for specific scenarios. The best practice is to enable it only at the session level for the specific transactions that require it.

Large Data Transactions

This is the primary use case. When performing a massive , , or , normal replication requires the originating node to hold the entire transaction locally and then send a very large write-set at commit time. This can cause two problems:

1. A significant replication lag, as the entire cluster waits for the large write-set to be transferred and applied.

2. The replica nodes, while busy applying the large transaction, cannot commit other transactions, which can trigger and throttle the entire cluster.

With Streaming Replication, the node replicates the data in fragments throughout the transaction's lifetime. This spreads the network load and allows replica nodes to apply other concurrent transactions between fragments, minimizing the impact on the overall

Long-Running Transactions

A transaction that remains open for a long time has a higher chance of that commits first. When this happens, the long-running transaction is aborted.

Streaming Replication mitigates this by committing the transaction in fragments. Once a fragment is , it is "locked in" and cannot be aborted by a new conflicting transaction.

High-Contention ("Hot") Records

For applications that frequently update the same row (e.g., a counter, a job queue, or a locking scheme), Streaming Replication can be used to force a critical update to replicate immediately. This effectively locks the , preventing other transactions from modifying it and increasing the chance that the critical transaction will commit successfully.

How to Enable and Use Streaming Replication

Streaming Replication should be enabled at the session level just for the transactions that need it. This is controlled by two session variables:

• defines what a "unit" of replication is.

• defines how many units make up a fragment.

To enable streaming, you set both variables:

In the above example, the node will create, certify, and replicate a fragment after every 10 SQL statements within the transaction.

The available fragment units for wsrep_trx_fragment_unit are:

To disable Streaming Replication, you can set wsrep_trx_fragment_size back to 0.

Managing a "Hot Record"

Consider an application that manages a work order queue. To prevent two users from getting the same queue position, you can use Streaming Replication for the single critical update.

1. Begin the transaction:

2. After reading necessary data, enable Streaming Replication for just the next statement:

3. Perform the critical update. This statement will be immediately fragmented and replicated:

4. Immediately disable Streaming Replication for the rest of the transaction:

This ensures the queue_position update is replicated and certified across the cluster before the rest of the transaction proceeds, preventing .

Limitations and Performance Considerations

Before using Streaming Replication, consider the following limitations:

Performance Overhead

When Streaming Replication is enabled, Galera records all write-sets to a log table () on every node to ensure persistence in case of a crash. This adds write overhead and can impact performance, which is why it should only be used when necessary.

Cost of Rollbacks

If a streaming transaction needs to be rolled back after some fragments have already been applied, the rollback operation consumes system resources on all nodes as they undo the previously applied fragments. Frequent rollbacks of streaming transactions can become a performance problem.

For these reasons, it is always a good application design policy to use shorter, smaller transactions whenever possible.

_{This page is licensed: CC BY-SA / Gnu FDL}

MariaDB Galera Cluster provides a synchronous replication system that uses an approach often called eager replication. In this model, nodes in a cluster synchronize with all other nodes by applying replicated updates as a single transaction. This means that when a transaction COMMITs, all nodes in the cluster have the same value. This process is accomplished using write-set replication through a group communication framework.

Core Architectural Components

The internal architecture of MariaDB Galera Cluster revolves around four primary components:

MariaDB Server (DBMS)

The foundation of the cluster is the standard MariaDB Server, typically using the InnoDB storage engine. This component serves clients that connect to it and executes queries as a normal database server would.

The wsrep API

The wsrep API is a generic replication plugin interface for databases. It defines a set of application callbacks and replication plugin calls that connect the MariaDB Server to the replication provider. It consists of two main elements:

• wsrep Hooks: These hooks integrate with the database server engine to enable write-set replication.

• dlopen(): This function is used to make the (the Galera Plugin) available to the wsrep hooks.

In this model, the wsrep API considers the database to have a "state." When clients modify the database content, its state changes. The API represents these changes as a series of atomic transactions. In a healthy cluster, all nodes always have the same because they synchronize by replicating and applying these state changes in the same serial order.

From a technical perspective, the process flow is as follows:

1. A state change (transaction) occurs on one node in the cluster.

2. Within the MariaDB Server, the wsrep hooks translate these changes into a write-set.

3. The dlopen() function makes the wsrep provider's functions available to the hooks.

Galera Replication Plugin

The Galera Replication Plugin implements the wsrep API and acts as the wsrep Provider. It handles the core replication service functionality. The plugin itself consists of the following components:

Group Communication (GComm) Framework

The Galera Replication Plugin uses a Group Communication (GComm) framework for its messaging layer. The GComm system implements a virtual synchrony Quality of Service (QoS), which unifies data delivery and cluster into a clear, formal model.

While virtual synchrony guarantees data consistency, it does not guarantee the temporal synchrony needed for smooth multi-primary operations. To address this, Galera Cluster implements its own runtime-configurable , which keeps nodes synchronized to within a fraction of a second.

The GComm framework also provides a total ordering of messages from multiple sources, which it uses to generate Global Transaction IDs in a multi-primary cluster.

Fundamental Concepts

Global Transaction ID (GTID)

To keep the database state identical across all nodes, the wsrep API uses a Global Transaction ID (GTID). This allows the cluster to uniquely identify every state change and to know the current state of any node in relation to others. An example GTID looks like this:

45eec521-2f34-11e0-0800-2a36050b826b:94530586304

It consists of two components:

• State UUID: A unique identifier for the database state and its sequence of changes.

• Ordinal Sequence Number (seqno): A 64-bit signed integer that denotes the position of the transaction in the sequence.

Read and Write Scaling

A direct benefit of Galera's multi-master architecture is the ability to scale both read and write operations.

• Write Scaling: Because every node in the cluster can accept write operations, you can distribute your application's write traffic across multiple nodes. This can increase write throughput, though it's important to remember that all writes must still be replicated and certified on all nodes, which can introduce .

• Read Scaling: This is the most significant performance advantage. Since all nodes are kept synchronized, they all contain the same data. This allows you to distribute read queries across all nodes in the cluster, providing excellent horizontal scaling for read-heavy applications. This architecture is ideal for use with a load balancer (like MariaDB MaxScale) that can perform read-write splitting.

This scaling is typically managed by a which distributes traffic intelligently across the cluster.

_{This page is licensed: CC BY-SA / Gnu FDL}

In a MariaDB Galera Cluster, an individual node is considered to have "failed" when it loses communication with the cluster's Primary Component. This can happen for many reasons, including hardware failure, a software crash, loss of network connectivity, or a critical error during a state transfer.

From the perspective of the cluster, a node has failed when the other members can no longer see it. From the perspective of the failed node itself (assuming it hasn't crashed), it has simply lost its connection to the Primary Component and will enter a non-operational state to protect data integrity.

The EVS Protocol

Node failure detection is handled automatically by Galera's group communication system, which uses an Extended Virtual Synchrony (EVS) protocol. This process is controlled by several evs.* parameters in your configuration file.

The cluster determines a node's health based on the last time it received a network packet from that node. The process is as follows:

1. The cluster periodically checks for inactive nodes, controlled by .

2. If a node hasn't sent a packet within the , other nodes begin sending heartbeat beacons to it.

3. If the node remains silent for the duration of , the other nodes will mark it as "suspect."

4. Once all members of the Primary Component agree that a node is suspect, it is declared inactive and .

The Availability vs. Partition Tolerance Trade-off

Within the context of the CAP Theorem (Consistency, Availability, Partition Tolerance), Galera Cluster strongly prioritizes Consistency. This leads to a direct trade-off when configuring the failure detection timeouts, especially on unstable networks like a .

Low Timeouts

Setting low values for evs.suspect_timeout allows the cluster to detect a genuinely failed node very quickly, minimizing downtime. However, on an unstable network, this can lead to "false positives," where a temporarily slow node is incorrectly evicted.

High Timeouts

Setting higher values makes the cluster more tolerant of and slow nodes. However, if a node truly fails, the cluster will remain unavailable for a longer period while it waits for the timeout to expire.

Recovering a Single Failed Node

Recovery from a single node failure is typically automatic. If one node in a cluster with three or more members fails, the rest of the cluster and continues to operate. When the failed node comes back online, it will automatically connect to the cluster and initiate a to synchronize its data. No data is lost in a single node failure.

Recovering the Primary Component After a Full Cluster Outage

A full cluster outage occurs when all nodes shut down or when , leaving no Primary Component. In this scenario, you must manually intervene to safely restart the cluster.

Manual Bootstrap (Using grastate.dat)

This is the traditional recovery method. You must manually identify the node with the most recent data and force it to become the first node in a new cluster.

1. Stop all nodes in the cluster.

2. Identify the most advanced node by checking the seqno value in the in each node's data directory. The node with the highest seqno is the correct one to start from.

3. the new Primary Component by starting the MariaDB service on that single advanced node using a special command (e.g., galera_new_cluster).

Automatic Recovery with pc.recovery

Modern versions of Galera Cluster enable the pc.recovery parameter by default. This feature attempts to automate the recovery of the Primary Component.

When pc.recovery is enabled, nodes that were part of the last known Primary Component will save the state of that component to a file on disk called gvwstate.dat. If the entire cluster goes down, it can automatically recover its state once all the nodes from that last saved component achieve connectivity with each other.

Understanding the gvwstate.dat file

The gvwstate.dat file is created in the data directory of a node when it is part of a Primary Component and is deleted upon graceful shutdown. It contains the node's own UUID and its view of the other members of the component. An example:

• my_uuid: The UUID of the node that owns this file.

• view_id: An identifier for the specific cluster view.

• member: The UUIDs of all nodes that were part of this saved Primary Component.

Advanced Procedure: Modifying the Saved State

In the rare case that you need to force a specific set of nodes to form a new Primary Component, you can manually edit the gawtate.dat file on each of those nodes. By ensuring that each node's file lists itself and all other desired members in the member fields, you can force them to recognize each other and form a new component when you start them.

Failures During State Transfers

A node failure can also occur if a is interrupted. This will cause the receiving node (the "joiner") to abort its startup process. To recover, simply restart the MariaDB service on the failed joiner node.

_{This page is licensed: CC BY-SA / Gnu FDL}

Sometimes it can be helpful to perform a "manual SST" when Galera's normal SSTs fail. This can be especially useful when the cluster's is very large, since a normal SST can take a long time to fail in that case.

A manual SST essentially consists of taking a backup of the donor, loading the backup on the joiner, and then manually editing the cluster state on the joiner node. This page will show how to perform this process with .

Process

For example, a very common version number is "2.1".

MariaDB's are very useful when used with , which is primarily what that feature was developed for. Galera Cluster, on the other hand, was developed by Codership for all MySQL and MariaDB variants, and the initial development of the technology pre-dated MariaDB's implementation. As a side effect, MariaDB Galera Cluster (at least until ) only partially supports MariaDB's implementation.

GTID Support for Write Sets Replicated by Galera Cluster

Galera Cluster has its own certification-based replication method that is substantially different from . However, it would still be beneficial if MariaDB Galera Cluster was able to associate a Galera Cluster write set with a that is globally unique but that is also consistent for that write set on each cluster node.

Wsrep GTID Mode

MariaDB supports .

MariaDB has a feature called wsrep GTID mode. When this mode is enabled, MariaDB uses some tricks to try to associate each Galera Cluster write set with a that is globally unique, but that is also consistent for that write set on each cluster node. These tricks work in some cases, but can still become inconsistent among cluster nodes.

Enabling Wsrep GTID Mode

Several things need to be configured for wsrep GTID mode to work, such as

• needs to be set on all nodes in the cluster.

• needs to be set to the same value on all nodes in a given cluster, so that each cluster node uses the same domain when assigning for Galera Cluster's write sets. When replicating between two clusters, each cluster should have this set to a different value, so that each cluster uses different domains when assigning for their write sets.

• needs to be enabled on all nodes in the cluster. See .

And as an extra safety measure:

• should be set to a different value on all nodes in a given cluster, and each of these values should be different than the configured value. This is to prevent a node from using the same domain used for Galera Cluster's write sets when assigning for non-Galera transactions, such as DDL executed with set or DML executed with set.

If you want to avoid writes accidentally local GTIDS, you can avoid it with by setting this:

In this case you get an error:

You can overwrite it temporarily with:

For information on setting , see .

GTIDs for Transactions Applied by Slave Thread

If a Galera Cluster node is also a , then that node's will be applying transactions that it replicates from its replication master. If the node has set, then each transaction that the applies will also generate a Galera Cluster write set that is replicated to the rest of the nodes in the cluster.

The node acting as slave includes the transaction's original Gtid_Log_Event in the replicated write set, so all nodes should associate the write set with its original GTID. See .

_{This page is licensed: CC BY-SA / Gnu FDL}

Overview

Defines the path to the SSL certificate.

The wsrep_provider_options system variable applies to MariaDB Enterprise Cluster, powered by Galera and to Galera Cluster available with MariaDB Community Server. This page relates specifically to the socket.ssl_cert wsrep_provider_options.

Details

The node uses the certificate as a self-signed public key in encrypting replication traffic over SSL. You can use either an absolute path or one relative to the working directory. The file must use PEM format.

Examples

Display Current Value

wsrep_provider_options define optional settings the node passes to the wsrep provider.

To display current wsrep_provider_options values:

The expected output will display the option and the value. Options with no default value, for example SSL options, will not be displayed in the output.

Set in Configuration File

When changing a setting for a wsrep_provider_options in the config file, you must list EVERY option that is to have a value other than the default value. Options that are not explicitly listed are reset to the default value.

Options are set in the my.cnf configuration file. Use the ; delimiter to set multiple options.

The configuration file must be updated on each node. A restart to each node is needed for changes to take effect.

Use a quoted string that includes every option where you want to override the default value. Options that are not in the list will reset to their default value.

To set the option in the configuration file:

Set Dynamically

The socket.ssl_cert option cannot be set dynamically. It can only be set in the configuration file.

Trying to change a non-dynamic option with SET results in an error:

Overview

X509 cert in PEM format (implies --ssl).

Details

Parameters

Overview

CA directory (check OpenSSL docs, implies --ssl).

Details

Parameters

Quickstart Guide: MariaDB Galera Cluster

MariaDB Galera Cluster provides a multi-primary (active-active) cluster solution for MariaDB, enabling high availability, read/write scalability, and true synchronous replication. This means any node can handle read and write operations, with changes instantly replicated to all other nodes, ensuring no replica lag and no lost transactions. It's exclusively available on Linux.

Quickstart Guide: MariaDB Galera Cluster Usage

This guide provides essential information for effectively using and interacting with a running MariaDB Galera Cluster. It covers connection methods, operational considerations, monitoring, and best practices for applications.

Summary

In MariaDB Cluster, transactions are replicated using the wsrep API, synchronously ensuring consistency across nodes. Synchronous replication offers high availability and consistency but is complex and potentially slower compared to asynchronous replication. Due to these challenges, asynchronous replication is often preferred for database performance and scalability, as seen in popular systems like MySQL and PostgreSQL, which typically favor asynchronous or semi-synchronous solutions.

In MariaDB Cluster, the server replicates a transaction at commit time by broadcasting the write set associated with the transaction to every node in the cluster. The client connects directly to the DBMS and experiences behavior that is similar to native MariaDB in most cases. The wsrep API (write set replication API) defines the interface between Galera replication and MariaDB.

A number of options need to be set in order for Galera Cluster to work when using MariaDB. These should be set in the .

Mandatory Options

Several options are mandatory, which means that they must be set in order for Galera Cluster to be enabled or to work properly with MariaDB. The mandatory options are:

• — Path to the Galera library

MariaDB starting with

Since , the patch has been merged into MariaDB Server. Therefore, in and above, the functionality of MariaDB Galera Cluster can be obtained by installing the standard MariaDB Server packages and the Galera wsrep provider library package.

Beginning in , ships with the MariaDB Server. Upgrading a Galera Cluster node is very similar to upgrading a server from to . For more information on that process as well as incompatibilities between versions, see the .

Performing a Rolling Upgrade

Incremental State Transfer (IST)

An Incremental State Transfer (IST) is the fast and efficient process where a joining node receives only the missing it needs to catch up with the cluster, rather than receiving a full copy of the entire database.

Overview

Defines the path to the SSL Certificate Authority (CA) file.

The wsrep_provider_options system variable applies to MariaDB Enterprise Cluster, powered by Galera and to Galera Cluster available with MariaDB Community Server. This page relates specifically to the socket.ssl_ca wsrep_provider_options.

Overview

CA file in PEM format (check OpenSSL docs, implies --ssl).

Details

Overview

Explicitly enables TLS usage by the wsrep provider.

The wsrep_provider_options system variable applies to MariaDB Enterprise Cluster, powered by Galera and to Galera Cluster available with MariaDB Community Server. This page relates specifically to the socket.ssl wsrep_provider_options.

Overview

X509 key in PEM format (implies --ssl).

Details

Galera Cluster ships with the MariaDB Server. Upgrading a Galera Cluster node is very similar to upgrading a server from to . For more information on that process as well as incompatibilities between versions, see the .

Methods

• Stopping all nodes, upgrading all nodes, then starting the nodes

• Rolling upgrade with IST (however, see )

• Note that rolling upgrade with SST does not work

Performing a Rolling Upgrade

The following steps can be used to perform a rolling upgrade from to when using Galera Cluster. In a rolling upgrade, each node is upgraded individually, so the cluster is always operational. There is no downtime from the application's perspective.

First, before you get started:

1. First, take a look at to see what has changed between the major versions.

2. Check whether any system variables or options have been changed or removed. Make sure that your server's configuration is compatible with the new MariaDB version before upgrading.

3. Check whether replication has changed in the new MariaDB version in any way that could cause issues while the cluster contains upgraded and non-upgraded nodes.

4. Check whether any new features have been added to the new MariaDB version. If a new feature in the new MariaDB version cannot be replicated to the old MariaDB version, then do not use that feature until all cluster nodes have been upgrades to the new MariaDB version.

Before you upgrade, it would be best to take a backup of your database. This is always a good idea to do before an upgrade. We would recommend .

Then, for each node, perform the following steps:

When this process is done for one node, move onto the next node.

_{This page is licensed: CC BY-SA / Gnu FDL}

Overview

Defines the path to the SSL certificate key.

The wsrep_provider_options system variable applies to MariaDB Enterprise Cluster, powered by Galera and to Galera Cluster available with MariaDB Community Server. This page relates specifically to the socket.ssl_key wsrep_provider_options.

Details

The node uses the certificate key, a self-signed private key, in encrypting replication traffic over SSL. You can use either an absolute path or one relative to the working directory. The file must use PEM format.

Examples

Display Current Value

wsrep_provider_options define optional settings the node passes to the wsrep provider.

To display current wsrep_provider_options values:

The expected output will display the option and the value. Options with no default value, for example SSL options, will not be displayed in the output.

Set in Configuration File

When changing a setting for a wsrep_provider_options in the config file, you must list EVERY option that is to have a value other than the default value. Options that are not explicitly listed are reset to the default value.

Options are set in the my.cnf configuration file. Use the ; delimiter to set multiple options.

The configuration file must be updated on each node. A restart to each node is needed for changes to take effect.

Use a quoted string that includes every option where you want to override the default value. Options that are not in the list will reset to their default value.

To set the option in the configuration file:

Set Dynamically

The socket.ssl_key option cannot be set dynamically. It can only be set in the configuration file.

Trying to change a non-dynamic option with SET results in an error:

From a , you can check the status of throughout the cluster using standard queries. Status variables that relate to write-set replication have the prefix wsrep_, meaning that you can display them all using the following query:

Understanding Quorum and Cluster Integrity

The most fundamental aspect of a healthy cluster is . Quorum is a mechanism that ensures data consistency by requiring a majority of nodes to be online and in communication to form a Primary Component. Only the Primary Component will process transactions. This prevents where a network partition could otherwise lead to data conflicts.

You can check the cluster's integrity and Quorum status using these key variables. For a healthy cluster, the values for these variables must be identical on every node.

wsrep_sst_mariabackup Variables

The wsrep_sst_mariabackup script handles the actual data transfer and processing during an SST. The variables it reads from the [sst] group control aspects of the backup format, compression, transfer mechanism, and logging.

The wsrep_sst_mariadbbackup script parses the following options: