1 of 63

Galera Cluster

MariaDB Galera Cluster

MariaDB Galera Cluster is a virtually synchronous multi-master cluster that runs on Linux only. Its Enterprise version is MariaDB Enterprise Cluster (powered by Galera).

MariaDB Galera Cluster Overview

MariaDB Enterprise Cluster is a solution designed to handle high workloads exceeding the capacity of a single server. It is based on Galera Cluster technology integrated with MariaDB Enterprise Server and includes features like data-at-rest encryption for added security. This multi-primary replication alternative is ideal for maintaining data consistency across multiple servers, providing enhanced reliability and scalability.

Overview

MariaDB Enterprise Cluster, powered by Galera, is available with MariaDB Enterprise Server. MariaDB Galera Cluster is available with MariaDB Community Server.

In order to handle increasing load and especially when that load exceeds what a single server can process, it is best practice to deploy multiple MariaDB Enterprise Servers with a replication solution to maintain data consistency between them. MariaDB Enterprise Cluster is a multi-primary replication solution that serves as an alternative to the single-primary MariaDB Replication.

An Introduction to Database Replication

Database replication is the process of continuously copying data from one database server (a "node") to another, creating a distributed and resilient system. The goal is for all nodes in this system to contain the same set of data, forming what is known as a database cluster. From the perspective of a client application, this distributed nature is often transparent, allowing it to interact with the cluster as if it were a single database.

Replication Architectures

Primary/Replica

The most common replication architecture is Primary/Replica (also known as Master/Slave). In this model:

The Primary node is the authoritative source. It is the only node that accepts write operations (e.g., INSERT, UPDATE, DELETE).
The Primary logs these changes and sends them to one or more Replica nodes.
The Replicas receive the stream of changes and apply them to their own copy of the data. Replicas are typically used for read-only queries, backups, or as a hot standby for failover.

Multi-Primary Replication

In a multi-primary system, every node in the cluster acts as a primary. This means any node can accept write operations. When a node receives an update, it automatically propagates that change to all other primary nodes in the cluster. Each primary node logs its own changes and communicates them to its peers to maintain synchronization.

Replication Protocols: Asynchronous vs. Synchronous

Beyond the architecture, the replication protocol determines how transactions are confirmed across the cluster.

Asynchronous Replication (Lazy Replication)

In asynchronous replication, the primary node commits a transaction locally first and then sends the changes to the replicas in the background. The transaction is confirmed as complete to the client immediately after it's saved on the primary. This means there is a brief period, known as replication lag, where the replicas have not yet received the latest data.

Synchronous Replication (Eager Replication)

In synchronous replication, a transaction is not considered complete (committed) until it has been successfully applied and confirmed on all participating nodes. When the client receives confirmation, it is a guarantee that the data exists consistently across the cluster.

The Trade-offs of Synchronous Replication

Advantages

Synchronous replication offers several powerful advantages over its asynchronous counterpart:

High Availability: Since all nodes are fully synchronized, if one node fails, there is zero data loss. Traffic can be immediately directed to another node without complex failover procedures, as all data replicas are guaranteed to be consistent.
Read-After-Write Consistency: Synchronous replication guarantees causality. A SELECT query issued immediately after a transaction will always see the effects of that transaction, even if the query is executed on a different node in the cluster.

Disadvantages

Traditionally, eager replication protocols coordinate nodes one operation at a time. They use a two phase commit, or distributed locking. A system with $𝑛$ number of nodes due to process $𝑜$ operations with a throughput of $𝑡$ transactions per second gives you $𝑚$ messages per second with:

$𝑚=𝑛×𝑜×𝑡$

What this means that any increase in the number of nodes leads to an exponential growth in the transaction response times and in the probability of conflicts and deadlock rates.

For this reason, asynchronous replication remains the dominant replication protocol for database performance, scalability and availability. Widely adopted open source databases, such as MySQL and PostgreSQL only provide asynchronous replication solutions.

Galera's Solution: Modern Synchronous Replication

Galera Cluster solves the traditional problems of synchronous replication by using a modern, certification-based approach built on several key innovations:

Group Communication: A robust messaging layer ensures that information is delivered to all nodes reliably and in the correct order, forming a solid foundation for data consistency.
Write-Set Replication: Instead of coordinating on every individual operation, database changes (writes) are grouped into a single package called a "write-set." This write-set is replicated as a single message, avoiding the high overhead of traditional two-phase commit.
Optimistic Execution: Transactions are first executed optimistically on a local node. The resulting write-set is then broadcast to the cluster for a fast, parallel certification process. If it passes certification (meaning no conflicts), it is committed on all nodes.
Transaction Reordering: This clever technique allows for the reordering of non-conflicting transactions before they are committed, which significantly increases parallelism and reduces the rate of transaction rollbacks.

The certification-based replication system that Galera Cluster uses is built on these powerful approaches, delivering the benefits of synchronous replication without the traditional performance bottlenecks.

How it Works

MariaDB Enterprise Cluster is built on MariaDB Enterprise Server with Galera Cluster and MariaDB MaxScale. In MariaDB Enterprise Server 10.5 and later, it features enterprise-specific options, such as data-at-rest encryption for the write-set cache, that are not available in other Galera Cluster implementations.

As a multi-primary replication solution, any MariaDB Enterprise Server can operate as a Primary Server. This means that changes made to any node in the cluster replicate to every other node in the cluster, using certification-based replication and global ordering of transactions for the InnoDB storage engine.

MariaDB Enterprise Cluster is only available for Linux operating systems.

Architecture

There are a few things to consider when planning the hardware, virtual machines, or containers for MariaDB Enterprise Cluster.

MariaDB Enterprise Cluster architecture involves deploying MariaDB MaxScale with multiple instances of MariaDB Enterprise Server. The Servers are configured to use multi-primary replication to maintain consistency between themselves while MariaDB MaxScale routes reads and writes between them.

The application establishes a client connection to MariaDB MaxScale. MaxScale then routes statements to one of the MariaDB Enterprise Servers in the cluster. Writes made to any node in this cluster replicate to all the other nodes of the cluster.

When MariaDB Enterprise Servers start in a cluster:

Each Server attempts to establish network connectivity with the other Servers in the cluster
Groups of connected Servers form a component
When a Server establishes network connectivity with the Primary Component, it synchronizes its local database with that of the cluster
As a member of the Primary Component, the Server becomes operational — able to accept read and write queries from clients
During startup, the Primary Component is the Server bootstrapped to run as the Primary Component. Once the cluster is online, the Primary Component is any combination of Servers which includes a minimum of more than half the total number of Servers.
A Server or group of Servers that loses network connectivity with the majority of the cluster becomes non-operational.

In planning the number of systems to provision for MariaDB Enterprise Cluster, it is important to keep cluster operation in mind. Ensuring that it has enough disk space and that it is able to maintain a Primary Component in the event of outages.

Each Server requires the minimum amount of disk space needed to store the entire database. The upper storage limit for MariaDB Enterprise Cluster is that of the smallest disk in use.
Each switch in use should have an odd number of Servers above three.
In a cluster that spans multiple switches, each data center in use should have an odd number of switches above three.
In a cluster that spans multiple data centers, use an odd number of data centers above three.
Each data center in use should have at least one Server dedicated to backup operations. This can be another cluster node or a separate Replica Server kept in sync using MariaDB Replication.

In case of planning Servers to the switch, switches to the data center, and data centers in the cluster, this model helps preserve the Primary Component. A minimum of three in use means that a single Server or switch can fail without taking down the cluster.

Using an odd number above three reduces the risk of a split-brain situation (that is, a case where two separate groups of Servers believe that they are part of the Primary Component and remain operational).

Cluster Configuration

Nodes in MariaDB Enterprise Cluster are individual MariaDB Enterprise Servers configured to perform multi-primary cluster replication. This configuration is set using a series of system variables in the configuration file.

[mariadb]

# General Configuration
bind_address             = 0.0.0.0
innodb_autoinc_lock_mode = 2

# Cluster Configuration
wsrep_cluster_name    = "accounting_cluster"
wsrep_cluster_address = "gcomm://192.0.2.1,192.0.2.2,192.0.2.3"

# wsrep Provider
wsrep_provider = /usr/lib/galera/libgalera_enterprise_smm.so
wsrep_provider_options = "evs.suspect_timeout=PT10S"

Additional information on system variables is available in the Reference chapter.

General Configuration

The innodb_autoinc_lock_mode system variable must be set to a value of 2 to enable interleaved lock mode. MariaDB Enterprise Cluster does not support other lock modes.

Ensure also that the bind_address system variable is properly set to allow MariaDB Enterprise Server to listen for TCP/IP connections:

bind_address             = 0.0.0.0
innodb_autoinc_lock_mode = 2

Cluster Name and Address

MariaDB Enterprise Cluster requires that you set a name for your cluster, using the wsrep_cluster_name system variable. When nodes connect to each other, they check the cluster name to ensure that they've connected to the correct cluster before replicating data. All Servers in the cluster must have the same value for this system variable.

Using the wsrep_cluster_address system variable, you can define the back-end protocol (always gcomm) and comma-separated list of the IP addresses or domain names of the other nodes in the cluster.

wsrep_cluster_name    = "accounting_cluster"
wsrep_cluster_address = "gcomm://192.0.2.1,192.0.2.2,192.0.2.3"

It is best practice to list all nodes on this system variable, as this is the list the node searches when attempting to reestablish network connectivity with the primary component.

Note: In certain environments, such as deployments in the cloud, you may also need to set the wsrep_node_address system variable, so that MariaDB Enterprise Server properly informs other Servers how to reach it.

Galera Replicator Plugin

MariaDB Enterprise Server connects to other Servers and replicates data from the cluster through a wsrep Provider called the Galera Replicator plugin. In order to enable clustering, specify the path to the relevant .so file using the wsrep_provider system variable.

MariaDB Enterprise Server 10.4 and later installations use an enterprise-build of the Galera Enterprise 4 plugin. This includes all the features of Galera Cluster 4 as well as enterprise features like GCache encryption.

To enable MariaDB Enterprise Cluster, use the libgalera_enterprise_smm.so library:

wsrep_provider = /usr/lib/galera/libgalera_enterprise_smm.so

MariaDB Enterprise Server use the older community-release of the Galera 3 plugin. This is set using the libgalera_smm.so library:

wsrep_provider = /usr/lib/galera/libgalera_smm.so

In addition to system variables, there is a set of options that you can pass to the wsrep Provider to configure or to otherwise adjust its operations. This is done through the wsrep_provider_options system variable:

wsrep_provider_options = "evs.suspect_timeout=PT10S"

Additional information is available in the Reference chapter.

Cluster Replication

MariaDB Enterprise Cluster implements a multi-primary replication solution.

When you write to a table on a node, the node collects the write into a write-set transaction, which it then replicates to the other nodes in the cluster.

Your application can write to any node in the cluster. Each node certifies the replicated write-set. If the transaction has no conflicts, the nodes apply it. If the transaction does have conflicts, it is rejected and all of the nodes revert the changes.

Quorum

The first node you start in MariaDB Enterprise Cluster bootstraps the Primary Component. Each subsequent node that establishes a connection joins and synchronizes with the Primary Component. A cluster achieves a quorum when more than half the nodes are joined to the Primary Component.

When a component forms that has less than half the nodes in the cluster, it becomes non-operational, since it believes there is a running Primary Component to which it has lost network connectivity.

These quorum requirements, combined with the requisite number of odd nodes, avoid a split brain situation, or one in which two separate components believe they are each the Primary Component.

Dynamically Bootstrapping the Cluster

In cases where the cluster goes down and your nodes become non-operational, you can dynamically bootstrap the cluster.

First, find the most up-to-date node (that is, the node with the highest value for the wsrep_last_committed status variable):

SHOW STATUS LIKE 'wsrep_last_committed';

Once you determine the node with the most recent transaction, you can designate it as the Primary Component by running the following on it:

SET GLOBAL wsrep_provider_options="pc.bootstrap=YES";

The node bootstraps the Primary Component onto itself. Other nodes in the cluster with network connectivity then submit state transfer requests to this node to bring their local databases into sync with what's available on this node.

State Transfers

From time to time a node can fall behind the cluster. This can occur due to expensive operations being issued to it or due to network connectivity issues that lead to write-sets backing up in the queue. Whatever the cause, when a node finds that it has fallen too far behind the cluster, it attempts to initiate a state transfer.

In a state transfer, the node connects to another node in the cluster and attempts to bring its local database back in sync with the cluster. There are two types of state transfers:

Incremental State Transfer (IST)
State Snapshot Transfer (SST)

When the donor node receives a state transfer request, it checks its write-set cache (that is, the GCache) to see if it has enough saved write-sets to bring the joiner into sync. If the donor node has the intervening write-sets, it performs an IST operation, where the donor node only sends the missing write-sets to the joiner. The joiner applies these write-sets following the global ordering to bring its local databases into sync with the cluster.

When the donor does not have enough write-sets cached for an IST, it runs an SST operation. In an SST, the donor uses a backup solution, like MariaDB Enterprise Backup, to copy its data directory to the joiner. When the joiner completes the SST, it begins to process the write-sets that came in during the transfer. Once it's in sync with the cluster, it becomes operational.

IST's provide the best performance for state transfers and the size of the GCache may need adjustment to facilitate their use.

Flow Control

MariaDB Enterprise Server uses Flow Control to sometimes throttle transactions and ensure that all nodes work equitably.

Write-sets that replicate to a node are collected by the node in its received queue. The node then processes the write-sets according to global ordering. Large transactions, expensive operations, or simple hardware limitations can lead to the received queue backing up over time.

When a node's received queue grows beyond certain limits, the node initiates Flow Control. In Flow Control, the node pauses replication to work through the write-sets it already has. Once it has worked the received queue down to a certain size, it re-initiates replication.

Eviction

A node or nodes will be removed or evicted from a cluster if it becomes non-responsive.

In MariaDB Enterprise Cluster, each node monitors network connectivity and response times from every other node. MariaDB Enterprise Cluster evaluates network performance using the EVS Protocol.

When a node finds another to have poor network connectivity, it adds an entry to the delayed list. If the node becomes active again and the network performance improves for a certain amount of time, an entry for it is removed from the delayed list. That is, the longer a node has network problems the longer it has to be active again to be removed from the delayed list.

If the number of entries for a node in the delayed list exceeds a threshold established for the cluster, the EVS Protocol evicts the node from the cluster.

Evicted nodes become non-operational components. They cannot rejoin the cluster until you restart MariaDB Enterprise Server.

Streaming Replication

Under normal operation, huge transactions and long-running transactions are difficult to replicate. MariaDB Enterprise Cluster rejects conflicting transactions and rolls back the changes. A transaction that takes several minutes or longer to run can encounter issues if a small transaction is run on another node and attempts to write to the same table. The large transaction fails because it encounters a conflict when it attempts to replicate.

MariaDB Enterprise Server 10.4 and later support streaming replication for MariaDB Enterprise Cluster. In streaming replication, huge transactions are broken into transactional fragments, which are replicated and applied as the operation runs. This makes it more difficult for intervening sessions to introduce conflicts.

Initiate Streaming Replication

To initiate streaming replication, set the wsrep_trx_fragment_unit and wsrep_trx_fragment_size system variables. You can set the unit to BYTES, ROWS, or STATEMENTS:

SET SESSION wsrep_trx_fragment_unit='STATEMENTS';

SET SESSION wsrep_trx_fragment_size=5;

Then, run your transaction.

Streaming replication works best with very large transactions where you don't expect to encounter conflicts. If the statement does encounter a conflict, the rollback operation is much more expensive than usual. As such, it's best practice to enable streaming replication at a session-level and to disable it by setting the wsrep_trx_fragment_size system variable to 0 when it's not needed.

SET SESSION wsrep_trx_fragment_size=0;

Galera Arbitrator

Deployments on mixed hardware can introduce issues where some MariaDB Enterprise Servers perform better than others. A Server in one part of the world might perform more reliably or be physically closer to most users than others. In cases where a particular MariaDB Enterprise Server holds logical significance for your cluster, you can weight its value in quorum calculations.

Galera Arbitrator is a separate process that runs alongside MariaDB Enterprise Server. While the Arbitrator does not take part in replication, whenever the cluster performs quorum calculations it gives the Arbitrator a vote as though it were another MariaDB Enterprise Server. In effect this means that the system has the vote of MariaDB Enterprise Server plus any running Arbitrators in determining whether it's part of the Primary Component.

Bear in mind that the Galera Arbitrator is a separate package, galera-arbitrator-4, which is not installed by default with MariaDB Enterprise Server.

Scale-out

MariaDB Enterprise Servers that join a cluster attempt to connect to the IP addresses provided to the wsrep_cluster_address system variable. This variable adjusts itself at runtime to include the addresses of all connected nodes.

To scale-out MariaDB Enterprise Cluster, start new MariaDB Enterprise Servers with the appropriate wsrep_cluster_address list and the same wsrep_cluster_name value. The new nodes establish network connectivity with the running cluster and request a state transfer to bring their local database into sync with the cluster.

Once the MariaDB Enterprise Server reports itself as being in sync with the cluster, MariaDB MaxScale can begin including it in the load distribution for the cluster.

Being a multi-primary replication solution means that any MariaDB Enterprise Server in the cluster can handle write operations, but write scale-out is minimal as every Server in the cluster needs to apply the changes.

Failover

MariaDB Enterprise Cluster does not provide failover capabilities on its own. MariaDB MaxScale is used to route client connections to MariaDB Enterprise Server.

Unlike a traditional load balancer, MariaDB MaxScale is aware of changes in the node and cluster states.

MaxScale takes nodes out of the distribution that initiate a blocking SST operation or Flow Control or otherwise go down, which allows them to recover or catch up without stopping service to the rest of the cluster.

Backups

With MariaDB Enterprise Cluster, each node contains a replica of all the data in the cluster. As such, you run on any node to back up the available data. The process for backing up a node is the same as for a single MariaDB Enterprise Server.

Encryption

MariaDB Enterprise Server supports data-at-rest encryption to secure data on disk, and data-in-transit encryption to secure data on the network.

MariaDB Enterprise Server support data-at-rest encryption of the GCache, the file used by Galera systems to cache write sets. Encrypting GCache ensures the Server encrypts both data it temporarily caches from the cluster as well as the data it permanently stores in tablespaces.

For data-in-transit, MariaDB Enterprise Cluster supports encryption the same as MariaDB Server and additionally provides data-in-transit encryption for Galera replication traffic and for State Snapshot Transfer (SST) traffic.

Data-in-Transit Encryption

MariaDB Enterprise Server 10.6 encrypts Galera replication and SST traffic using the server's TLS configuration by default. With the wsrep_ssl_mode system variable, you can configure the node to use the TLS configuration of wsrep Provider options.

MariaDB Enterprise Server 10.5 and earlier support encrypting Galera replication and SST traffic through wsrep Provider options.

TLS encryption is only available when used by all nodes in the cluster.

Enabling GCache Encryption

To encrypt data-at-rest such as GCache, stop the server, set encrypt_binlog=ON within the MariaDB Enterprise Server configuration file, and restart the server. This variable also controls encryption of the binary log and the relay log when used.

[mariadb]
...
# Controls Binary Log, Relay Log, and GCache Encryption
encrypt_binlog=ON

Disabling GCache Encryption

To stop using encryption on the GCache file, stop the server, set encrypt_binlog=OFF within the MariaDB Enterprise Server configuration file, and restart the server. This variable also controls encryption of the binary log and the relay log when used.

[mariadb]
...
# Controls Binary Log, Relay Log, and GCache Encryption
encrypt_binlog=OFF

Quickstart Guides

Get started quickly with MariaDB Galera Cluster using these guides. Follow step-by-step instructions to deploy and configure a highly available, multi-master cluster for your applications.

MariaDB Galera Cluster Guide

MariaDB Galera Cluster quickstart guide

Quickstart Guide: MariaDB Galera Cluster

MariaDB Galera Cluster provides a multi-primary (active-active) cluster solution for MariaDB, enabling high availability, read/write scalability, and true synchronous replication. This means any node can handle read and write operations, with changes instantly replicated to all other nodes, ensuring no replica lag and no lost transactions. It's exclusively available on Linux.

1. Prerequisites

Before starting, ensure you have:

At least three nodes: For redundancy and avoiding split-brain scenarios (bare-metal or virtual machines).
Linux Operating System: A compatible Debian-based (e.g., Ubuntu, Debian) or RHEL-based (e.g., CentOS, Fedora) distribution.
Synchronized Clocks: All nodes should have NTP configured for time synchronization.
SSH Access: Root or sudo access to all nodes for installation and configuration.
Network Connectivity: All nodes must be able to communicate with each other over specific ports (see Firewall section). Low latency between nodes is ideal.
rsync: Install rsync on all nodes, as it's commonly used for State Snapshot Transfers (SST).
- sudo apt install rsync (Debian/Ubuntu)
- sudo yum install rsync (RHEL/CentOS)

2. Installation (on Each Node)

Install MariaDB Server and the Galera replication provider on all nodes of your cluster.

a. Add MariaDB Repository:

It's recommended to install from the official MariaDB repositories to get the latest stable versions. Use the MariaDB Repository Configuration Tool (search "MariaDB Repository Generator") to get specific instructions for your OS and MariaDB version.

Example for Debian/Ubuntu (MariaDB 10.11):

b. Install MariaDB Server and Galera:

c. Secure MariaDB Installation:

Run the security script on each node to set the root password and remove insecure defaults.

Set a strong root password.
Answer Y to remove anonymous users, disallow remote root login, remove test database, and reload privilege tables.

3. Firewall Configuration (on Each Node)

Open the necessary ports on each node's firewall to allow inter-node communication.

Adjust for your firewall system (e.g., firewalld for RHEL-based systems).

4. Configure Galera Cluster (`galera.cnf` on Each Node)

Create a configuration file (e.g., /etc/mysql/conf.d/galera.cnf) on each node. The content will be largely identical, with specific changes for each node's name and address.

Example galera.cnf content:

Important:

wsrep_cluster_address: List the IP addresses of all nodes in the cluster on every node.
wsrep_node_name: Must be unique for each node (e.g., node1, node2, node3).
wsrep_node_address: Set to the specific IP address of the node you are configuring.

5. Start the Cluster

a. Bootstrapping the First Node:

Start MariaDB on the first node with the --wsrep-new-cluster option. This tells it to form a new cluster. Do this only once for the initial node of a new cluster.

b. Starting Subsequent Nodes:

For the second and third nodes, start the MariaDB service normally. They will discover and join the existing cluster using the wsrep_cluster_address specified in their configuration.

6. Verify Cluster Operation

After all nodes are started, verify that they have joined the cluster.

a. Check Cluster Size (on any node):

Connect to MariaDB on any node and check the cluster status:

Inside the MariaDB shell:

The Value should match the number of nodes in your cluster (e.g., 3).

b. Test Replication:

On node1, create a new database and a table:
On node2 (or node3), connect to MariaDB and check for the new database and table:
Insert data from node2:
Verify on node1 that the new data is present:

This confirms synchronous replication is working.

Further Resources:

MariaDB Galera Cluster Replication Guide

MariaDB Galera Cluster Replication quickstart guide

Quickstart Guide: About Galera Replication

Galera Replication is a core technology enabling MariaDB Galera Cluster to provide a highly available and scalable database solution. It is characterized by its virtually synchronous replication, ensuring strong data consistency across all cluster nodes.

1. What is Galera Replication?

Galera Replication is a multi-primary replication solution for database clustering. Unlike traditional asynchronous or semi-synchronous replication, Galera ensures that transactions are committed on all nodes (or fail on all) before the client receives a success confirmation. This mechanism eliminates data loss and minimizes replica lag, making all nodes active and capable of handling read and write operations.

2. How Galera Replication Works

The core of Galera Replication revolves around the concept of write sets and the wsrep API:

Write Set Broadcasting: When a client commits a transaction on any node in the cluster, that node (the "donor" for that specific transaction) captures the changes (the "write set") associated with that transaction. This write set is then broadcasted to all other nodes in the cluster.
Certification and Application: Each receiving node performs a "certification" test to ensure that the incoming write set does not conflict with any concurrent transactions being committed locally.
- If the write set passes certification, it is applied to the local database, and the transaction is committed on that node.
- If a conflict is detected, the conflicting transaction (usually the one that was executed locally) is aborted, ensuring data consistency across the cluster.
Virtually Synchronous: The term "virtually synchronous" means that while the actual data application might happen slightly after the commit on the initiating node, the commit order is globally consistent, and all successful transactions are guaranteed to be applied on all active nodes. A transaction is not truly considered committed until it has passed certification on all nodes.
wsrep API: This API defines the interface between the Galera replication library (the "wsrep provider") and the database server (MariaDB). It allows the database to expose hooks for Galera to capture and apply transaction write sets.

3. Key Characteristics

Multi-Primary (Active-Active): All nodes in a Galera Cluster can be simultaneously used for both read and write operations.
Synchronous Replication (Virtual): Data is consistent across all nodes at all times, preventing data loss upon node failures.
Automatic Node Provisioning (SST/IST): When a new node joins or an existing node rejoins, Galera automatically transfers the necessary state to bring it up to date.
- State Snapshot Transfer (SST): A full copy of the database is transferred from an existing node to the joining node.
- Incremental State Transfer (IST): Only missing write sets are transferred if the joining node is not too far behind.
Automatic Membership Control: Nodes automatically detect and manage cluster membership changes (nodes joining or leaving).

Galera Replication essentially transforms a set of individual MariaDB servers into a robust, highly available, and consistent distributed database system.

Further Resources:

MariaDB Galera Cluster Usage Guide

MariaDB Galera Cluster usage guide

Quickstart Guide: MariaDB Galera Cluster Usage

This guide provides essential information for effectively using and interacting with a running MariaDB Galera Cluster. It covers connection methods, operational considerations, monitoring, and best practices for applications.

1. Connecting to the Cluster

Since Galera Cluster is multi-primary, any node can accept read and write connections.

a. Using a Load Balancer (Recommended for Production):

Deploying a load balancer or proxy (like MariaDB MaxScale, ProxySQL, or HAProxy) is the recommended approach.

MariaDB MaxScale: Provides intelligent routing (e.g., readwritesplit, router), connection pooling, and advanced cluster awareness (e.g., binlogrouter for replication clients, switchover for failover).
Other Load Balancers: Configure them to distribute connections across your Galera nodes, typically using health checks on port 3306 or other cluster-specific checks.

b. Direct Connection:

You can connect directly to any individual node's IP address or hostname using standard MariaDB client tools or connectors (e.g., mariadb command-line client, MariaDB Connector/J, Connector/Python).

Example (Command-line):Bash
```
mariadb -h <node_ip_address> -u <username> -p
```
While simple, this method lacks automatic failover; your application would need to handle connection retries and failover logic.

2. Basic Operations (Reads & Writes)

Active-Active: You can perform both read and write operations on any node in the cluster. All successful write operations are synchronously replicated to all other nodes.
Transactions: Standard SQL transactions (START TRANSACTION, COMMIT, ROLLBACK) work as expected. Galera handles the replication of committed transactions.

3. DDL (Data Definition Language) Operations

DDL operations (like CREATE TABLE, ALTER TABLE, DROP TABLE) require special attention in a synchronous multi-primary cluster to avoid conflicts and outages.

Total Order Isolation (TOI) - Default:
- This is Galera's default DDL method.
- The DDL statement is executed on all nodes in the same order, and it temporarily blocks other transactions on all nodes while it applies.
- It ensures consistency but can cause brief pauses in application activity, especially on busy clusters.
- Best Practice: Execute DDL during maintenance windows or low-traffic periods.
Rolling Schema Upgrade (RSU) / Percona's pt-online-schema-change:
- For large tables or critical production systems, use tools like pt-online-schema-change (from Percona Toolkit) which performs DDL without blocking writes.
- This tool works by creating a new table, copying data, applying changes, and then swapping the tables. It's generally preferred for minimizing downtime for ALTER TABLE operations.
wsrep_OSU_method:
- This system variable controls how DDL operations are executed.
- TOI (default): Total Order Isolation.
- RSU: Rolling Schema Upgrade (requires manual steps with pt-online-schema-change).
- NBO (Non-Blocking Operations): A newer method allowing non-blocking DDL for some operations, but not fully implemented for all DDL types. Use with caution and test thoroughly.

4. Monitoring Cluster Status

Regularly monitor your Galera Cluster to ensure its health and consistency.

wsrep_cluster_size: Number of nodes currently in the Primary Component.SQL
```
SHOW STATUS LIKE 'wsrep_cluster_size';
```
Expected value: the total number of nodes configured (e.g., 3).
wsrep_local_state_comment / wsrep_local_state: The state of the current node.
- Synced (4): Node is fully synchronized and operational.
- Donor/Desync (2): Node is transferring state to another node.
- Joining (1): Node is in the process of joining the cluster.
- Donor/Stalled (1): Node is stalled.
wsrep_incoming_addresses: List of incoming connections from other cluster nodes.
wsrep_cert_deps_distance: Indicates flow control. A high value suggests that this node is falling behind and flow control may activate.
wsrep_flow_control_paused: Percentage of time the node was paused due to flow control. High values indicate a bottleneck.
wsrep_local_recv_queue / wsrep_local_send_queue: Size of the receive/send queue. Ideally, these should be close to 0. Sustained high values indicate replication lag or node issues.

5. Handling Node Failures and Recovery

Galera Cluster is designed for automatic recovery, but understanding the process is key.

Node Failure: If a node fails, the remaining nodes continue to operate as the Primary Component. The failed node will automatically attempt to rejoin when it comes back online.
Split-Brain Scenarios: If the network partitions the cluster, nodes will try to form a "Primary Component." The partition with the majority of nodes forms the new Primary Component. If no majority can be formed (e.g., a 2-node cluster splits), the cluster will become inactive. A 3-node or higher cluster is recommended to avoid this.
Manual Bootstrapping (Last Resort): If the entire cluster goes down or a split-brain occurs where no Primary Component forms, you might need to manually "bootstrap" a new Primary Component from one of the healthy nodes.
- Choose the node that was most up-to-date.
- Stop MariaDB on that node.
- Start it with: sudo galera_new_cluster or sudo systemctl start mariadb --wsrep-new-cluster.
- Start other nodes normally; they will rejoin the bootstrapped component.

6. Application Best Practices

Use Connection Pooling: Essential for managing connections efficiently in high-traffic applications.
Short Transactions: Keep transactions as short and concise as possible to minimize conflicts and improve throughput. Long-running transactions increase the risk of rollbacks due to certification failures.
Primary Keys: All tables should have a primary key. Galera relies on primary keys for efficient row-level replication. Tables without primary keys can cause performance degradation and issues.
Retry Logic: Implement retry logic in your application for failed transactions (e.g., due to certification failures, deadlock, or temporary network issues).
Connect to a Load Balancer: Always direct your application's connections through a load balancer or proxy to leverage automatic failover and intelligent routing.

By following these guidelines, you can effectively manage and operate your MariaDB Galera Cluster for high availability and performance.

Further Resources:

Galera Management

Galera Management in MariaDB handles synchronous multi-master replication, ensuring high availability, data consistency, failover, and seamless node provisioning across clusters.

Known Limitations

This article contains information on known problems and limitations of MariaDB Galera Cluster.

Limitations from codership.com:

Currently, replication works only with the . Any writes to tables of other types, including system (MySQL) tables are not replicated (this limitation excludes DDL statements such as , which implicitly modify the mysql. tables — those are replicated). There is, however, experimental support for - see the system variable)
Unsupported explicit locking include , , (, ,…). Using transactions properly should be able to overcome these limitations. Global locking operators like are supported.
All tables should have a primary key (multi-column primary keys are supported). operations are unsupported on tables without a primary key. Also, rows in tables without a primary key may appear in a different order on different nodes.
The and the cannot be directed to a table. If you enable these logs, then you must forward the log to a file by setting .
Transaction size. While Galera does not explicitly limit the transaction size, a write set is processed as a single memory-resident buffer, and as a result, extremely large transactions (e.g. ) may adversely affect node performance. To avoid that, the and system variables limit transaction rows to 128K and the transaction size to 2Gb by default. If necessary, users may want to increase those limits. Future versions will add support for transaction fragmentation.

Other observations, in no particular order:

If you are using for state transfer, and it fails for whatever reason (e.g., you do not have the database account it attempts to connect with, or it does not have the necessary permissions), you will see an SQL SYNTAX error in the server . Don't let it fool you, this is just a fancy way to deliver a message (the pseudo-statement inside the bogus SQL will contain the error message).
Do not use transactions of any essential size. Just to insert 100K rows, the server might require an additional 200-300 Mb. In a less fortunate scenario, it can be 1.5 GB for 500K rows, or 3.5 GB for 1M rows. See for some numbers (you'll see that it's closed, but it's not closed because it was fixed).
Locking is lax when DDL is involved. For example, if your DML transaction uses a table, and a parallel DDL statement is started, in the normal MySQL setup, it would have waited for the metadata lock, but in the Galera context, it will be executed right away. It happens even if you are running a single node, as long as you have configured it as a cluster node. See also . This behavior might cause various side effects; the consequences have not been investigated yet. Try to avoid such parallelism.
Do not rely on auto-increment values to be sequential. Galera uses a mechanism based on autoincrement increment to produce unique non-conflicting sequences, so on every single node, the sequence will have gaps. See
A command may fail with ER_UNKNOWN_COM_ERROR, producing 'WSREP has not yet prepared node for application use' (or 'Unknown command' in older versions) error message. It happens when a cluster is suspected to be split, and the node is in a smaller part, for example, during a network glitch, when nodes temporarily lose each other. It can also occur during state transfer. The node takes this measure to prevent data inconsistency. It's usually a temporary state; it can be detected by checking the value. The node, however, allows the SHOW and SET commands during this period.
After a temporary split, if the 'good' part of the cluster is still reachable and its state was modified, resynchronization occurs. As a part of it, nodes of the 'bad' part of the cluster drop all client connections. It might be quite unexpected, especially if the client was idle and did not even know anything was happening. Please also note that after the connection to the isolated node is restored, if there is a flow on the node, it takes a long time for it to synchronize, during which the "good" node says that the cluster is already of the normal size and synced, while the rejoining node says it's only joined (but not synced). The connections keep getting 'unknown command'. It should pass eventually.
While is checked on startup and can only be ROW (see ), it can be changed at runtime. Do NOT change at runtime, it is likely to cause replication failure, but make all other nodes crash.
If you are using rsync for state transfer, and a node crashes before the state transfer is over, the rsync process might hang forever, occupying the port and preventing the node. The problem will show up as 'port in use' in the server error log. Find the orphaned rsync process and kill it manually.
Performance: By design performance of the cluster cannot be higher than the performance of the slowest node; however, even if you have only one node, its performance can be considerably lower compared to running the same server in a standalone mode (without wsrep provider). It is particularly true for big enough transactions (even those which are well within current limitations on transaction size quoted above).
Windows is not supported.
Replication filters: When using a Galera cluster, replication filters should be used with caution. See for more details. See also and .
Flashback isn't supported in Galera due to an incompatible binary log format.
FLUSH PRIVILEGES is not replicated.
The needed to be disabled by setting prior to MariaDB Galera Cluster 5.5.40, MariaDB Galera Cluster 10.0.14, and
In an asynchronous replication setup where a master replicates to a Galera node acting as a slave, parallel replication (slave-parallel-threads > 1) on the slave is currently not supported (see ).
The disk-based is not encrypted ().
Nodes may have different table definitions, especially temporarily during operations, but the same apply as they do for row-based replication

_{This page is licensed: CC BY-SA / Gnu FDL}

Installation & Deployment

From Source

Installing Galera from Source

There are binary installation packages available for RPM and Debian-based distributions, which will pull in all required Galera dependencies.

If these are not available, you will need to build Galera from source.

The wsrep API for Galera Cluster is included by default. Follow the usual instructions

Preparation

make cannot manage dependencies for the build process, so the following packages need to be installed first:

RPM-based:

Debian-based:

If running on an alternative system, or the commands are available, the following packages are required. You will need to check the repositories for the correct package names on your distribution—these may differ between distributions or require additional packages:

MariaDB Database Server with wsrep API

Git, CMake (on Fedora, both cmake and cmake-fedora are required), GCC and GCC-C++, Automake, Autoconf, and Bison, as well as development releases of libaio and ncurses.

Building

You can use Git to download the source code, as MariaDB source code is available through GitHub. Clone the repository:

Check out the branch (e.g., 10.5-galera or 11.1-galera), for example:

Building the Database Server

The standard and Galera Cluster database servers are the same, except that for Galera Cluster, the wsrep API patch is included. Enable the patch with the CMake configuration options. WITH_WSREP and WITH_INNODB_DISALLOW_WRITES. To build the database server, run the following commands:

There are also some build scripts in the *BUILD/* directory, which may be more convenient to use. For example, the following pre-configures the build options discussed above:

There are several others as well, so you can select the most convenient.

Besides the server with the Galera support, you will also need a Galera provider.

Preparation

make cannot manage dependencies itself, so the following packages need to be installed first:

If running on an alternative system, or the commands are available, the following packages are required. You will need to check the repositories for the correct package names on your distribution - these may differ between distributions, or require additional packages:

Galera Replication Plugin

SCons, as well as development releases of Boost (libboost_program_options, libboost_headers1), Check and OpenSSL.

Building

Run:

After this, the source files for the Galera provider will be in the galera directory.

Building the Galera Provider

The Galera Replication Plugin both implements the wsrep API and operates as the database server's wsrep Provider. To build, cd into the galera/ directory and do:

The path to libgalera_smm.so needs to be defined in the my.cnf configuration file.

Building Galera Replication Plugin from source on FreeBSD runs into issues due to Linux dependencies. To overcome these, either install the binary package: pkg install galera, or use the ports build available at /usr/ports/databases/galera.

Configuration

After building, a number of other steps are necessary:

Create the database server user and group:

Install the database (the path may be different if you specified CMAKE_INSTALL_PREFIX):

If you want to install the database in a location other than /usr/local/mysql/data , use the --basedir or --datadir options.
Change the user and group permissions for the base directory.

Create a system unit for the database server.

Galera Cluster can now be started using the service command and is set to start at boot.

_{This page is licensed: CC BY-SA / Gnu FDL}

Building the Galera wsrep Package on Ubuntu and Debian

The instructions on this page were used to create the galera package on the Ubuntu and Debian Linux distributions. This package contains the wsrep provider for MariaDB Galera Cluster.

The version of the wsrep provider is 25.3.5. We also provide 25.2.9 for those that need or want it. Prior to that, the wsrep version was 23.2.7.

Install prerequisites:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get -y install check debhelper libasio-dev libboost-dev libboost-program-options-dev libssl-dev scons

Clone galera.git from github.com/mariadb and checkout mariadb-3.x banch:

git init repo
cd repo
git clone -b mariadb-3.x https://github.com/MariaDB/galera.git

Build the packages by executing build.sh under scripts/ directory with -p switch:

cd galera
./scripts/build.sh -p

When finished, you will have the Debian packages for galera library and arbitrator in the parent directory.

Running galera test suite

If you want to run the galera test suite (mysql-test-run --suite=galera), you need to install the galera library as either /usr/lib/galera/libgalera_smm.so or /usr/lib64/galera/libgalera_smm.so

_{This page is licensed: CC BY-SA / Gnu FDL}

Galera Test Repositories

To facilitate development and QA, we have created some test repositories for the Galera wsrep provider.

These are test repositories. There will be periods when they do not work at all, or work incorrectly, or possibly cause earthquakes, typhoons, and tornadoes. You have been warned.

Galera Test Repositories for YUM

Replace ${dist} in the code below for the YUM-based distribution you are testing. Valid distributions are:

centos5-amd64
centos5-x86
centos6-amd64
centos6-x86
centos7-amd64
rhel5-amd64
rhel5-x86
rhel6-amd64
rhel6-x86
rhel6-ppc64
rhel7-amd64
rhel7-ppc64
rhel7-ppc64le
fedora22-amd64
fedora22-x86
fedora23-amd64
fedora23-x86
fedora24-amd64
fedora24-x86
opensuse13-amd64
opensuse13-x86
sles11-amd64
sles11-x86
sles12-amd64
sles12-ppc64le

Galera Test Repositories for APT

Replace ${dist} in the code below for the APT-based distribution you are testing. Valid ones are:

wheezy
jessie
sid
precise
trusty
xenial

_{This page is licensed: CC BY-SA / Gnu FDL}

Configuration

Configuring MariaDB Galera Cluster

A number of options need to be set in order for Galera Cluster to work when using MariaDB. These should be set in the MariaDB option file.

Mandatory Options

Several options are mandatory, which means that they must be set in order for Galera Cluster to be enabled or to work properly with MariaDB. The mandatory options are:

wsrep_provider — Path to the Galera library
wsrep_cluster_address — See Galera Cluster address format and usage
binlog_format=ROW — See
wsrep_on=ON — Enable wsrep replication
default_storage_engine=InnoDB — This is the default value, or alternately wsrep_replicate_myisam=1 (before MariaDB 10.6) or galera-cluster-system-variables/#wsrep_mode=REPLICATE_ARIA,REPLICATE_MYISAM (MariaDB 10.6 and later)
- innodb_doublewrite=1 — This is the default value, and should not be changed.

These are optional optimizations that can be made to improve performance.

innodb_flush_log_at_trx_commit=0 — This is not usually recommended in the case of standard MariaDB. However, it is a safer, recommended option with Galera Cluster, since inconsistencies can always be fixed by recovering from another node.
innodb_autoinc_lock_mode=2 — This tells InnoDB to use interleaved method. Interleaved is the fastest and most scalable lock mode, and should be used when BINLOG_FORMAT is set to ROW. Setting the auto-increment lock mode for InnoDB to interleaved, you’re allowing slaves threads to operate in parallel.
wsrep_slave_threads=4 — This makes state transfers quicker for new nodes. You should start with four slave threads per CPU core. The logic here is that, in a balanced system, four slave threads can typically saturate a CPU core. However, I/O performance can increase this figure several times over. For example, a single-core ThinkPad R51 with a 4200 RPM drive can use thirty-two slave threads. The value should not be set higher than wsrep_cert_deps_distance.

Writing Replicated Write Sets to the Binary Log

Like with , write sets that are received by a node with Galera Cluster's certification-based replication are not written to the by default. If you would like a node to write its replicated write sets to the , then you will have to set log_slave_updates=ON. This is especially helpful if the node is a replication master. See Using MariaDB Replication with MariaDB Galera Cluster: Configuring a Cluster Node as a Replication Master.

Replication Filters

Like with , can be used to filter write sets from being replicated by Galera Cluster's certification-based replication. However, they should be used with caution because they may not work as you'd expect.

The following replication filters are honored for InnoDB DML, but not DDL:

The following replication filters are honored for DML and DDL for tables that use both the InnoDB and MyISAM storage engines:

However, it should be kept in mind that if replication filters cause inconsistencies that lead to replication errors, then nodes may abort.

Network Ports

Galera Cluster needs access to the following ports:

Standard MariaDB Port (default: 3306) - For MySQL client connections and State Snapshot Transfers that use the mysqldump method. This can be changed by setting port.
Galera Replication Port (default: 4567) - For Galera Cluster replication traffic, multicast replication uses both UDP transport and TCP on this port. Can be changed by setting wsrep_node_address.
Galera Replication Listening Interface (default: 0.0.0.0:4567) needs to be set using gmcast.listen_addr, either
- in wsrep_provider_options: wsrep_provider_options='gmcast.listen_addr=tcp://<IP_ADDR>:<PORT>;'
- or in wsrep_cluster_address
IST Port (default: 4568) - For Incremental State Transfers. Can be changed by setting ist.recv_addr in wsrep_provider_options.
SST Port (default: 4444) - For all State Snapshot Transfer methods other than mysqldump. Can be changed by setting wsrep_sst_receive_address.

Mutiple Galera Cluster Instances on One Server

If you want to run multiple Galera Cluster instances on one server, then you can do so by starting each instance with mysqld_multi, or if you are using systemd, then you can use the relevant systemd method for interacting with multiple MariaDB instances.

You need to ensure that each instance is configured with a different datadir.

You also need to ensure that each instance is configured with different network ports.

_{This page is licensed: CC BY-SA / Gnu FDL}

Galera Cluster Address

URLs in take a particular format:

Schema

gcomm⁣- This is the option to use for a working implementation.
dummy⁣- Used for running tests and profiling, does not do any actual replication, and all following parameters are ignored.

Cluster address

The cluster address shouldn't be empty like gcomm://. This should never be hardcoded into any configuration files.
To connect the node to an existing cluster, the cluster address should contain the address of any member of the cluster you want to join.
The cluster address can also contain a comma-separated list of multiple members of the cluster. It is good practice to list all possible members of the cluster, for example. ⁣gcomm:<node1 name or ip>,<node2 name or ip2>,<node3 name or ip> Alternately, if multicast is used, put the multicast address instead of the list of nodes. Each member address or multicast address can specify <node name or ip>:<port>if a non-default port is used.

Option list

The variable is used to set a . These parameters can also be provided (and overridden) as part of the URL. Unlike options provided in a configuration file, they will not endure and need to be resubmitted with each connection.

A useful option to set is pc.wait_prim=noto ensure the server will start running even if it can't determine a primary node. This is useful if all members go down at the same time.

Port

By default, gcomm listens on all interfaces. The port is either provided in the cluster address or will default to 4567 if not set.

_{This page is licensed: CC BY-SA / Gnu FDL}

Monitoring & Recovery

State Snapshot Transfers (SSTs) in Galera Cluster

State Snapshot Transfers (SSTs) in MariaDB Galera Cluster copy the full dataset from a donor node to a new or recovering joiner node, ensuring data consistency before the joiner joins replication.

Manual SST of Galera Cluster Node With mariadb-backup

Sometimes it can be helpful to perform a "manual SST" when Galera's fail. This can be especially useful when the cluster's is very large, since a normal SST can take a long time to fail in that case.

A manual SST essentially consists of taking a backup of the donor, loading the backup on the joiner, and then manually editing the cluster state on the joiner node. This page will show how to perform this process with .

Process

Check that the donor and joiner nodes have the same mariadb-backup version.

Create backup directory on donor.

Take a the of the donor node with mariadb-backup. The option should also be provided, so that the node's cluster state is also backed up.

Verify that the MariaDB Server process is stopped on the joiner node. This will depend on your .

For example, on systems, you can execute::

Create the backup directory on the joiner node.

Copy the backup from the donor node to the joiner node.

on the joiner node.

Get the Galera Cluster version ID from the donor node's grastate.dat file.

For example, a very common version number is "2.1".

Get the node's cluster state from the file in the backup that was copied to the joiner node.

The file contains the values of the and status variables.

The values are written in the following format:

For example:

Create a grastate.dat file in the backup directory of the joiner node. The Galera Cluster version ID, the cluster uuid, and the seqno from previous steps will be used to fill in the relevant fields.

For example, with the example values from the last two steps, we could do:

Remove the existing contents of the on the joiner node.

Copy the contents of the backup directory to the the on joiner node.

Make sure the permissions of the are correct on the joiner node.

Start the MariaDB Server process on the joiner node. This will depend on your .

For example, on systems, you can execute::

Watch the MariaDB on the joiner node and verify that the node does not need to perform a due to the manual SST.

Upgrading Galera Cluster

Articles on upgrading between MariaDB versions with Galera Cluster

Upgrading from MariaDB 10.3 to MariaDB 10.4 with Galera Cluster

MariaDB starting with

Since , the MySQL-wsrep patch has been merged into MariaDB Server. Therefore, in and above, the functionality of MariaDB Galera Cluster can be obtained by installing the standard MariaDB Server packages and the Galera wsrep provider library package.

Beginning in , Galera Cluster ships with the MariaDB Server. Upgrading a Galera Cluster node is very similar to upgrading a server from to . For more information on that process as well as incompatibilities between versions, see the .

Performing a Rolling Upgrade

The following steps can be used to perform a rolling upgrade from to when using Galera Cluster. In a rolling upgrade, each node is upgraded individually, so the cluster is always operational. There is no downtime from the application's perspective.

First, before you get started:

First, take a look at to see what has changed between the major versions.
Check whether any system variables or options have been changed or removed. Make sure that your server's configuration is compatible with the new MariaDB version before upgrading.
Check whether replication has changed in the new MariaDB version in any way that could cause issues while the cluster contains upgraded and non-upgraded nodes.
Check whether any new features have been added to the new MariaDB version. If a new feature in the new MariaDB version cannot be replicated to the old MariaDB version, then do not use that feature until all cluster nodes have been upgrades to the new MariaDB version.
Next, make sure that the Galera version numbers are compatible.
If you are upgrading from the most recent release to , then the versions will be compatible. uses Galera 3 (i.e. Galera wsrep provider versions 25.3.x), and uses Galera 4 (i.e. Galera wsrep provider versions 26.4.x). This means that upgrading to also upgrades the system to Galera 4. However, Galera 3 and Galera 4 should be compatible for the purposes of a rolling upgrade, as long as you are using Galera 26.4.2 or later.
See What is MariaDB Galera Cluster? Galera wsrep provider Versions for information on which MariaDB releases uses which Galera wsrep provider versions.
Ideally, you want to have a large enough gcache to avoid a State Snapshot Transfer (SST) during the rolling upgrade. The gcache size can be configured by setting gcache.size For example:wsrep_provider_options="gcache.size=2G"

Before you upgrade, it would be best to take a backup of your database. This is always a good idea to do before an upgrade. We would recommend .

Then, for each node, perform the following steps:

Modify the repository configuration, so the system's package manager installs . For example,

On Debian, Ubuntu, and other similar Linux distributions, see for more information.
On RHEL, CentOS, Fedora, and other similar Linux distributions, see for more information.
On SLES, OpenSUSE, and other similar Linux distributions, see for more information.

If you use a load balancing proxy such as MaxScale or HAProxy, make sure to drain the server from the pool so it does not receive any new connections.
.
Uninstall the old version of MariaDB and the Galera wsrep provider.

On Debian, Ubuntu, and other similar Linux distributions, execute the following:sudo apt-get remove mariadb-server galera
On RHEL, CentOS, Fedora, and other similar Linux distributions, execute the following:sudo yum remove MariaDB-server galera
On SLES, OpenSUSE, and other similar Linux distributions, execute the following:sudo zypper remove MariaDB-server galera

Install the new version of MariaDB and the Galera wsrep provider.

On Debian, Ubuntu, and other similar Linux distributions, see for more information.
On RHEL, CentOS, Fedora, and other similar Linux distributions, see for more information.
On SLES, OpenSUSE, and other similar Linux distributions, see for more information.

Make any desired changes to configuration options in , such as my.cnf. This includes removing any system variables or options that are no longer supported.
On Linux distributions that use systemd you may need to increase the service startup timeout as the default timeout of 90 seconds may not be sufficient. See for more information.
.
Run with the --skip-write-binlog option.

mysql_upgrade does two things:
1. Ensures that the system tables in the l database are fully compatible with the new version.
2. Does a very quick check of all tables and marks them as compatible with the new version of MariaDB.

When this process is done for one node, move onto the next node.

Note that when upgrading the Galera wsrep provider, sometimes the Galera protocol version can change. The Galera wsrep provider should not start using the new protocol version until all cluster nodes have been upgraded to the new version, so this is not generally an issue during a rolling upgrade. However, this can cause issues if you restart a non-upgraded node in a cluster where the rest of the nodes have been upgraded.

_{This page is licensed: CC BY-SA / Gnu FDL}

Upgrading from MariaDB 10.4 to MariaDB 10.5 with Galera Cluster

ships with the MariaDB Server. Upgrading a Galera Cluster node is very similar to upgrading a server from to . For more information on that process as well as incompatibilities between versions, see the .

Performing a Rolling Upgrade

First, before you get started:

First, take a look at to see what has changed between the major versions.
Check whether any system variables or options have been changed or removed. Make sure that your server's configuration is compatible with the new MariaDB version before upgrading.
Check whether replication has changed in the new MariaDB version in any way that could cause issues while the cluster contains upgraded and non-upgraded nodes.
Check whether any new features have been added to the new MariaDB version. If a new feature in the new MariaDB version cannot be replicated to the old MariaDB version, then do not use that feature until all cluster nodes have been upgrades to the new MariaDB version.
Next, make sure that the Galera version numbers are compatible.
If you are upgrading from the most recent release to , then the versions will be compatible.
See ?: s for information on which MariaDB releases uses which Galera wsrep provider versions.
You want to have a large enough gcache to avoid a during the rolling upgrade. The gcache size can be configured by setting For example:wsrep_provider_options="gcache.size=2G"

Before you upgrade, it would be best to take a backup of your database. This is always a good idea to do before an upgrade. We would recommend .

Then, for each node, perform the following steps:

Modify the repository configuration, so the system's package manager installs . For example,

On Debian, Ubuntu, and other similar Linux distributions, see for more information.
On RHEL, CentOS, Fedora, and other similar Linux distributions, see for more information.
On SLES, OpenSUSE, and other similar Linux distributions, see for more information.

If you use a load balancing proxy such as MaxScale or HAProxy, make sure to drain the server from the pool so it does not receive any new connections.
Uninstall the old version of MariaDB and the Galera wsrep provider.

On Debian, Ubuntu, and other similar Linux distributions, execute the following:sudo apt-get remove mariadb-server galera
On RHEL, CentOS, Fedora, and other similar Linux distributions, execute the following:sudo yum remove MariaDB-server galera
On SLES, OpenSUSE, and other similar Linux distributions, execute the following:sudo zypper remove MariaDB-server galera

Install the new version of MariaDB and the Galera wsrep provider.

On Debian, Ubuntu, and other similar Linux distributions, see for more information.
On RHEL, CentOS, Fedora, and other similar Linux distributions, see for more information.
On SLES, OpenSUSE, and other similar Linux distributions, see for more information.

Make any desired changes to configuration options in , such as my.cnf. This includes removing any system variables or options that are no longer supported.
On Linux distributions that use systemd you may need to increase the service startup timeout as the default timeout of 90 seconds may not be sufficient. See for more information.
Run with the --skip-write-binlog option.

mariadb-upgrade does two things:
1. Ensures that the system tables in the database are fully compatible with the new version.
2. Does a very quick check of all tables and marks them as compatible with the new version of MariaDB.

When this process is done for one node, move onto the next node.

_{This page is licensed: CC BY-SA / Gnu FDL}

Upgrading from MariaDB 10.5 to MariaDB 10.6 with Galera Cluster

Performing a Rolling Upgrade

First, before you get started:

First, take a look at to see what has changed between the major versions.
Check whether any system variables or options have been changed or removed. Make sure that your server's configuration is compatible with the new MariaDB version before upgrading.
Check whether replication has changed in the new MariaDB version in any way that could cause issues while the cluster contains upgraded and non-upgraded nodes.
Check whether any new features have been added to the new MariaDB version. If a new feature in the new MariaDB version cannot be replicated to the old MariaDB version, then do not use that feature until all cluster nodes have been upgrades to the new MariaDB version.
Next, make sure that the Galera version numbers are compatible.
If you are upgrading from the most recent release to , then the versions will be compatible.
See ?: for information on which MariaDB releases uses which Galera wsrep provider versions.
You want to have a large enough gcache to avoid a during the rolling upgrade. The gcache size can be configured by setting for example:wsrep_provider_options="gcache.size=2G"

Before you upgrade, it would be best to take a backup of your database. This is always a good idea to do before an upgrade. We would recommend .

Then, for each node, perform the following steps:

Modify the repository configuration, so the system's package manager installs For example,

On Debian, Ubuntu, and other similar Linux distributions, see for more information.
On RHEL, CentOS, Fedora, and other similar Linux distributions, see for more information.
On SLES, OpenSUSE, and other similar Linux distributions, see for more information.

If you use a load balancing proxy such as MaxScale or HAProxy, make sure to drain the server from the pool so it does not receive any new connections.
.
Uninstall the old version of MariaDB and the Galera wsrep provider.

On Debian, Ubuntu, and other similar Linux distributions, execute the following:sudo apt-get remove mariadb-server galera-4
On RHEL, CentOS, Fedora, and other similar Linux distributions, execute the following:sudo yum remove MariaDB-server galera-4
On SLES, OpenSUSE, and other similar Linux distributions, execute the following:sudo zypper remove MariaDB-server galera-4

Install the new version of MariaDB and the Galera wsrep provider.

On Debian, Ubuntu, and other similar Linux distributions, see for more information.
On RHEL, CentOS, Fedora, and other similar Linux distributions, see for more information.
On SLES, OpenSUSE, and other similar Linux distributions, see for more information.

Make any desired changes to configuration options in , such as my.cnf. This includes removing any system variables or options that are no longer supported.
On Linux distributions that use systemd you may need to increase the service startup timeout as the default timeout of 90 seconds may not be sufficient. See for more information.
Run with the --skip-write-binlog option.

mariadb-upgrade does two things:
1. Ensures that the system tables in the database are fully compatible with the new version.
2. Does a very quick check of all tables and marks them as compatible with the new version of MariaDB.

When this process is done for one node, move onto the next node.

_{This page is licensed: CC BY-SA / Gnu FDL}

Upgrading from MariaDB 10.6 to MariaDB 10.11 with Galera Cluster

Methods

Stopping all nodes, upgrading all nodes, then starting the nodes
Rolling upgrade with IST (however, see )
Note that rolling upgrade with SST does not work

Performing a Rolling Upgrade

First, before you get started:

First, take a look at to see what has changed between the major versions.
Check whether any system variables or options have been changed or removed. Make sure that your server's configuration is compatible with the new MariaDB version before upgrading.
Check whether replication has changed in the new MariaDB version in any way that could cause issues while the cluster contains upgraded and non-upgraded nodes.
Check whether any new features have been added to the new MariaDB version. If a new feature in the new MariaDB version cannot be replicated to the old MariaDB version, then do not use that feature until all cluster nodes have been upgrades to the new MariaDB version.
Next, make sure that the Galera version numbers are compatible.
If you are upgrading from the most recent release to , then the versions will be compatible.
See ?: for information on which MariaDB releases uses which Galera wsrep provider versions.
You want to have a large enough gcache to avoid a during the rolling upgrade. The gcache size can be configured by setting For example:wsrep_provider_options="gcache.size=2G"

Before you upgrade, it would be best to take a backup of your database. This is always a good idea to do before an upgrade. We would recommend .

Then, for each node, perform the following steps:

Modify the repository configuration, so the system's package manager installs . For example,

On Debian, Ubuntu, and other similar Linux distributions, see for more information.
On RHEL, CentOS, Fedora, and other similar Linux distributions, see for more information.
On SLES, OpenSUSE, and other similar Linux distributions, see for more information

If you use a load balancing proxy such as MaxScale or HAProxy, make sure to drain the server from the pool so it does not receive any new connections.
.
Uninstall the old version of MariaDB and the Galera wsrep provider.

On Debian, Ubuntu, and other similar Linux distributions, execute the following:sudo apt-get remove mariadb-server galera-4
On RHEL, CentOS, Fedora, and other similar Linux distributions, execute the following:sudo yum remove MariaDB-server galera-4
On SLES, OpenSUSE, and other similar Linux distributions, execute the following:sudo zypper remove MariaDB-server galera-4

Install the new version of MariaDB and the Galera wsrep provider.

On Debian, Ubuntu, and other similar Linux distributions, see for more information.
On RHEL, CentOS, Fedora, and other similar Linux distributions, see for more information.
On SLES, OpenSUSE, and other similar Linux distributions, see for more information.

Make any desired changes to configuration options in , such as my.cnf. This includes removing any system variables or options that are no longer supported.
On Linux distributions that use systemd you may need to increase the service startup timeout as the default timeout of 90 seconds may not be sufficient. See for more information.
.
With wsrep off, run with the --skip-write-binlog option.

mariadb-upgrade does two things:
1. Ensures that the system tables in the database are fully compatible with the new version.
2. Does a very quick check of all tables and marks them as compatible with the new version of MariaDB.

Restart the server with wsrep on.

When this process is done for one node, move on to the next node.

_{This page is licensed: CC BY-SA / Gnu FDL}

Performance Tuning

Galera Security

MariaDB Galera security encrypts replication/SST traffic and ensures integrity through firewalls, secure credentials, and network isolation.

Securing Communications in Galera Cluster

By default, Galera Cluster replicates data between each node without encrypting it. This is generally acceptable when the cluster nodes runs on the same host or in networks where security is guaranteed through other means. However, in cases where the cluster nodes exist on separate networks or they are in a high-risk network, the lack of encryption does introduce security concerns as a malicious actor could potentially eavesdrop on the traffic or get a complete copy of the data by triggering an SST.

To mitigate this concern, Galera Cluster allows you to encrypt data in transit as it is replicated between each cluster node using the Transport Layer Security (TLS) protocol. TLS was formerly known as Secure Socket Layer (SSL), but strictly speaking the SSL protocol is a predecessor to TLS and, that version of the protocol is now considered insecure. The documentation still uses the term SSL often and for compatibility reasons TLS-related server system and status variables still use the prefix ssl_, but internally, MariaDB only supports its secure successors.

In order to secure connections between the cluster nodes, you need to ensure that all servers were compiled with TLS support. See to determine how to check whether a server was compiled with TLS support.

For each cluster node, you also need a certificate, private key, and the Certificate Authority (CA) chain to verify the certificate. If you want to use self-signed certificates that are created with OpenSSL, then see for information on how to create those.

Securing Galera Cluster Replication Traffic

In order to enable TLS for Galera Cluster's replication traffic, there are a number of wsrep_provider_options that you need to set, such as:

You need to set the path to the server's certificate by setting the socket.ssl_cert wsrep_provider_option.
You need to set the path to the server's private key by setting the socket.ssl_key wsrep_provider_option.
You need to set the path to the certificate authority (CA) chain that can verify the server's certificate by setting the socket.ssl_ca wsrep_provider_option.
If you want to restrict the server to certain ciphers, then you also need to set the socket.ssl_cipher wsrep_provider_option.

It is also a good idea to set MariaDB Server's regular TLS-related system variables, so that TLS will be enabled for regular client connections as well. See for information on how to do that.

For example, to set these variables for the server, add the system variables to a relevant server in an :

[mariadb]
...
ssl_cert = /etc/my.cnf.d/certificates/server-cert.pem
ssl_key = /etc/my.cnf.d/certificates/server-key.pem
ssl_ca = /etc/my.cnf.d/certificates/ca.pem
wsrep_provider_options="socket.ssl_cert=/etc/my.cnf.d/certificates/server-cert.pem;socket.ssl_key=/etc/my.cnf.d/certificates/server-key.pem;socket.ssl_ca=/etc/my.cnf.d/certificates/ca.pem"

And then restart the server to make the changes persistent.

By setting both MariaDB Server's TLS-related system variables and Galera Cluster's TLS-related wsrep_provider_options, the server can secure both external client connections and Galera Cluster's replication traffic.

Securing State Snapshot Transfers

The method that you would use to enable TLS for State Snapshot Transfers (SSTs) would depend on the value of wsrep_sst_method.

mariadb-backup

See mariadb-backup SST Method: TLS for more information.

xtrabackup-v2

See xtrabackup-v2 SST Method: TLS for more information.

mysqldump

This SST method simply uses the (previously mysqldump) utility, so TLS would be enabled by following the guide at

rsync

This SST method supports encryption in transit via stunnel. See Introduction to State Snapshot Transfers (SSTs): rsync for more information.

_{This page is licensed: CC BY-SA / Gnu FDL}

Galera Use Cases

MariaDB Galera Cluster ensures high availability and disaster recovery through synchronous multi-master replication. It's ideal for active-active setups, providing strong consistency and automatic failover, perfect for critical applications needing continuous uptime.

Common use cases for Galera replication include:

Read Master

WAN Clustering

High Availability

MariaDB ensures high availability with Replication for async/semi-sync data copying and Galera Cluster for sync multi-master with failover and zero data loss.

Galera Load Balancer

Galera Load Balancer is a simple load balancer specifically designed for . Like Galera, it only runs on Linux. Galera Load Balancer is developed and maintained by Codership. Documentation is available .

Galera Load Balancer is inspired by Pen, which is a generic TCP load balancer. However, since Pen is a generic TCP connection load balancer, the techniques it uses are not well-suited to the particular use case of database servers. Galera Load Balancer is optimized for this type of workload.

Several balancing policies are supported. Each node can be assigned a different weight. Nodes with a higher weight are preferred. Depending on the selected policy, other nodes can even be ignored until the preferred nodes crash.

A lightweight daemon called glbd receives the connections from clients, and it redirects them to nodes. No specific client exists for this demo: a generic TCP client, such as nc, can be used to send administrative commands and read the usage statistics.

_{This page is licensed: CC BY-SA / Gnu FDL}

Using MariaDB Replication with MariaDB Galera Cluster

MariaDB Galera Cluster provides high availability with synchronous replication, while adding asynchronous replication boosts redundancy for disaster recovery or reporting.

Using MariaDB GTIDs with MariaDB Galera Cluster

MariaDB's are very useful when used with , which is primarily what that feature was developed for. , on the other hand, was developed by Codership for all MySQL and MariaDB variants, and the initial development of the technology pre-dated MariaDB's implementation. As a side effect, (at least until ) only partially supports MariaDB's implementation.

GTID Support for Write Sets Replicated by Galera Cluster

Galera Cluster has its own that is substantially different from . However, it would still be beneficial if was able to associate a Galera Cluster write set with a that is globally unique but that is also consistent for that write set on each cluster node.

Before , did not replicate the original with the write set except in cases where the transaction was originally applied by a . Each node independently generated its own for each write set in most cases. See .

Wsrep GTID Mode

MariaDB supports .

MariaDB has a feature called wsrep GTID mode. When this mode is enabled, MariaDB uses some tricks to try to associate each Galera Cluster write set with a that is globally unique, but that is also consistent for that write set on each cluster node. These tricks work in some cases, but can still become inconsistent among cluster nodes.

Enabling Wsrep GTID Mode

Several things need to be configured for wsrep GTID mode to work, such as

needs to be set on all nodes in the cluster.
needs to be set to the same value on all nodes in a given cluster, so that each cluster node uses the same domain when assigning for Galera Cluster's write sets. When replicating between two clusters, each cluster should have this set to a different value, so that each cluster uses different domains when assigning for their write sets.
needs to be enabled on all nodes in the cluster. See .
needs to be set to the same path on all nodes in the cluster. See .

And as an extra safety measure:

should be set to a different value on all nodes in a given cluster, and each of these values should be different than the configured value. This is to prevent a node from using the same domain used for Galera Cluster's write sets when assigning for non-Galera transactions, such as DDL executed with set or DML executed with set.

If you want to avoid writes accidentialy local GTIDS, you can avoid it with = DISALLOW_LOCAL_GTID

In this case you get an error: ERROR 4165 (HY000): Galera replication not supported

You can overwrite it temporarily with set = 0;

For information on setting , see .

Known Problems with Wsrep GTID Mode

Until , there were known cases where could become inconsistent across the cluster nodes.

A known issue (fixed in ) is:

Implicitly dropped temporary tables can make GTIDs inconsistent. See and .

This does not necessarily imply that wsrep GTID mode works perfectly in all other situations. If you discover any other issues with it, please .

GTIDs for Transactions Applied by Slave Thread

If a Galera Cluster node is also a , then that node's will be applying transactions that it replicates from its replication master. If the node has set, then each transaction that the applies will also generate a Galera Cluster write set that is replicated to the rest of the nodes in the cluster.

In and earlier, the node acting as slave would apply the transaction with the original GTID that it received from the master, and the other Galera Cluster nodes would generate their own GTIDs for the transaction when they replicated the write set.

In and later, the node acting as slave will include the transaction's original Gtid_Log_Event in the replicated write set, so all nodes should associate the write set with its original GTID. See about that.

_{This page is licensed: CC BY-SA / Gnu FDL}

Using MariaDB Replication with MariaDB Galera Cluster

and can be used together. However, there are some things that have to be taken into account.

Tutorials

If you want to use and together, then the following tutorials may be useful:

Configuring a Cluster Node as a Replication Master

If a Galera Cluster node is also a , then some additional configuration may be needed.

Like with , write sets that are received by a node with are not written to the by default.

If the node is a replication master, then its replication slaves only replicate transactions that are in the binary log, so this means that the transactions that correspond to Galera Cluster write-sets would not be replicated by any replication slaves by default. If you would like a node to write its replicated write sets to the , then you will have to set . If the node has any replication slaves, then this would also allow those slaves to replicate the transactions that corresponded to those write sets.

See for more information.

Configuring a Cluster Node as a Replication Slave

If a Galera Cluster node is also a , then some additional configuration may be needed.

If the node is a replication slave, then the node's will be applying transactions that it replicates from its replication master. Transactions applied by the slave SQL thread will only generate Galera Cluster write-sets if the node has set. Therefore, in order to replicate these transactions to the rest of the nodes in the cluster, must be set.

If the node is a replication slave, then it is probably also a good idea to enable . When this is enabled, the node will restart its whenever it rejoins the cluster.

Replication Filters

Both and support , so extra caution must be taken when using all of these features together. See for more details on how MariaDB Galera Cluster interprets replication filters.

Setting server_id on Cluster Nodes

Setting the Same server_id on Each Cluster Node

It is most common to set to the same value on each node in a given cluster. Since uses a , all nodes should have the same data, so in a logical sense, a cluster can be considered in many cases a single logical server for purposes related to . The of each cluster node might even contain roughly the same transactions and if is set and if is enabled and if non-Galera transactions are not being executed on any nodes.

Setting a Different server_id on Each Cluster Node

There are cases when it might make sense to set a different value on each node in a given cluster. For example, if is set and if another cluster or a standard MariaDB Server is using to replicate transactions from each cluster node individually, then it would be required to set a different value on each node for this to work.

Keep in mind that if replication is set up in a scenario where each cluster node has a different value, and if the replication topology is set up in such a way that a cluster node can replicate the same transactions through Galera and through MariaDB replication, then you may need to configure the cluster node to ignore these transactions when setting up MariaDB replication. You can do so by setting to the server IDs of all nodes in the same cluster when executing . For example, this might be required when circular replication is set up between two separate clusters, and each cluster node has a different value, and each cluster has set.

_{This page is licensed: CC BY-SA / Gnu FDL}

Reference

Galera Cluster for MariaDB offers synchronous multi-master replication with high availability, no data loss, and simplified, consistent scaling.

WSREP Variable Details

wsrep_cluster_name

Overview

Name for the cluster.

Details

This system variable specifies the logical name of the cluster. Every Cluster Node that connects to each other must have the same logical name in order to form a component or join the Primary Component.

Parameters

Examples

Configuration

Set the cluster name using an options file:

Show Configuration

To view the current cluster name, use the statement:

wsrep_sst_method

Overview

State snapshot transfer method.

DETAILS

PARAMETERS

wsrep_sst_common

`wsrep_sst_common` Variables

The wsrep_sst_common script provides shared functionality used by various State Snapshot Transfer (SST) methods in Galera Cluster. It centralizes the handling of common configurations such as authentication credentials, SSL/TLS encryption parameters, and other security-related settings. This ensures consistent and secure communication between cluster nodes during the SST process.

The wsrep_sst_common script parses the following options:

WSREP_SST_OPT_AUTH (wsrep-sst-auth)
- Description: Defines the authentication credentials used by the State Snapshot Transfer (SST) process, typically formatted as user:password. These credentials are essential for authenticating the SST user on the donor node, ensuring that only authorized joiner nodes can initiate and receive data during the SST operation. Proper configuration of this variable is critical to maintain the security and integrity of the replication process between Galera cluster nodes.

tcert (tca)
- Description: Specifies the Certificate Authority (CA) certificate file used for SSL/TLS encryption during State Snapshot Transfers (SSTs). When encryption is enabled, this certificate allows the joining node (client) to authenticate the identity of the donor node, ensuring secure and trusted communication between them.

tcap (tcapath)
- Description: Specifies the path to a directory that contains a collection of trusted Certificate Authority (CA) certificates. Instead of providing a single CA certificate file, this option allows the use of multiple CA certificates stored in separate files within the specified directory. It is useful in environments where trust needs to be established with multiple certificate authorities.

tpem (tcert)
- Description: This variable stores the path to the TLS/SSL certificate file for the specific node. The certificate, typically in PEM format, is used by the node to authenticate itself to other nodes during secure SST operations. It is derived from the tcert option in the [sst] section.

tkey (tkey)
- Description: Represents the private key file that corresponds to the public key certificate specified by tpem. This private key is essential for decrypting data and establishing a secure connection during State Snapshot Transfer (SST). It enables the receiving node to authenticate encrypted information and participate in secure replication within the cluster.

wsrep_sst_mariabackup

`wsrep_sst_mariabackup` Variables

The wsrep_sst_mariabackup script handles the actual data transfer and processing during an SST. The variables it reads from the [sst] group control aspects of the backup format, compression, transfer mechanism, and logging.

The wsrep_sst_mariadbbackup script parses the following options:

sfmt (streamfmt)
- Default: mbstream
- Description: Defines the streaming format used by mariabackup for the SST. mbstream indicates that mariabackup will output a continuous stream of data. Other potential values (though not explicitly shown as defaults) might be related to different backup methods or tools.

tfmt (transferfmt)
- Default: socat
- Description: Specifies the transfer format or utility used to move the data stream from the donor to the joiner node. socat is a common command-line tool for data transfer, often used for setting up various network connections.

sockopt (socket options)
- Description: Allows additional socket options to be passed to the underlying network communication. This could include settings for TCP buffers, keep-alives, or other network-related tunables to optimize the transfer performance.

progress
- Description: Likely controls whether progress information about the SST is displayed or logged. Setting this could enable visual indicators or detailed log entries about the transfer's advancement.

ttime (time)
- Default: 0
- Description: Possibly a timeout value in seconds for certain operations during the SST, or a flag related to timing the transfer. A value of 0 might indicate no timeout or that timing is handled elsewhere.

cpat
- Description: Appears to be related to a "copy pattern" or specific path handling during the SST. Its exact function would depend on how the wsrep_sst_mariabackup script uses this pattern for file or directory management.

scomp (compressor)
- Description: Specifies the compression utility to be used on the data stream before transfer. Common values could include gzip, pigz, lz4, or qpress, which reduce the data size for faster transmission over the network.

sdecomp (decompressor)
- Description: Specifies the decompression utility to be used on the receiving end (joiner node) to decompress the data stream that was compressed by scomp. It should correspond to the scomp setting.

rlimit (resource limit)
- Description: Potentially sets resource limits for the mariabackup process during the SST. This could include limits on CPU usage, memory, or file descriptors, preventing the SST from consuming excessive resources and impacting the server's performance.

uextra (use-extra)
- Default: 0
- Description: A boolean flag (0 or 1) that likely indicates whether to use extra or advanced features/parameters during the SST. The specific "extra" features would be determined by the mariabackup implementation.

speciald (sst-special-dirs)
- Default: 1
- Description: A boolean flag (0 or 1) that likely controls whether mariabackup should handle special directories (e.g., innodb_log_group_home_dir, datadir) in a specific way during the SST, rather than just copying them as regular files. This is important for maintaining data consistency.

stimeout (sst-initial-timeout)
- Default: 300
- Description: Sets an initial timeout in seconds for the SST process. If the SST doesn't make progress or complete within this initial period, it might be aborted.

ssyslog (sst-syslog)
- Default: 0
- Description: A boolean flag (0 or 1) that likely controls whether SST-related messages should be logged to syslog. This can be useful for centralized logging and monitoring of Galera cluster events.

sstlogarchive (sst-log-archive)
- Default: 1
- Description: A boolean flag (0 or 1) that likely determines whether SST logs should be archived. Archiving logs helps in post-mortem analysis and troubleshooting of SST failures.

sstlogarchivedir (sst-log-archive-dir)
- Description: Specifies the directory where SST logs should be archived if sstlogarchive is enabled.

socket.ssl

Overview

Explicitly enables TLS usage by the wsrep provider.

The wsrep_provider_options system variable applies to MariaDB Enterprise Cluster, powered by Galera and to Galera Cluster available with MariaDB Community Server. This page relates specifically to the socket.ssl wsrep_provider_options.

Details

The socket.ssl option is used to specify if SSL encryption should be used.

Option Name

socket.ssl

Default Value

Dynamic

Debug

NO

Examples

Display Current Value

wsrep_provider_options define optional settings the node passes to the wsrep provider.

To display current wsrep_provider_options values:

SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options';

The expected output will display the option and the value. Options with no default value, for example SSL options, will not be displayed in the output.

Set in Configuration File

When changing a setting for a wsrep_provider_options in the config file, you must list EVERY option that is to have a value other than the default value. Options that are not explicitly listed are reset to the default value.

Options are set in the my.cnf configuration file. Use the ; delimiter to set multiple options.

The configuration file must be updated on each node. A restart to each node is needed for changes to take effect.

Use a quoted string that includes every option where you want to override the default value. Options that are not in the list will reset to their default value.

To set the option in the configuration file:

wsrep_provider_options='socket.ssl=YES;gcache.debug=YES;gcs.fc_limit=NO;socket.send_buf_size=NO;evs.keepalive_period=PT3S'

Set Dynamically

The socket.ssl option cannot be set dynamically. It can only be set in the configuration file.

Trying to change a non-dynamic option with SET results in an error:

ERROR 1210 (HY000): Incorrect arguments to SET

socket.ssl_ca

Overview

Defines the path to the SSL Certificate Authority (CA) file.

Details

The node uses the CA file to verify the signature on the certificate. You can use either an absolute path or one relative to the working directory. The file must use PEM format.

Option Name

socket.ssl_ca

Default Value

"" (an empty string)

Dynamic

Debug

NO

Examples

Display Current Value

wsrep_provider_options define optional settings the node passes to the wsrep provider.

To display current wsrep_provider_options values:

SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options';

The expected output will display the option and the value. Options with no default value, for example SSL options, will not be displayed in the output.

Set in Configuration File

Options are set in the my.cnf configuration file. Use the ; delimiter to set multiple options.

The configuration file must be updated on each node. A restart to each node is needed for changes to take effect.

Use a quoted string that includes every option where you want to override the default value. Options that are not in the list will reset to their default value.

To set the option in the configuration file:

wsrep_provider_options='socket.ssl_ca=/path/to/ca-cert.pem;gcache.debug=YES;gcs.fc_limit=NO;socket.send_buf_size=NO;evs.keepalive_period=PT3S'

Set Dynamically

The socket.ssl_ca option cannot be set dynamically. It can only be set in the configuration file.

Trying to change a non-dynamic option with SET results in an error:

ERROR 1210 (HY000): Incorrect arguments to SET

socket.ssl_cert

Overview

Defines the path to the SSL certificate.

Details

The node uses the certificate as a self-signed public key in encrypting replication traffic over SSL. You can use either an absolute path or one relative to the working directory. The file must use PEM format.

Examples

Display Current Value

wsrep_provider_options define optional settings the node passes to the wsrep provider.

To display current wsrep_provider_options values:

The expected output will display the option and the value. Options with no default value, for example SSL options, will not be displayed in the output.

Set in Configuration File

Options are set in the my.cnf configuration file. Use the ; delimiter to set multiple options.

The configuration file must be updated on each node. A restart to each node is needed for changes to take effect.

Use a quoted string that includes every option where you want to override the default value. Options that are not in the list will reset to their default value.

To set the option in the configuration file:

Set Dynamically

The socket.ssl_cert option cannot be set dynamically. It can only be set in the configuration file.

Trying to change a non-dynamic option with SET results in an error:

socket.ssl_key

Overview

Defines the path to the SSL certificate key.

Details

The node uses the certificate key, a self-signed private key, in encrypting replication traffic over SSL. You can use either an absolute path or one relative to the working directory. The file must use PEM format.

Option Name

socket.ssl_key

Maximum Value

"" (an empty string)

Dynamic

Debug

NO

Examples

Display Current Value

wsrep_provider_options define optional settings the node passes to the wsrep provider.

To display current wsrep_provider_options values:

SHOW GLOBAL VARIABLES LIKE 'wsrep_provider_options';

The expected output will display the option and the value. Options with no default value, for example SSL options, will not be displayed in the output.

Set in Configuration File

Options are set in the my.cnf configuration file. Use the ; delimiter to set multiple options.

The configuration file must be updated on each node. A restart to each node is needed for changes to take effect.

Use a quoted string that includes every option where you want to override the default value. Options that are not in the list will reset to their default value.

To set the option in the configuration file:

wsrep_provider_options='socket.ssl_key=/path/to/server-key.pem;gcache.debug=YES;gcs.fc_limit=NO;socket.send_buf_size=NO;evs.keepalive_period=PT3S'

Set Dynamically

The socket.ssl_key option cannot be set dynamically. It can only be set in the configuration file.

Trying to change a non-dynamic option with SET results in an error:

ERROR 1210 (HY000): Incorrect arguments to SET

ssl_ca

Overview

CA file in PEM format (check OpenSSL docs, implies --ssl).

Details

Parameters

Command-line

--ssl_ca=arg

Configuration file

Supported

Dynamic

Scope

Global

Data Type

VARCHAR

Product Default Value

"" (an empty string)

ssl_capath

Overview

CA directory (check OpenSSL docs, implies --ssl).

Details

Parameters

ssl_cert

Overview

X509 cert in PEM format (implies --ssl).

Details

Parameters

Command-line

--ssl_cert=arg

Configuration file

Supported

Dynamic

Scope

Global

Data Type

VARCHAR

Product Default Value

"" (an empty string)

ssl_key

Overview

X509 key in PEM format (implies --ssl).

Details

Parameters

MariaDB Galera Cluster Overview

Overview

MariaDB Enterprise Cluster, powered by Galera, is available with MariaDB Enterprise Server. MariaDB Galera Cluster is available with MariaDB Community Server.

An Introduction to Database Replication

Replication Architectures

Primary/Replica

The most common replication architecture is Primary/Replica (also known as Master/Slave). In this model:

The Primary node is the authoritative source. It is the only node that accepts write operations (e.g., INSERT, UPDATE, DELETE).
The Primary logs these changes and sends them to one or more Replica nodes.
The Replicas receive the stream of changes and apply them to their own copy of the data. Replicas are typically used for read-only queries, backups, or as a hot standby for failover.

Multi-Primary Replication

Replication Protocols: Asynchronous vs. Synchronous

Beyond the architecture, the replication protocol determines how transactions are confirmed across the cluster.

Asynchronous Replication (Lazy Replication)

Synchronous Replication (Eager Replication)

The Trade-offs of Synchronous Replication

Advantages

Synchronous replication offers several powerful advantages over its asynchronous counterpart:

High Availability: Since all nodes are fully synchronized, if one node fails, there is zero data loss. Traffic can be immediately directed to another node without complex failover procedures, as all data replicas are guaranteed to be consistent.
Read-After-Write Consistency: Synchronous replication guarantees causality. A SELECT query issued immediately after a transaction will always see the effects of that transaction, even if the query is executed on a different node in the cluster.

Disadvantages

$𝑚=𝑛×𝑜×𝑡$

What this means that any increase in the number of nodes leads to an exponential growth in the transaction response times and in the probability of conflicts and deadlock rates.

Galera's Solution: Modern Synchronous Replication

Galera Cluster solves the traditional problems of synchronous replication by using a modern, certification-based approach built on several key innovations:

Group Communication: A robust messaging layer ensures that information is delivered to all nodes reliably and in the correct order, forming a solid foundation for data consistency.
Write-Set Replication: Instead of coordinating on every individual operation, database changes (writes) are grouped into a single package called a "write-set." This write-set is replicated as a single message, avoiding the high overhead of traditional two-phase commit.
Optimistic Execution: Transactions are first executed optimistically on a local node. The resulting write-set is then broadcast to the cluster for a fast, parallel certification process. If it passes certification (meaning no conflicts), it is committed on all nodes.
Transaction Reordering: This clever technique allows for the reordering of non-conflicting transactions before they are committed, which significantly increases parallelism and reduces the rate of transaction rollbacks.

How it Works

MariaDB Enterprise Cluster is only available for Linux operating systems.

Architecture

There are a few things to consider when planning the hardware, virtual machines, or containers for MariaDB Enterprise Cluster.

When MariaDB Enterprise Servers start in a cluster:

Each Server attempts to establish network connectivity with the other Servers in the cluster
Groups of connected Servers form a component
When a Server establishes network connectivity with the Primary Component, it synchronizes its local database with that of the cluster
As a member of the Primary Component, the Server becomes operational — able to accept read and write queries from clients
During startup, the Primary Component is the Server bootstrapped to run as the Primary Component. Once the cluster is online, the Primary Component is any combination of Servers which includes a minimum of more than half the total number of Servers.
A Server or group of Servers that loses network connectivity with the majority of the cluster becomes non-operational.

Each Server requires the minimum amount of disk space needed to store the entire database. The upper storage limit for MariaDB Enterprise Cluster is that of the smallest disk in use.
Each switch in use should have an odd number of Servers above three.
In a cluster that spans multiple switches, each data center in use should have an odd number of switches above three.
In a cluster that spans multiple data centers, use an odd number of data centers above three.
Each data center in use should have at least one Server dedicated to backup operations. This can be another cluster node or a separate Replica Server kept in sync using MariaDB Replication.

Cluster Configuration

[mariadb]

# General Configuration
bind_address             = 0.0.0.0
innodb_autoinc_lock_mode = 2

# Cluster Configuration
wsrep_cluster_name    = "accounting_cluster"
wsrep_cluster_address = "gcomm://192.0.2.1,192.0.2.2,192.0.2.3"

# wsrep Provider
wsrep_provider = /usr/lib/galera/libgalera_enterprise_smm.so
wsrep_provider_options = "evs.suspect_timeout=PT10S"

Additional information on system variables is available in the Reference chapter.

General Configuration

The innodb_autoinc_lock_mode system variable must be set to a value of 2 to enable interleaved lock mode. MariaDB Enterprise Cluster does not support other lock modes.

Ensure also that the bind_address system variable is properly set to allow MariaDB Enterprise Server to listen for TCP/IP connections:

bind_address             = 0.0.0.0
innodb_autoinc_lock_mode = 2

Cluster Name and Address

Using the wsrep_cluster_address system variable, you can define the back-end protocol (always gcomm) and comma-separated list of the IP addresses or domain names of the other nodes in the cluster.

wsrep_cluster_name    = "accounting_cluster"
wsrep_cluster_address = "gcomm://192.0.2.1,192.0.2.2,192.0.2.3"

It is best practice to list all nodes on this system variable, as this is the list the node searches when attempting to reestablish network connectivity with the primary component.

Galera Replicator Plugin

To enable MariaDB Enterprise Cluster, use the libgalera_enterprise_smm.so library:

wsrep_provider = /usr/lib/galera/libgalera_enterprise_smm.so

MariaDB Enterprise Server use the older community-release of the Galera 3 plugin. This is set using the libgalera_smm.so library:

wsrep_provider = /usr/lib/galera/libgalera_smm.so

wsrep_provider_options = "evs.suspect_timeout=PT10S"

Additional information is available in the Reference chapter.

Cluster Replication

MariaDB Enterprise Cluster implements a multi-primary replication solution.

When you write to a table on a node, the node collects the write into a write-set transaction, which it then replicates to the other nodes in the cluster.

Quorum

When a component forms that has less than half the nodes in the cluster, it becomes non-operational, since it believes there is a running Primary Component to which it has lost network connectivity.

These quorum requirements, combined with the requisite number of odd nodes, avoid a split brain situation, or one in which two separate components believe they are each the Primary Component.

Dynamically Bootstrapping the Cluster

In cases where the cluster goes down and your nodes become non-operational, you can dynamically bootstrap the cluster.

First, find the most up-to-date node (that is, the node with the highest value for the wsrep_last_committed status variable):

SHOW STATUS LIKE 'wsrep_last_committed';

Once you determine the node with the most recent transaction, you can designate it as the Primary Component by running the following on it:

SET GLOBAL wsrep_provider_options="pc.bootstrap=YES";

State Transfers

In a state transfer, the node connects to another node in the cluster and attempts to bring its local database back in sync with the cluster. There are two types of state transfers:

Incremental State Transfer (IST)
State Snapshot Transfer (SST)

IST's provide the best performance for state transfers and the size of the GCache may need adjustment to facilitate their use.

Flow Control

MariaDB Enterprise Server uses Flow Control to sometimes throttle transactions and ensure that all nodes work equitably.

Eviction

A node or nodes will be removed or evicted from a cluster if it becomes non-responsive.

In MariaDB Enterprise Cluster, each node monitors network connectivity and response times from every other node. MariaDB Enterprise Cluster evaluates network performance using the EVS Protocol.

If the number of entries for a node in the delayed list exceeds a threshold established for the cluster, the EVS Protocol evicts the node from the cluster.

Evicted nodes become non-operational components. They cannot rejoin the cluster until you restart MariaDB Enterprise Server.

Streaming Replication

Initiate Streaming Replication

To initiate streaming replication, set the wsrep_trx_fragment_unit and wsrep_trx_fragment_size system variables. You can set the unit to BYTES, ROWS, or STATEMENTS:

SET SESSION wsrep_trx_fragment_unit='STATEMENTS';

SET SESSION wsrep_trx_fragment_size=5;

Then, run your transaction.

SET SESSION wsrep_trx_fragment_size=0;

Galera Arbitrator

Bear in mind that the Galera Arbitrator is a separate package, galera-arbitrator-4, which is not installed by default with MariaDB Enterprise Server.

Scale-out

Once the MariaDB Enterprise Server reports itself as being in sync with the cluster, MariaDB MaxScale can begin including it in the load distribution for the cluster.

Failover

MariaDB Enterprise Cluster does not provide failover capabilities on its own. MariaDB MaxScale is used to route client connections to MariaDB Enterprise Server.

Unlike a traditional load balancer, MariaDB MaxScale is aware of changes in the node and cluster states.

Backups

Encryption

MariaDB Enterprise Server supports data-at-rest encryption to secure data on disk, and data-in-transit encryption to secure data on the network.

Data-in-Transit Encryption

MariaDB Enterprise Server 10.5 and earlier support encrypting Galera replication and SST traffic through wsrep Provider options.

TLS encryption is only available when used by all nodes in the cluster.

Enabling GCache Encryption

[mariadb]
...
# Controls Binary Log, Relay Log, and GCache Encryption
encrypt_binlog=ON

Disabling GCache Encryption

[mariadb]
...
# Controls Binary Log, Relay Log, and GCache Encryption
encrypt_binlog=OFF

wsrep_provider_options

The following options can be set as part of the Galera wsrep_provider_options variable. Dynamic options can be changed while the server is running.

Options need to be provided as a semicolon (;) separated list on a single line. Options that are not explicitly set are set to their default value.

Note that before Galera 3, the repl tag was named replicator.

`base_dir`

Description: Specifies the data directory

`base_host`

Description: For internal use. Should not be manually set.
Default: 127.0.0.1 (detected network address)

`base_port`

Description: For internal use. Should not be manually set.
Default: 4567

`cert.log_conflicts`

Description: Certification failure log details.
Dynamic: Yes
Default: no

`cert.optimistic_pa`

Description: Controls parallel application of actions on the replica. If set, the full range of parallelization as determined by the certification algorithm is permitted. If not set, the parallel applying window will not exceed that seen on the primary, and applying will start no sooner than after all actions it has seen on the master are committed.
Dynamic: Yes
Default: yes

`debug`

Description: Enable debugging.
Dynamic: Yes
Default: no

`evs.auto_evict`

Description: Number of entries the node permits for a given delayed node before triggering the Auto Eviction protocol. An entry is added to a delayed list for each delayed response from a node. If set to 0, the default, the Auto Eviction protocol is disabled for this node. See Auto Eviction for more.
Dynamic: No
Default: 0

`evs.causal_keepalive_period`

Description: Used by the developers only, and not manually serviceable.
Dynamic: No
Default: The evs.keepalive_period.

`evs.debug_log_mask`

Description: Controls EVS debug logging. Only effective when wsrep_debug is on.
Dynamic: Yes
Default: 0x1

`evs.delay_margin`

Description: Time that response times can be delayed before this node adds an entry to the delayed list. See evs.auto_evict. Must be set to a higher value than the round-trip delay time between nodes.
Dynamic: No
Default: PT1S

`evs.delayed_keep_period`

Description: Time that this node requires a previously delayed node to remain responsive before being removed from the delayed list. See evs.auto_evict.
Dynamic: No
Default: PT30S

`evs.evict`

Description: When set to the gcomm UUID of a node, that node is evicted from the cluster. When set to an empty string, the eviction list is cleared on the node where it is set. See evs.auto_evict.
Dynamic: No
Default: Empty string

`evs.inactive_check_period`

Description: Frequency of checks for peer inactivity (looking for nodes with delayed responses), after which nodes may be added to the delayed list, and later evicted.
Dynamic: No
Default: PT0.5S

`evs.inactive_timeout`

Description: Time limit that a node can be inactive before being pronounced as dead.
Dynamic: No
Default: PT15S

`evs.info_log_mask`

Description: Controls extra EVS info logging. Bits:
- 0x1 – extra view change information
- 0x2 – extra state change information
- 0x4 – statistics
- 0x8 – profiling (only available in builds with profiling enabled)
Dynamic: No
Default: 0

`evs.install_timeout`

Description: Timeout on waits for install message acknowledgments. Replaces evs.consensus_timeout.
Dynamic: Yes
Default: PT7.5S

`evs.join_retrans_period`

Description: Time period for how often retransmission of EVS join messages when forming cluster membership should occur.
Dynamic: Yes
Default: PT1S

`evs.keepalive_period`

Description: How often keepalive signals should be transmitted when there's no other traffic.
Dynamic: Yes
Default: PT1S

`evs.max_install_timeouts`

Description: Number of membership install rounds to attempt before timing out. The total rounds will be this value plus two.
Dynamic: No
Default: 3

`evs.send_window`

Description: Maximum number of packets that can be replicated at a time, Must be more than evs.user_send_window, which applies to data packets only (double is recommended). In WAN environments can be set much higher than the default, for example 512.
Dynamic: Yes
Default: 4

`evs.stats_report_period`

Description: Reporting period for EVS statistics.
Dynamic: No
Default: PT1M

`evs.suspect_timeout`

Description: A node will be suspected to be dead after this period of inactivity. If all nodes agree, the node is dropped from the cluster before evs.inactive_timeout is reached.
Dynamic: No
Default: PT5S

`evs.use_aggregate`

Description: If set to true (the default), small packets will be aggregated into one where possible.
Dynamic: No
Default: true

`evs.user_send_window`

Description: Maximum number of data packets that can be replicated at a time. Must be smaller than evs.send_window (half is recommended). In WAN environments can be set much higher than the default, for example 512.
Dynamic: Yes
Default: 2

`evs.version`

Description: EVS protocol version. Defaults to 0 for backward compatibility. Certain EVS features (e.g. auto eviction) require more recent versions.
Dynamic: No
Default: 0

`evs.view_forget_timeout`

Description: Time after which past views will be dropped from the view history.
Dynamic: No
Default: P1D

`gcache.dir`

Description: Directory where GCache files are placed.
Dynamic: No
Default: The working directory

`gcache.keep_pages_size`

Description: Total size of the page storage pages for caching. One page is always present if only page storage is enabled.
Dynamic: No
Default: 0

`gcache.mem_size`

Description: Maximum size of size of the malloc() store for setups that have spare RAM.
Dynamic: No
Default: 0

`gcache.name`

Description: Gcache ring buffer storage file name. By default placed in the working directory, changing to another location or partition can reduce disk IO.
Dynamic: No
Default: ./galera.cache

`gcache.page_size`

Description: Size of the page storage page files. These are prefixed by gcache.page. Can be set to as large as the disk can handle.
Dynamic: No
Default: 128M

`gcache.recover`

Description: Whether or not gcache recovery takes place when the node starts up. If it is possible to recover gcache, the node can then provide IST to other joining nodes, which assists when the whole cluster is restarted.
Dynamic: No
Default: no
Introduced: , ,

`gcache.size`

Description: Gcache ring buffer storage size (the space the node uses for caching write sets), preallocated on startup.
Dynamic: No
Default: 128M

`gcomm.thread_prio`

fifo or rr real-time scheduling policies requires mariadb service permissions at the OS level.

Description: Gcomm thread policy and priority (in the format policy:priority. Priority is an integer, while policy can be one of:
- fifo: First-in, first-out scheduling. Always preempt other, batch or idle threads and can only be preempted by other fifo threads of a higher priority or blocked by an I/O request.
- rr: Round-robin scheduling. Always preempt other, batch or idle threads. Runs for a fixed period of time after which the thread is stopped and moved to the end of the list, being replaced by another round-robin thread with the same priority. Otherwise runs until preempted by other rr threads of a higher priority or blocked by an I/O request.
- other: Default scheduling on Linux. Threads run until preempted by a thread of a higher priority or a superior scheduling designation, or blocked by an I/O request.
Permissions: Using the fifo or rr real-time scheduling policies requires granting the mariadb service the necessary permissions at the OS level. On systemd-based distributions, this is done by adjusting the resource limits for the service.
The recommended method is to create a systemd override file:
1. Open the MariaDB service unit for editing:
  sudo systemctl edit mariadb
2. Add the following content to the file. This grants the service the ability to set real-time priorities:
  [Service] LimitRTPRIO=infinity
3. Save the file and exit the editor.
4. Reload the systemd daemon and restart the MariaDB service to apply the changes:
  sudo systemctl daemon-reload sudo systemctl restart mariadb
Dynamic: No
Default: Empty string

`gcs.fc_debug`

Description: If set to a value greater than zero (the default), debug statistics about SST flow control will be posted each timegcs.fc_master_slave after the specified number of writesets.
Dynamic: No
Default: 0

`gcs.fc_factor`

Description:Fraction below gcs.fc_limit which if the recv queue drops below, replication resumes.
Dynamic: Yes
Default: 1.0

`gcs.fc_limit`

Description: If the recv queue exceeds this many writesets, replication is paused. Can increase greatly in master-slave setups. Replication will resume again according to the gcs.fc_factor setting.
Dynamic: Yes
Default: 16

`gcs.fc_master_slave`

Description: Whether to assume that the cluster only contains one master. Deprecated since Galera 4.10 (, , , , ) - see gcs.fc_single_primary
Dynamic: No
Default: no

`gcs.fc_single_primary`

Description: Defines whether there is more than one source of replication. As the number of nodes in the cluster grows, the larger the calculated gcs.fc_limit gets. At the same time, the number of writes from the nodes increases. When this parameter value is set to NO (multi-primary), the gcs.fc_limit parameter is dynamically modified to give more margin for each node to be a bit further behind applying writes. The gcs.fc_limit parameter is modified by the square root of the cluster size, that is, in a four-node cluster it is two times higher than the base value. This is done to compensate for the increasing replication rate noise.
Dynamic: No
Default: no

`gcs.max_packet_size`

Description: Maximum packet size, after which writesets become fragmented.
Dynamic: No
Default: 64500

`gcs.max_throttle`

Description: How much we can throttle replication rate during state transfer (to avoid running out of memory). Set it to 0.0 if stopping replication is acceptable for the sake of completing state transfer.
Dynamic: No
Default: 0.25

`gcs.recv_q_hard_limit`

Description: Maximum size of the recv queue. If exceeded, the server aborts. Half of available RAM plus swap is a recommended size.
Dynamic: No
Default: LLONG_MAX

`gcs.recv_q_soft_limit`

Description: Fraction of gcs.recv_q_hard_limit after which replication rate is throttled. The rate of throttling increases linearly from zero (the regular, varying rate of replication) at and below csrecv_q_soft_limit to one (full throttling) at gcs.recv_q_hard_limit
Dynamic: No
Default: 0.25

`gcs.sync_donor`

Description: Whether or not the rest of the cluster should stay in sync with the donor. If set to YES (NO is default), if the donor is blocked by state transfer, the whole cluster is also blocked.
Dynamic: No
Default: no

`gmcast.listen_addr`

Description: Address Galera listens for connections from other nodes. Can be used to override the default port to listen, which is obtained from the connection address.
Dynamic: No
Default: tcp://0.0.0.0:4567

`gmcast.mcast_addr`

Description: Not set by default, but if set, UDP multicast will be used for replication. Must be identical on all nodes.For example, gmcast.mcast_addr=239.192.0.11
Dynamic: No
Default: None

`gmcast.mcast_ttl`

Description: Multicast packet TTL (time to live) value.
Dynamic: No
Default: 1

`gmcast.peer_timeout`

Description: Connection timeout for initiating message relaying.
Dynamic: No
Default: PT3S

`gmcast.segment`

Description: Defines the segment to which the node belongs. By default, all nodes are placed in the same segment (0). Usually, you would place all nodes in the same datacenter in the same segment. Galera protocol traffic is only redirected to one node in each segment, and then relayed to other nodes in that same segment, which saves cross-datacenter network traffic at the expense of some extra latency. State transfers are also, preferably but not exclusively, taken from the same segment. If there are no nodes available in the same segment, state transfer will be taken from a node in another segment.
Dynamic: No
Default: 0
Range: 0 to 255

`gmcast.time_wait`

Description: Waiting time before allowing a peer that was declared outside of the stable view to reconnect.
Dynamic: No
Default: PT5S

`gmcast.version`

Description: Deprecated option. Gmcast version.
Dynamic: No
Default: 0

`ist.recv_addr`

Description: Address for listening for Incremental State Transfer.
Dynamic: No
Default::<port+1> from wsrep_node_address

`ist.recv_bind`

Description:
Dynamic: No
Default: Empty string
- Introduced: , ,

`pc.announce_timeout`

Description: Period of time for which cluster joining announcements are sent every 1/2 second.
Dynamic: No
Default: PT3S

`pc.checksum`

Description: For debug purposes, by default false (true in earlier releases), indicates whether to checksum replicated messages on PC level. Safe to turn off.
Dynamic: No
Default: false

`pc.ignore_quorum`

Description: Whether to ignore quorum calculations, for example when a master splits from several slaves, it will remain in operation if set to true (false is default). Use with care however, as in master-slave setups, slaves will not automatically reconnect to the master if set.
Dynamic: Yes
Default: false

`pc.ignore_sb`

Description: Whether to permit updates to be processed even in the case of split brain (when a node is disconnected from its remaining peers). Safe in master-slave setups, but could lead to data inconsistency in a multi-master setup.
Dynamic: Yes
Default: false

`pc.linger`

Description: Time that the PC protocol waits for EVS termination.
Dynamic: No
Default: PT20S

`pc.npvo`

Description: If set to true (false is default), when there are primary component conficts, the most recent component will override the older.
Dynamic: No
Default: false

`pc.recovery`

Description: If set to true (the default), the Primary Component state is stored on disk and in the case of a full cluster crash (e.g power outages), automatic recovery is then possible. Subsequent graceful full cluster restarts will require explicit bootstrapping for a new Primary Component.
Dynamic: No
Default: true

`pc.version`

Description: Deprecated option. PC protocol version.
Dynamic: No
Default: 0

`pc.wait_prim`

Description: When set to true, the default, the node will wait for a primary component for the period of time specified by pc.wait_prim_timeout. Used to bring up non-primary components and make them primary using pc.bootstrap.
Dynamic: No
Default: true

`pc.wait_prim_timeout`

Description: Ttime to wait for a primary component. See pc.wait_prim.
Dynamic: No
Default: PT30S

`pc.weight`

Description: Node weight, used for quorum calculation. See the Codership article Weighted Quorum.
Dynamic: Yes
Default: 1

`protonet.backend`

Description: Deprecated option. Transport backend to use. Only ASIO is supported currently.
Dynamic: No
Default: asio

`protonet.version`

Description: Deprecated option. Protonet version.
Dynamic: No
Default: 0

`repl.causal_read_timeout`

Description: Timeout period for causal reads.
Dynamic: Yes
Default: PT30S

`repl.commit_order`

Description: Whether or not out-of-order committing is permitted, and under what conditions. By default it is not permitted, but setting this can improve parallel performance.
- 0 BYPASS: No commit order monitoring is done (useful for measuring the performance penalty).
- 1 OOOC: Out-of-order committing is permitted for all transactions.
- 2 LOCAL_OOOC: Out-of-order committing is permitted for local transactions only.
- 3 NO_OOOC: Out-of-order committing is not permitted at all.
Dynamic: No
Default: 3

`repl.key_format`

Description: Format for key replication. Can be one of:
- FLAT8 - shorter key with a higher probability of false positives when matching
- FLAT16 - longer key with a lower probability of false positives when matching
- FLAT8A - shorter key with a higher probability of false positives when matching, includes annotations for debug purposes
- FLAT16A - longer key with a lower probability of false positives when matching, includes annotations for debug purposes
Dynamic: Yes
Default: FLAT8

`repl.max_ws_size`

Description:
Dynamic:
Default: 2147483647

`repl.proto_max`

Description:
Dynamic:
Default: 9

`socket.checksum`

Description: Method used for generating checksum. Note: If Galera 25.2.x and 25.3.x are both being used in the cluster, MariaDB with Galera 25.3.x must be started with wsrep_provider_options='socket.checksum=1' in order to make it backward compatible with Galera v2. Galera wsrep providers other than 25.3.x or 25.2.x are not supported.
Dynamic: No
Default: 2

`socket.dynamic`

Description: Allow both encrypted and unencrypted connections between nodes. Typically this should be set to false (the default), when set to true encrypted connections will still be preferred, but will fall back to unencrypted connections when encryption is not possible, e.g. not enabled on all nodes yet. Needs to be true on all nodes when wanting to enable or disable encryption via a rolling restart. As this can't be changed at runtime a rolling restart to enable or disable encryption may need three restarts per node in total: one to enable socket.dynamic on each node, one to change the actual encryption settings on each node, and a final round to change socket.dynamic back to false.
Dynamic: No
Default: false
Introduced: , ,

`socket.recv_buf_size`

Description: Size in bytes of the receive buffer used on the network sockets between nodes, passed on to the kernel via the SO_RCVBUF socket option.
Dynamic: No
Default:
- = , , : Auto
- < : , : 212992

`socket.send_buf_size`

Description: Size in bytes of the send buffer used on the network sockets between nodes, passed on to the kernel via the SO_SNDBUF socket option.
Dynamic: No
Default:: Auto
Introduced: , ,

`socket.ssl`

Description: Explicitly enables TLS usage by the wsrep Provider.
Dynamic: No
Default: NO

`socket.ssl_ca`

Description: Path to Certificate Authority (CA) file. Implicitly enables the socket.ssl option.
Dynamic: No

`socket.ssl_cert`

Description: Path to TLS certificate. Implicitly enables the socket.ssl option.
Dynamic: No

`socket.ssl_cipher`

Description: TLS cipher to use. Implicitly enables the socket.ssl option. Since defaults to the value of the system variable.
Dynamic: No
Default: system default, before defaults to AES128-SHA.

`socket.ssl_compression`

Description: Compression to use on TLS connections. Implicitly enables the socket.ssl option.
Dynamic: No

`socket.ssl_key`

Description: Path to TLS key file. Implicitly enables the socket.ssl option.
Dynamic: No

`socket.ssl_password_file`

Description: Path to password file to use in TLS connections. Implicitly enables the socket.ssl option.
Dynamic: No

Galera Cluster System Variables

/This page documents system variables related to Galera Cluster. For options that are not system variables, see .

See for a complete list of system variables and instructions on setting them.

Also see the .

`wsrep_allowlist`

Description:
- Allowed IP addresses, comma delimited.
- Note that setting gmcast.listen_addr=tcp://[::]:4567 on a dual-stack system (eg. Linux with net.ipv6.bindv6only = 0), IPv4 addresses need to allowlisted using the IPv4-mapped IPv6 address (eg. ::ffff:1.2.3.4).
Commandline: --wsrep-allowlist=value1[,value2...]
Scope: Global
Dynamic: No
Data Type: String
Default Value: None
Introduced:

`wsrep_auto_increment_control`

Description: If set to 1 (the default), will automatically adjust the and variables according to the size of the cluster, and when the cluster size changes. This avoids replication conflicts due to . In a primary-replica environment, can be set to OFF.
Commandline: --wsrep-auto-increment-control[={0|1}]
Scope: Global
Dynamic: Yes
Data Type: Boolean
Default Value: ON

`wsrep_causal_reads`

Description: If set to ON (OFF is default), enforces characteristics across the cluster. In the case that a primary applies an event more quickly than a replica, the two could briefly be out-of-sync. With this variable set to ON, the replica will wait for the event to be applied before processing further queries. Setting to ON also results in larger read latencies. Deprecated by wsrep_sync_wait=1.
Commandline: --wsrep-causal-reads[={0|1}]
Scope: Session
Dynamic: Yes
Data Type: Boolean
Default Value: OFF
Removed:

`wsrep_certificate_expiration_hours_warning`

`wsrep_certification_rules`

Description: Certification rules to use in the cluster. Possible values are:
- strict: Stricter rules that could result in more certification failures. For example with foreign keys, certification failure could result if different nodes receive non-conflicting insertions at about the same time that point to the same row in a parent table
- optimized: relaxed rules that allow more concurrency and cause less certification failures.
Commandline: --wsrep-certifcation-rules
Scope: Global
Dynamic: Yes
Data Type: Enumeration
Default Value: strict
Valid Values: strict, optimized

`wsrep_certify_nonPK`

Description: When set to ON (the default), Galera will still certify transactions for tables with no . However, this can still cause undefined behavior in some circumstances. It is recommended to define primary keys for every InnoDB table when using Galera.
Commandline: --wsrep-certify-nonPK[={0|1}]
Scope: Global
Dynamic: Yes
Data Type: Boolean
Default Value: ON

`wsrep_cluster_address`

Description: The addresses of cluster nodes to connect to when starting up.
- Good practice is to specify all possible cluster nodes, in the form gcomm://<node1 or ip:port>,<node2 or ip2:port>,<node3 or ip3:port>.
- Specifying an empty ip (gcomm://) will cause the node to start a new cluster (which should not be done in the my.cnf file, as after each restart the server will not rejoin the current cluster).
- The variable can be changed at runtime in some configurations, and will result in the node closing the connection to any current cluster, and connecting to the new address.
- If specifying a port, note that this is the Galera port, not the MariaDB port.
- For example:
  - gcomm://192.168.0.1,192.168.0.2,192.168.0.3
  - gcomm://192.168.0.1:1234,192.168.0.2:1234,192.168.0.3:1234?gmcast.listen_addr=tcp://0.0.0.0:1234
- See also gmcast.listen_addr
Commandline: --wsrep-cluster-address=value
Scope: Global
Dynamic: No
Data Type: String

`wsrep_cluster_name`

Description: The name of the cluster. Nodes cannot connect to clusters with a different name, so needs to be identical on all nodes in the same cluster. The variable can be set dynamically, but note that doing so may be unsafe and cause an outage, and that the wsrep provider is unloaded and loaded.
Commandline: --wsrep-cluster-name=value
Scope: Global
Dynamic: Yes
Data Type: String
Default Value: my_wsrep_cluster

`wsrep_convert_LOCK_to_trx`

Description: Converts / statements to and . Used mainly for getting older applications to work with a multi-primary setup, use carefully, as can result in extremely large writesets.
Commandline: --wsrep-convert-LOCK-to-trx[={0|1}]
Scope: Global
Dynamic: Yes
Data Type: Boolean
Default Value: OFF

`wsrep_data_home_dir`

Description: Directory where wsrep provider will store its internal files.
Commandline: --wsrep-data-home-dir=value
Scope: Global
Dynamic: No
Data Type: String
Default Value: The variable value.

`wsrep_dbug_option`

Description: Unused. The mechanism to pass the DBUG options to the wsrep provider hasn't been implemented.
Commandline: --wsrep-dbug-option=value
Scope: Global
Dynamic: Yes
Data Type: String

`wsrep_debug`

Description: WSREP debug level logging.
- Before , DDL logging was only logged on the originating node. From , it is logged on other nodes as well.

It is an enum. Valid values are:0: NONE: Off (default)1: SERVER: MariaDB server code contains WSREP_DEBUG log writes, and these will be added to server error log2: TRANSACTION: Logging from wsrep-lib transaction is added to the error log3: STREAMING: Logging from streaming transactions in wsrep-lib is added to the error log4: CLIENT: Logging from wsrep-lib client state is added to the error log.

Commandline:
- --wsrep-debug[={NONE|SERVER|TRANSACTION|STREAMING|CLIENT}]
Scope: Global
Dynamic: Yes
Data Type: Enumeration
Default Value: NONE
Valid Values: NONE, SERVER, TRANSACTION, STREAMING, CLIENT

`wsrep_desync`

Description: When a node receives more write-sets than it can apply, the transactions are placed in a received queue. If the node's received queue has too many write-sets waiting to be applied (as defined by the gcs.fc_limit WSREP provider option), then the node would usually engage Flow Control. However, when this option is set to ON, Flow Control will be disabled for the desynced node. The desynced node works through the received queue until it reaches a more manageable size. The desynced node continues to receive write-sets from the other nodes in the cluster. The other nodes in the cluster do not wait for the desynced node to catch up, so the desynced node can fall even further behind the other nodes in the cluster. You can check if a node is desynced by checking if the wsrep_local_state_comment status variable is equal to Donor/Desynced.
Commandline: --wsrep-desync[={0|1}]
Scope: Global
Dynamic: Yes
Data Type: Boolean
Default Value: OFF

`wsrep_dirty_reads`

Description: By default, when not synchronized with the group (wsrep_ready=OFF) a node will reject all queries other than SET and SHOW. If wsrep_dirty_reads is set to 1, queries which do not change data, like SELECT queries (dirty reads), creating of prepare statement, etc. will be accepted by the node.
Commandline: --wsrep-dirty-reads[={0|1}]
Scope: Global,Session
Dynamic: Yes
Data Type: Boolean
Default Value: OFF
Valid Values: ON, OFF

`wsrep_drupal_282555_workaround`

Description: If set to ON, a workaround for Drupal/MySQL/InnoDB bug #282555 is enabled. This is a bug where, in some cases, when inserting a DEFAULT value into an column, a duplicate key error may be returned.
Commandline: --wsrep-drupal-282555-workaround[={0|1}]
Scope: Global
Dynamic: Yes
Data Type: Boolean
Default Value: OFF

`wsrep_forced_binlog_format`

Description: A that will override any session binlog format settings.
Commandline: --wsrep-forced-binlog-format=value
Scope: Global
Dynamic: Yes
Default Value: NONE
Data Type: Enum
Valid Values: STATEMENT, ROW, MIXED or NONE (which resets the forced binlog format state).

`wsrep_gtid_domain_id`

Description: This system variable defines the domain ID that is used for wsrep GTID mode.
- When wsrep_gtid_mode is set to ON, wsrep_gtid_domain_id is used in place of for all Galera Cluster write sets.
- When wsrep_gtid_mode is set to OFF, wsrep_gtid_domain_id is simply ignored to allow for backward compatibility.
- There are some additional requirements that need to be met in order for this mode to generate consistent . For more information, see Using MariaDB GTIDs with MariaDB Galera Cluster.
Commandline: --wsrep-gtid-domain-id=#
Scope: Global
Dynamic: Yes
Data Type: numeric
Default Value: 0
Range: 0 to 4294967295

`wsrep_gtid_mode`

Description: Wsrep GTID mode attempts to keep consistent for Galera Cluster write sets on all cluster nodes. state is initially copied to a joiner node during an SST. If you are planning to use Galera Cluster with , then wsrep GTID mode can be helpful.
- When wsrep_gtid_mode is set to ON, wsrep_gtid_domain_id is used in place of for all Galera Cluster write sets.
- When wsrep_gtid_mode is set to OFF, wsrep_gtid_domain_id is simply ignored to allow for backward compatibility.
- There are some additional requirements that need to be met in order for this mode to generate consistent . For more information, see Using MariaDB GTIDs with MariaDB Galera Cluster.
Commandline: --wsrep-gtid-mode[={0|1}]
Scope: Global
Dynamic: Yes
Data Type: boolean
Default Value: OFF

`wsrep_gtid_seq_no`

Description: Internal server usage, manually set WSREP GTID seqno.
Commandline: None
Scope: Session only
Dynamic: Yes
Data Type: numeric
Range: 0 to 18446744073709551615
Introduced:

`wsrep_ignore_apply_errors`

Description: Bitmask determining whether errors are ignored, or reported back to the provider.
- 0: No errors are skipped.
- 1: Ignore some DDL errors (DROP DATABASE, DROP TABLE, DROP INDEX, ALTER TABLE).
- 2: Skip DML errors (Only ignores DELETE errors).
- 4: Ignore all DDL errors.
Commandline: --wsrep-ignore-apply-errors
Scope: Global
Dynamic: Yes
Data Type: Numeric
Default Value: 7
Range: 0 to 7

`wsrep_load_data_splitting`

Description: If set to ON, supports big data files by introducing transaction splitting. The setting has been deprecated in Galera 4, and defaults to OFF
Commandline: --wsrep-load-data-splitting[={0|1}]
Scope: Global
Dynamic: Yes
Data Type: Boolean
Default Value: OFF
Deprecated:
Removed:

`wsrep_log_conflicts`

Description: If set to ON (OFF is default), details of conflicting MDL as well as InnoDB locks in the cluster will be logged.
Commandline: --wsrep-log-conflicts[={0|1}]
Scope: Global
Dynamic: Yes
Data Type: Boolean
Default Value: OFF

`wsrep_max_ws_rows`

Description: Maximum permitted number of rows per writeset. The support for this variable has been added and in order to be backward compatible the default value has been changed to 0, which essentially allows writesets to be any size.
Commandline: --wsrep-max-ws-rows=#
Scope: Global
Dynamic: Yes
Data Type: Numeric
Default Value:
- 0
Range: 0 to 1048576

`wsrep_max_ws_size`

Description: Maximum permitted size in bytes per write set. Writesets exceeding 2GB will be rejected.
Commandline: --wsrep-max-ws-size=#
Scope: Global
Dynamic: Yes
Data Type: Numeric
Default Value:
- 2147483647 (2GB)
Range: 1024 to 2147483647

`wsrep_mode`

Description: Turns on WSREP features which are not part of default behavior.
- BINLOG_ROW_FORMAT_ONLY: Only ROW is supported.
- DISALLOW_LOCAL_GTID: Nodes can have GTIDs for local transactions in a number of scenarios. If DISALLOW_LOCAL_GTID is set, these operations produce an error ERROR HY000: Galera replication not supported. Scenarios include:
  - A DDL statement is executed with wsrep_OSU_method=RSU set.
  - A DML statement writes to a non-InnoDB table.
  - A DML statement writes to an InnoDB table with wsrep_on=OFF set.
- REPLICATE_ARIA: Whether or not DML updates for tables will be replicated. This functionality is experimental and should not be relied upon in production systems.
- REPLICATE_MYISAM: Whether or not DML updates for a tables will be replicated. This functionality is experimental and should not be relied upon in production systems.
- REQUIRED_PRIMARY_KEY: Table should have PRIMARY KEY defined.
- STRICT_REPLICATION: Same as the old wsrep_strict_ddl setting.
Commandline: --wsrep-mode=value
Scope: Global
Dynamic: Yes
Data Type: Enumeration
Default Value: (Empty)
Valid Values: BINLOG_ROW_FORMAT_ONLY, DISALLOW_LOCAL_GTID, REQUIRED_PRIMARY_KEY, REPLICATE_ARIA, REPLICATE_MYISAM and STRICT_REPLICATION
Introduced:

`wsrep_mysql_replication_bundle`

Description: Determines the number of replication events that are grouped together. Experimental implementation aimed to assist with bottlenecks when a single replica faces a large commit time delay. If set to 0 (the default), there is no grouping.
Commandline: --wsrep-mysql-replication-bundle=#
Scope: Global
Dynamic: No
Data Type: Numeric
Default Value: 0
Range: 0 to 1000

`wsrep_node_address`

Description: Specifies the node's network address, in the format ip address[:port]. It supports IPv6. The default behavior is for the node to pull the address of the first network interface on the system and the default Galera port. This autoguessing can be unreliable, particularly in the following cases:
- cloud deployments
- container deployments
- servers with multiple network interfaces.
- servers running multiple nodes.
- network address translation (NAT).
- clusters with nodes in more than one region.
See also wsrep_provider_options -> gmcast.listen_addr
Commandline: --wsrep-node-address=value
Scope: Global
Dynamic: No
Data Type: String
Default Value: Primary network address, usually eth0 with a default port of 4567, or 0.0.0.0 if no IP address.

`wsrep_node_incoming_address`

Description: This is the address from which the node listens for client connections. If an address is not specified or it's set to AUTO (default), mysqld uses either or --wsrep-node-address, or tries to get one from the list of available network interfaces, in the same order. See also wsrep_provider_options -> gmcast.listen_addr.
Commandline: --wsrep-node-incoming-address=value
Scope: Global
Dynamic: No
Data Type: String
Default Value: AUTO

`wsrep_node_name`

Description: Name of this node. This name can be used in wsrep_sst_donor as a preferred donor. Note that multiple nodes in a cluster can have the same name.
Commandline: --wsrep-node-name=value
Scope: Global
Dynamic: Yes
Data Type: String
Default Value: The server's hostname.

`wsrep_notify_cmd`

Description: Command to be executed each time the node state or the cluster membership changes. Can be used for raising an alarm, configuring load balancers and so on. See the Codership Notification Script page for more details.
Commandline: --wsrep-notify-command=value
Scope: Global
Dynamic:
- No (>= )
- Yes (<= )
Data Type: String
Default Value: Empty

`wsrep_on`

Description: Whether or not wsrep replication is enabled. If the global value is set to OFF , it is not possible to load the provider and join the node in the cluster. If only the session value is set to OFF, the operations from that particular session are not replicated in the cluster, but other sessions and applier threads will continue as normal. The session value of the variable does not affect the node's membership and thus, regardless of its value, the node keeps receiving updates from other nodes in the cluster. It is set to OFF by default and must be turned on to enable Galera replication.
Commandline: --wsrep-on[={0|1}]
Scope: Global, Session
Dynamic: Yes
Data Type: Boolean
Default Value: OFF
Valid Values: ON, OFF

`wsrep_OSU_method`

Description: Online schema upgrade method. The default is TOI, specifying the setting without the optional parameter will set to RSU.
- TOI: Total Order Isolation. In each cluster node, DDL is processed in the same order regarding other transactions, guaranteeing data consistency. However, affected parts of the database will be locked for the whole cluster.
- RSU: Rolling Schema Upgrade. DDL processing is only done locally on the node, and the user needs perform the changes manually on each node. The node is desynced from the rest of the cluster while the processing takes place to avoid the blocking other nodes. Schema changes to avoid breaking replication when the DDL processing is complete on the single node, and replication recommences.
Commandline: --wsrep-OSU-method[=value]
Scope: Global, Session
Dynamic: Yes
Data Type: Enum
Default Value: TOI
Valid Values: TOI, RSU

`wsrep_patch_version`

Description: Wsrep patch version, for example wsrep_25.10.
Commandline: None
Scope: Global
Dynamic: No
Data Type: String
Default Value: None

`wsrep_provider`

Description: Location of the wsrep library, usually /usr/lib/libgalera_smm.so on Debian and Ubuntu, and /usr/lib64/libgalera_smm.so on Red Hat/CentOS.
Commandline: --wsrep-provider=value
Scope: Global
- No (>= )
- Yes (<= )
Data Type: String
Default Value: None

`wsrep_provider_options`

Description: Semicolon (;) separated list of wsrep options (see wsrep_provider_options).
Commandline: --wsrep-provider-options=value
Scope: Global
Dynamic: No
Data Type: String
Default Value: Empty

`wsrep_recover`

Description: If set to ON when the server starts, the server will recover the sequence number of the most recent write set applied by Galera, and it will be output to stderr, which is usually redirected to the . At that point, the server will exit. This sequence number can be provided to the wsrep_start_position system variable.
Commandline: --wsrep-recover[={0|1}]
Scope: Global
Dynamic: No
Data Type: Boolean
Default Value: OFF

`wsrep_reject_queries`

Description: Variable to set to reject queries from client connections, useful for maintenance. The node continues to apply write-sets, but an Error 1047: Unknown command error is generated by a client query.
- NONE - Not set. Queries will be processed as normal.
- ALL - All queries from client connections will be rejected, but existing client connections will be maintained.
- ALL_KILL All queries from client connections will be rejected, and existing client connections, including the current one, will be immediately killed.
Commandline: --wsrep-reject-queries[=value]
Scope: Global
Dynamic: Yes
Data Type: Enum
Default Value: NONE
Valid Values: NONE, ALL, ALL_KILL

`wsrep_replicate_myisam`

Description: Whether or not DML updates for tables will be replicated. This functionality is still experimental and should not be relied upon in production systems. Deprecated in , and removed in , use wsrep_mode instead.
Commandline: --wsrep-replicate-myisam[={0|1}]
Scope: Global
Dynamic: Yes
Default Value: OFF
Data Type: Boolean
Valid Values: ON, OFF
Deprecated:
Removed:

`wsrep_restart_slave`

Description: If set to ON, the replica is restarted automatically, when node joins back to cluster.
Commandline: --wsrep-restart-slave[={0|1}]
Scope: Global
Dynamic: Yes
Default Value: OFF
Data Type: Boolean

`wsrep_retry_autocommit`

Description: Number of times autocommited queries will be retried due to cluster-wide conflicts before returning an error to the client. If set to 0, no retries will be attempted, while a value of 1 (the default) or more specifies the number of retries attempted. Can be useful to assist applications using autocommit to avoid deadlocks.
Commandline: --wsrep-retry-autocommit=value
Scope: Global
Dynamic: No
Data Type: Numeric
Default Value: 1
Range: 0 to 10000

`wsrep_slave_FK_checks`

Description: If set to ON (the default), the applier replica thread performs foreign key constraint checks.
Commandline: --wsrep-slave-FK-checks[={0|1}]
Scope: Global
Dynamic: Yes
Data Type: Boolean
Default Value: ON

`wsrep_slave_threads`

Description: Number of replica threads used to apply Galera write sets in parallel. The Galera replica threads are able to determine which write sets are safe to apply in parallel. However, if your cluster nodes seem to have frequent consistency problems, then setting the value to 1 will probably fix the problem. See About Galera Replication: Galera Replica Threads for more information.
Commandline: --wsrep-slave-threads=#
Scope: Global
Dynamic: Yes
Data Type: Numeric
Default Value: 1
Range: 1 to 512

`wsrep_slave_UK_checks`

Description: If set to ON, the applier replica thread performs secondary index uniqueness checks.
Commandline: --wsrep-slave-UK-checks[={0|1}]
Scope: Global
Dynamic: Yes
Data Type: Boolean
Default Value: OFF

`wsrep_sr_store`

Description: Storage for streaming replication fragments.
Commandline: --wsrep-sr-store=val
Scope: Global
Dynamic: No
Data Type: Enum
Default Value: table
Valid Values: table, none

`wsrep_ssl_mode`

`wsrep_sst_auth`

Description: Username and password of the user to use for replication. Unused if wsrep_sst_method is set to rsync, while for other methods it should be in the format <user>:<password>. The contents are masked in logs and when querying the value with . See Introduction to State Snapshot Transfers (SSTs) for more information.
Commandline: --wsrep-sst-auth=value
Scope: Global
Dynamic: Yes
Data Type: String
Default Value: (Empty)

`wsrep_sst_donor`

Description: Comma-separated list (from 5.5.33) or name (as per wsrep_node_name) of the servers as donors, or the source of the state transfer, in order of preference. The donor-selection algorithm, in general, prefers a donor capable of transferring only the missing transactions (IST) to the joiner node, instead of the complete state (SST). Thus, it starts by looking for an IST-capable node in the given donor list followed by rest of the nodes in the cluster. In case multiple candidate nodes are found outside the specified donor list, the node in the same segment (gmcast.segment) as the joiner is preferred. If none of the existing nodes in the cluster can serve the missing transactions through IST, the algorithm moves on to look for a suitable node to transfer the entire state (SST). It first looks at the nodes specified in the donor list (irrespective of their segment). If no suitable donor is still found, the rest of the donor nodes are checked for suitability only if the donor list has a "terminating-comma". Note that a stateless node (the Galera arbitrator) can never be a donor. See Introduction to State Snapshot Transfers (SSTs) for more information. [NOTE] Although the variable is dynamic, the node will not use the new value unless the node requiring SST or IST disconnects from the cluster. To force this, set wsrep_cluster_address to an empty string and back to the nodes list. After setting this variable dynamically, on startup the value from the configuration file will be used again.
Commandline: --wsrep-sst-donor=value
Scope: Global
Dynamic: Yes (read note above)
Data Type: String
Default Value:

`wsrep_sst_donor_rejects_queries`

Description: If set to ON (OFF is default), the donor node will reject incoming queries, returning an UNKNOWN COMMAND error code. Can be used for informing load balancers that a node is unavailable.
Commandline: --wsrep-sst-donor-rejects-queries[={0|1}]
Scope: Global
Dynamic: Yes
Data Type: Boolean
Default Value: OFF

`wsrep_sst_method`

Description: Method used for taking the state snapshot transfer (SST). See Introduction to State Snapshot Transfers (SSTs): SST Methods for more information.
Commandline: --wsrep-sst-method=value
Scope: Global
Dynamic: Yes
Data Type: String
Default Value: rsync
Valid Values: rsync, mysqldump, xtrabackup, xtrabackup-v2, mariadb-backup

`wsrep_sst_receive_address`

Description: This is the address where other nodes (donor) in the cluster connect to in order to send the state-transfer updates. If an address is not specified or its set to AUTO (default), mysqld uses --wsrep_node_address's value as the receiving address. However, if --wsrep_node_address is not set, it uses address from either or tries to get one from the list of available network interfaces, in the same order. Note: setting it to localhost will make it impossible for nodes running on other hosts to reach this node. See Introduction to State Snapshot Transfers (SSTs) for more information.
Commandline: --wsrep-sst-receive-address=value
Scope: Global
Dynamic: Yes
Data Type: String
Default Value: AUTO

`wsrep_start_position`

Description: The start position that the node should use in the format: UUID:seq_no. The proper value to use for this position can be recovered with wsrep_recover.
Commandline: --wsrep-start-position=value
Scope: Global
Dynamic: Yes
Data Type: String
Default Value: 00000000-0000-0000-0000-000000000000:-1

`wsrep_status_file`

Description: wsrep status output filename.
Commandline: --wsrep-status-file=value
Scope: Global
Dynamic: No
Data Type: String
Default Value: None
Introduced:

`wsrep_strict_ddl`

Description: If set, reject DDL statements on affected tables not supporting Galera replication. This is done by checking if the table is InnoDB, which is the only table currently fully supporting Galera replication. MyISAM tables will not trigger the error if the experimental wsrep_replicate_myisam setting is ON. If set, should be set on all tables in the cluster. Affected DDL statements include: (e.g. CREATE TABLE t1(a int) engine=Aria) TRUNCATE TABLE DROP INDEX Statements in , , and are permitted as the affected tables are only known at execution. Furthermore, the various USER, ROLE, SERVER and DATABASE statements are also allowed as they do not have an affected table. Deprecated in and removed in . Use wsrep_mode=STRICT_REPLICATION instead.
Commandline: --wsrep-strict-ddl[={0|1}
Scope: Global
Dynamic: Yes
Data Type: boolean
Default Value: OFF
Introduced:
Deprecated:
Removed:

`wsrep_sync_wait`

Description: Setting this variable ensures causality checks will take place before executing an operation of the type specified by the value, ensuring that the statement is executed on a fully synced node. While the check is taking place, new queries are blocked on the node to allow the server to catch up with all updates made in the cluster up to the point where the check was begun. Once reached, the original query is executed on the node. This can result in higher latency. Note that when wsrep_dirty_reads is ON, values of wsrep_sync_wait become irrelevant. Sample usage (for a critical read that must have the most up-to-date data) SET SESSION wsrep_sync_wait=1; SELECT ...; SET SESSION wsrep_sync_wait=0;
- 0 - Disabled (default)
- 1 - READ (SELECT and BEGIN/START TRANSACTION). This is the same as wsrep_causal_reads=1.
- 2 - UPDATE and DELETE;
- 3 - READ, UPDATE and DELETE;
- 4 - INSERT and REPLACE;
- 5 - READ, INSERT and REPLACE;
- 6 - UPDATE, DELETE, INSERT and REPLACE;
- 7 - READ, UPDATE, DELETE, INSERT and REPLACE;
- 8 - SHOW
- 9 - READ and SHOW
- 10 - UPDATE, DELETE and SHOW
- 11 - READ, UPDATE, DELETE and SHOW
- 12 - INSERT, REPLACE and SHOW
- 13 - READ, INSERT, REPLACE and SHOW
- 14 - UPDATE, DELETE, INSERT, REPLACE and SHOW
- 15 - READ, UPDATE, DELETE, INSERT, REPLACE and SHOW
Commandline: --wsrep-sync-wait=#
Scope: Session
Dynamic: Yes
Data Type: Numeric
Default Value: 0
Range:
- 0 to 15

`wsrep_trx_fragment_size`

Description: Size of transaction fragments for streaming replication (measured in units as specified by wsrep_trx_fragment_unit)
Commandline: --wsrep-trx-fragment-size=#
Scope: Session
Dynamic: Yes
Data Type: numeric
Default Value: 0
Range: 0 to 2147483647

`wsrep_trx_fragment_unit`

Description: Unit for streaming replication transaction fragments' size:
- bytes: transaction’s binlog events buffer size in bytes
- rows: number of rows affected by the transaction
- statements: number of SQL statements executed in the multi-statement transaction
Commandline: --wsrep-trx-fragment-unit=value
Scope: Session
Dynamic: Yes
Data Type: enum
Default Value: bytes
Valid Values: bytes, rows or statements

_{This page is licensed: CC BY-SA / Gnu FDL}

Galera Cluster

MariaDB Galera Cluster

MariaDB Galera Cluster Overview

Overview

An Introduction to Database Replication

Replication Architectures

Primary/Replica

Multi-Primary Replication

Replication Protocols: Asynchronous vs. Synchronous

Asynchronous Replication (Lazy Replication)

Synchronous Replication (Eager Replication)

The Trade-offs of Synchronous Replication

Advantages

Disadvantages

Galera's Solution: Modern Synchronous Replication

How it Works

Architecture

Cluster Configuration

General Configuration

Cluster Name and Address

Galera Replicator Plugin

Cluster Replication

Quorum

Dynamically Bootstrapping the Cluster

State Transfers

Flow Control

Eviction

Streaming Replication

Initiate Streaming Replication

Galera Arbitrator

Scale-out

Failover

Backups

Encryption

Data-in-Transit Encryption

Enabling GCache Encryption

Disabling GCache Encryption

Quickstart Guides

MariaDB Galera Cluster Guide

Quickstart Guide: MariaDB Galera Cluster

1. Prerequisites

2. Installation (on Each Node)

3. Firewall Configuration (on Each Node)

4. Configure Galera Cluster (galera.cnf on Each Node)

5. Start the Cluster

6. Verify Cluster Operation

Further Resources:

MariaDB Galera Cluster Replication Guide

Quickstart Guide: About Galera Replication

1. What is Galera Replication?

2. How Galera Replication Works

3. Key Characteristics

Further Resources:

MariaDB Galera Cluster Usage Guide

Quickstart Guide: MariaDB Galera Cluster Usage

1. Connecting to the Cluster

2. Basic Operations (Reads & Writes)

3. DDL (Data Definition Language) Operations

4. Monitoring Cluster Status

5. Handling Node Failures and Recovery

6. Application Best Practices

Further Resources:

Galera Management

Known Limitations

Limitations from codership.com:

Other observations, in no particular order:

Installation & Deployment

From Source

Installing Galera from Source

Preparation

MariaDB Database Server with wsrep API

Building

Building the Database Server

Preparation

Galera Replication Plugin

Building

Building the Galera Provider

Configuration

Building the Galera wsrep Package on Ubuntu and Debian

Running galera test suite

4. Configure Galera Cluster (`galera.cnf` on Each Node)