1 of 27

Galera Management

Galera Management in MariaDB handles synchronous multi-master replication, ensuring high availability, data consistency, failover, and seamless node provisioning across clusters.

Installation & Deployment

Getting Started with MariaDB Galera Cluster

The most recent release of is: Stable (GA) ,

The current of the Galera wsrep provider library are 26.4.21 for 4. For convenience, packages containing these libraries are included in the MariaDB .

Currently, MariaDB Galera Cluster only supports the storage engine (although there is experimental support for and, from , ).

Galera Test Repositories

To facilitate development and QA, we have created some test repositories for the Galera wsrep provider.

These are test repositories. There will be periods when they do not work at all, or work incorrectly, or possibly cause earthquakes, typhoons, and tornadoes. You have been warned.

Galera Test Repositories for YUM

Advanced Installation (From Source)

Installing Galera from Source

There are binary installation packages available for RPM and Debian-based distributions, which will pull in all required Galera dependencies.

If these are not available, you will need to build Galera from source.

The wsrep API for Galera Cluster is included by default. Follow the usual instructions

Preparation

make cannot manage dependencies for the build process, so the following packages need to be installed first:

RPM-based:

Debian-based:

Building the Galera wsrep Package on Ubuntu and Debian

The instructions on this page were used to create the galera package on the Ubuntu and Debian Linux distributions. This package contains the wsrep provider for MariaDB Galera Cluster.

The version of the wsrep provider is 25.3.5. We also provide 25.2.9 for those that need or want it. Prior to that, the wsrep version was 23.2.7.

Install prerequisites:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get -y install check debhelper libasio-dev libboost-dev libboost-program-options-dev libssl-dev scons

Clone galera.git from github.com/mariadb and checkout mariadb-3.x banch:

Build the packages by executing build.sh under scripts/ directory with -p switch:

When finished, you will have the Debian packages for galera library and arbitrator in the parent directory.

Running galera test suite

If you want to run the galera test suite (mysql-test-run --suite=galera), you need to install the galera library as either /usr/lib/galera/libgalera_smm.so or /usr/lib64/galera/libgalera_smm.so

_{This page is licensed: CC BY-SA / Gnu FDL}

Configuration

Galera Cluster Address

URLs in Galera take a particular format:

Schema

gcomm⁣- This is the option to use for a working implementation.
dummy⁣- Used for running tests and profiling, does not do any actual replication, and all following parameters are ignored.

Cluster address

The cluster address shouldn't be empty like gcomm://. This should never be hardcoded into any configuration files.
To connect the node to an existing cluster, the cluster address should contain the address of any member of the cluster you want to join.
The cluster address can also contain a comma-separated list of multiple members of the cluster. It is good practice to list all possible members of the cluster, for example. ⁣gcomm:<node1 name or ip>,<node2 name or ip2>,<node3 name or ip> Alternately, if multicast is used, put the multicast address instead of the list of nodes. Each member address or multicast address can specify <node name or ip>:<port>

Option list

The variable is used to set a . These parameters can also be provided (and overridden) as part of the URL. Unlike options provided in a configuration file, they will not endure and need to be resubmitted with each connection.

A useful option to set is pc.wait_prim=noto ensure the server will start running even if it can't determine a primary node. This is useful if all members go down at the same time.

Port

By default, gcomm listens on all interfaces. The port is either provided in the cluster address or will default to 4567 if not set.

_{This page is licensed: CC BY-SA / Gnu FDL}

Configuring Auto-Eviction

Auto-Eviction enhances cluster stability by automatically removing non-responsive or "unhealthy" in MariaDB Galera Cluster. This prevents a single problematic node from degrading the entire cluster's . In a Galera Cluster, each node monitors the network response times of other nodes. If a node becomes unresponsive due to reasons like memory swapping, network congestion, or a hung process, it can delay and potentially disrupt cluster operations. Auto-Eviction provides a deterministic method to isolate these misbehaving nodes effectively.

Auto-Eviction Process

The Auto-Eviction process is based on a consensus mechanism among the healthy cluster members.

General Operations

Performing Schema Upgrades in Galera Cluster

Performing schema changes (i.e., Data Definition Language or DDL statements like ALTER TABLE, CREATE INDEX) in a MariaDB Galera Cluster requires special handling. Because Galera is a where all nodes must remain in sync, a schema change on one must be safely replicated to all other nodes without causing inconsistencies or blocking the entire cluster for an extended period.

MariaDB Galera Cluster provides two methods for handling schema upgrades:

Method

Description

Managing Sequences in Galera Cluster

allows for the generation of unique integers independent of any specific table. While standard sequences function normally in a standalone MariaDB server, using them in a MariaDB Galera Cluster requires specific configurations to ensure conflict-free operation and optimal performance.

Streaming Replication Support in MariaDB

Starting from MariaDB 10.11.16 (and Galera 26.4.16), sequences are fully supported in transactions utilizing streaming replication. In earlier versions, using NEXTVAL() within a transaction where wsrep_trx_fragment_size > 0 would cause an ERROR 1235. The WSREP API now ensures proper serialization of sequence state in transaction fragments, allowing sequences to be used effectively in large-scale ETL and batch operations. See MDEV-34124

Configuring Sequences for Galera

Because Galera is a multi-primary system, multiple nodes may attempt to generate sequence values simultaneously. To prevent duplicate values and certification failures, the cluster utilizes an offset-based generation strategy.

Mandatory: INCREMENT BY 0

For a sequence to function correctly in a multi-node environment, it must be defined with INCREMENT BY 0.

This setting instructs the node to ignore the sequence's internal increment logic. Instead, the node applies the cluster-wide wsrep_auto_increment_control logic using the following formula:

Where:

Node_Offset: The unique identifier for the specific node (e.g., 1, 2, or 3).
Cluster_Size: The total number of nodes in the cluster.
N: The iteration count (0, 1, 2, ...).

Visualizing the Offset Logic

The following diagram illustrates how two nodes in a 3-node cluster generate unique IDs simultaneously without communicating or locking.

This ensures that nodes generate interleaved, non-conflicting IDs preventing Certification Failures (Error 1213) without requiring network locks.

Cache Configuration Strategies

The CACHE option is the primary lever for balancing performance against data continuity. In Galera, replication introduces a "Flush-on-Sync" behavior: when any node commits a sequence update, other nodes must sync their state, discarding any unused values in their local cache.

Cache Setting

Usage Scenario

Description

Use Case: Active-Active Ticket Reservation System

A common requirement for Galera is a distributed system (e.g., ticket sales) where users in different regions must be able to book items simultaneously without "race conditions" or duplicate Booking IDs.

In this example, we configure a sequence to allow high-speed concurrent bookings from multiple nodes.

Troubleshooting

Error

Cause

Resolution

This page is licensed: CC BY-SA / Gnu FDL

Backing Up a MariaDB Galera Cluster

The recommended strategy for creating a full, consistent backup of a MariaDB Galera Cluster is to perform the backup on a single node. Because all nodes in a healthy cluster contain the same data, a complete backup from one node represents a snapshot of the entire cluster at a specific point in time.

The preferred tool for this is . mariadb-backup creates a "hot" backup without blocking the node from serving traffic for an extended period.

The Challenge of Consistency in a Live Cluster

While taking a backup, the donor node is still receiving and applying transactions from the rest of the cluster. If the backup process is long, it's possible for the data at the end of the backup to be newer than the data at the beginning, leading to an inconsistent state within the backup files.

To prevent this, it's important to temporarily pause the node's replication stream during the backup process.

Recommended Backup Procedure

This procedure ensures a fully consistent backup with minimal impact on the cluster's availability.

1. Select a Backup Node

Choose a node from your cluster to serve as the backup source. It's a good practice to use a non-primary node if you are directing writes to a single server.

2. Desynchronize the Node (Pause Replication)

To guarantee consistency, you should temporarily pause the node's ability to apply new replicated transactions. This is done by setting the wsrep_desync to ON.

Take the selected node out of your rotation so it no longer receives application traffic.
Connect to the node with a mariadb client and execute:
The node will finish applying any transactions already in its queue and then pause, entering a . The rest of the cluster will continue to operate normally.

3. Perform the Backup

With the node's replication paused, run the mariadb-backup to create a full backup.

4. Resynchronize the Node

Once the backup is complete, you can allow the node to rejoin the cluster's replication stream.

Connect to the node again and execute:
The node will now request an from its peers to receive all the transactions it missed while it was desynchronized and quickly catch up.
Once the node is fully synced (you can verify this by checking that is ), add it back to your load balancer's rotation.

This procedure ensures you get a fully consistent snapshot of your cluster's data with zero downtime for your application.

_{This page is licensed: CC BY-SA / Gnu FDL}

Upgrading Galera Cluster

Articles on upgrading between MariaDB versions with Galera Cluster

Upgrading from MariaDB 10.3 to MariaDB 10.4 with Galera Cluster

MariaDB starting with

Since , the patch has been merged into MariaDB Server. Therefore, in and above, the functionality of MariaDB Galera Cluster can be obtained by installing the standard MariaDB Server packages and the Galera wsrep provider library package.

Beginning in , ships with the MariaDB Server. Upgrading a Galera Cluster node is very similar to upgrading a server from to . For more information on that process as well as incompatibilities between versions, see the .

Performing a Rolling Upgrade

Upgrading from MariaDB 10.4 to MariaDB 10.5 with Galera Cluster

ships with the MariaDB Server. Upgrading a Galera Cluster node is very similar to upgrading a server from to . For more information on that process as well as incompatibilities between versions, see the .

Performing a Rolling Upgrade

The following steps can be used to perform a rolling upgrade from to when using Galera Cluster. In a rolling upgrade, each node is upgraded individually, so the cluster is always operational. There is no downtime from the application's perspective.

First, before you get started:

Upgrading from MariaDB 10.5 to MariaDB 10.6 with Galera Cluster

Performing a Rolling Upgrade

First, before you get started:

Performance Tuning

Using Streaming Replication for Large Transactions

Streaming Replication optimizes replication of large or long-running transactions in MariaDB Galera Cluster. Typically, a node executes a transaction fully and replicates the complete write-set to other nodes at time. Although efficient for most workloads, this approach can be challenging for very large or lengthy transactions.

With Streaming Replication, the initiating node divides the transaction into smaller fragments. These fragments are certified and replicated to other nodes while the transaction is ongoing. Once a fragment is certified and applied to the replicas, it becomes immune to abortion by conflicting transactions, thus improving the chances of the entire transaction succeeding. This method also supports processing of transaction write-sets over two Gigabytes.

Streaming Replication is available in Galera Cluster 4.0 and later versions. Both and newer, and and newer, on supported platforms, include Galera 4.

When to Use Streaming Replication

In most cases, the standard replication method is sufficient. Streaming Replication is a specialized tool for specific scenarios. The best practice is to enable it only at the session level for the specific transactions that require it.

Large Data Transactions

This is the primary use case. When performing a massive , , or , normal replication requires the originating node to hold the entire transaction locally and then send a very large write-set at commit time. This can cause two problems:

A significant replication lag, as the entire cluster waits for the large write-set to be transferred and applied.
The replica nodes, while busy applying the large transaction, cannot commit other transactions, which can trigger and throttle the entire cluster.

With Streaming Replication, the node replicates the data in fragments throughout the transaction's lifetime. This spreads the network load and allows replica nodes to apply other concurrent transactions between fragments, minimizing the impact on the overall

Long-Running Transactions

A transaction that remains open for a long time has a higher chance of that commits first. When this happens, the long-running transaction is aborted.

Streaming Replication mitigates this by committing the transaction in fragments. Once a fragment is , it is "locked in" and cannot be aborted by a new conflicting transaction.

Certification keys derive from record locks, not gap locks. If a streaming transaction holds a gap lock, another node's transaction can still apply a in that gap, potentially aborting the streaming transaction.

High-Contention ("Hot") Records

For applications that frequently update the same row (e.g., a counter, a job queue, or a locking scheme), Streaming Replication can be used to force a critical update to replicate immediately. This effectively locks the , preventing other transactions from modifying it and increasing the chance that the critical transaction will commit successfully.

How to Enable and Use Streaming Replication

Streaming Replication should be enabled at the session level just for the transactions that need it. This is controlled by two session variables:

defines what a "unit" of replication is.
defines how many units make up a fragment.

To enable streaming, you set both variables:

In the above example, the node will create, certify, and replicate a fragment after every 10 SQL statements within the transaction.

The available fragment units for wsrep_trx_fragment_unit are:

Parameter

Description

To disable Streaming Replication, you can set wsrep_trx_fragment_size back to 0.

Managing a "Hot Record"

Consider an application that manages a work order queue. To prevent two users from getting the same queue position, you can use Streaming Replication for the single critical update.

Begin the transaction:
After reading necessary data, enable Streaming Replication for just the next statement:
Perform the critical update. This statement will be immediately fragmented and replicated:
Immediately disable Streaming Replication for the rest of the transaction:

This ensures the queue_position update is replicated and certified across the cluster before the rest of the transaction proceeds, preventing .

Limitations and Performance Considerations

Before using Streaming Replication, consider the following limitations:

Performance Overhead

When Streaming Replication is enabled, Galera records all write-sets to a log table () on every node to ensure persistence in case of a crash. This adds write overhead and can impact performance, which is why it should only be used when necessary.

Cost of Rollbacks

If a streaming transaction needs to be rolled back after some fragments have already been applied, the rollback operation consumes system resources on all nodes as they undo the previously applied fragments. Frequent rollbacks of streaming transactions can become a performance problem.

For these reasons, it is always a good application design policy to use shorter, smaller transactions whenever possible.

_{This page is licensed: CC BY-SA / Gnu FDL}

Configuring MariaDB Galera Cluster

A number of options need to be set in order for Galera Cluster to work when using MariaDB. These should be set in the MariaDB option file.

Mandatory Options

Several options are mandatory, which means that they must be set in order for Galera Cluster to be enabled or to work properly with MariaDB. The mandatory options are:

wsrep_provider — Path to the Galera library
— See
— See
— Enable wsrep replication
— This is the default value, or alternately (before MariaDB 10.6) or (MariaDB 10.6 and later).
- — This is the default value, and should not be changed.

These are optional optimizations that can be made to improve performance.

— This is not usually recommended in the case of standard MariaDB. However, it is a safer, recommended option with Galera Cluster, since inconsistencies can always be fixed by recovering from another node.
— This tells InnoDB to use interleaved method. Interleaved is the fastest and most scalable lock mode, and should be used when BINLOG_FORMAT is set to ROW. Setting the auto-increment lock mode for InnoDB to interleaved, you’re allowing slaves threads to operate in parallel.
— This makes state transfers quicker for new nodes. You should start with four slave threads per CPU core. The logic here is that, in a balanced system, four slave threads can typically saturate a CPU core. However, I/O performance can increase this figure several times over. For example, a single-core ThinkPad R51 with a 4200 RPM drive can use thirty-two slave threads. The value should not be set higher than .

Writing Replicated Write Sets to the Binary Log

Like with , write sets that are received by a node with are not written to the by default. If you would like a node to write its replicated write sets to the , then you will have to set . This is especially helpful if the node is a replication master. See .

Replication Filters

Like with , can be used to filter write sets from being replicated by . However, they should be used with caution because they may not work as you'd expect.

The following replication filters are honored for DML, but not DDL:

The following replication filters are honored for DML and DDL for tables that use both the and storage engines:

However, it should be kept in mind that if replication filters cause inconsistencies that lead to replication errors, then nodes may abort.

Network Ports

Galera Cluster needs access to the following ports:

Standard MariaDB Port (default: 3306) - For MySQL client connections and that use the mysqldump method. This can be changed by setting .
Galera Replication Port (default: 4567) - For Galera Cluster replication traffic, multicast replication uses both UDP transport and TCP on this port. Can be changed by setting .
Galera Replication Listening Interface (default: 0.0.0.0:4567) needs to be set using , either

Mutiple Galera Cluster Instances on One Server

If you want to run multiple Galera Cluster instances on one server, then you can do so by starting each instance with , or if you are using , then you can use the relevant .

You need to ensure that each instance is configured with a different .

You also need to ensure that each instance is configured with different .

_{This page is licensed: CC BY-SA / Gnu FDL}

Using the Notification Command (wsrep_notify_cmd)

MariaDB Galera Cluster provides a powerful automation feature through the wsrep_notify_cmd system variable. When this variable is configured, the MariaDB server will automatically execute a specified command or script in response to changes in the cluster's membership or the local node's state.

This is extremely useful for integrating the cluster with external systems:

System

Description

Automatically add or remove nodes from the load balancer's pool as they join or leave the cluster.

Send custom alerts to a monitoring system when a node's status changes.

Service Discovery

Configuration

To use this feature, you set the wsrep_notify_cmd variable in your MariaDB configuration file (my.cnf) to the full path of the script you want to execute:

The MariaDB server user must have the permissions for the specified script.

Passed Parameters

When a cluster event occurs, the server executes the configured script and passes several arguments to it, providing context about the event. The script can then use these arguments to take appropriate action.

The script is called with the following parameters:

Position

Parameter

Description

Status Values (`$1`)

The first argument indicates the . The most common values are:

Joining: The node is starting to join the cluster.
Joined: The node has finished a state transfer and is catching up.
Synced: The node is a fully operational member of the cluster.

View ID (`$2`)

The View ID is a unique identifier composed of the view sequence number and the UUID of the node that initiated the view change. It changes every time a node joins or leaves the cluster.

Members List Format (`$3`)

The third argument is a comma-separated list of the wsrep_node_name of every .

Your script can parse this list to get a complete, real-time picture of the cluster's membership.

Example Script

Here is a simple example of a bash script that logs all cluster state changes to a file.

notify_script.sh:

This script would provide a simple, human-readable log of all membership and node state changes, which can be invaluable for troubleshooting.

_{This page is licensed: CC BY-SA / Gnu FDL}

Galera Management

Installation & Deployment

Getting Started with MariaDB Galera Cluster

Galera Test Repositories

Galera Test Repositories for YUM

Advanced Installation (From Source)

Installing Galera from Source

Preparation

Building the Galera wsrep Package on Ubuntu and Debian

Running galera test suite

Configuration

Galera Cluster Address

Schema

Cluster address

Option list

Port

Configuring Auto-Eviction

Auto-Eviction Process

General Operations

Performing Schema Upgrades in Galera Cluster

Managing Sequences in Galera Cluster

Streaming Replication Support in MariaDB

Configuring Sequences for Galera

Mandatory: INCREMENT BY 0

Cache Configuration Strategies

Use Case: Active-Active Ticket Reservation System

Troubleshooting

Backing Up a MariaDB Galera Cluster

The Challenge of Consistency in a Live Cluster