In MariaDB 5.5 and MariaDB 10.0, MariaDB Galera Server is a separate package installed instead of the standard MariaDB Server package. Since MariaDB 10.1, the MariaDB Server and MariaDB Galera Server packages have been combined and Galera packages and their dependencies get installed automatically when installing MariaDB. The Galera parts remain dormant until configured, like a plugin or storage engine.
Currently MariaDB Galera Cluster only supports the InnoDB/XtraDB storage engine.
- Downloading and Installing MariaDB Galera Cluster
- Getting started
- Bootstrapping a New Cluster
- Adding Another Node to a Cluster
- Restarting the Cluster
- State Snapshot Transfer
- SST Scripts
- SSTs and Systemd
- State Transfer Failure
- Minimal Cluster Size
- Manual SSTs
- Configuration and Monitoring
- See Also
Downloading and Installing MariaDB Galera Cluster
There are two things you need:
- MariaDB Galera Cluster
- The Galera wsrep provider library
Download and install MariaDB Galera Cluster binaries/packages as you would regular MariaDB Server. In 10.1 they are part of the regular download, in 10.0 and 5.5 they are separate downloads. This includes installing MariaDB Galera Cluster using Yum or Apt. The primary difference for 10.0 and 5.5 is to install the MariaDB Galera Server package instead of the MariaDB Server package and to install the Galera package (the Galera wsrep provider). The Galera package is included in the MariaDB repositories to make installation easier.
If you choose the package manager route, start by configuring your package manager using the Repository Configuration Tool. Then use Yum or Apt (depending on which package manager you use) to install MariaDB Galera Server and Galera. For example, on Ubuntu, you would do the following after configuring Apt:
sudo apt-get update # For server versions prior to 10.1 : sudo apt-get install mariadb-galera-server # For 10.1 versions : sudo apt-get install mariadb-server
Full instructions for installing MariaDB Galera Cluster with
apt-get are available at the following two locations:
If MariaDB is already installed on the server the package manager will uninstall the appropriate packages prior to installing the MariaDB Galera Cluster packages.
MariaDB Galera Cluster starting with 10.0.24
To unify the location of the
libgalera_smm.so library between the bintar,
rpm, and deb packages, the library is now found at
lib/galera/libgalera_smm.so in the bintar packages, with a symlink in the
lib directory that points to it.
One note if you decide to compile from source: When compiling MariaDB Galera Cluster from the source tarball set:
A great resource for Galera users is Codership on Google Groups (
com) - If you use Galera it is recommended you subscribe.
Swap Size Requirements
During normal operation a MariaDB Galera node does not consume much more memory than a regular MariaDB server. Additional memory is consumed for the certification index and uncommitted writesets, but normally this should not be noticeable in a typical application. There is one exception though:
- Writeset caching during state transfer. When a node is receiving a state transfer it cannot process and apply incoming writesets because it has no state to apply them to yet. Depending on a state transfer mechanism (e.g. mysqldump) the node that sends the state transfer may not be able to apply writesets as well. Thus they need to cache those writesets for a catch-up phase. Currently the writesets are cached in memory and, if the system runs out of memory either the state transfer will fail or the cluster will block waiting for the state transfer to end.
To control memory usage for writeset caching, check the Galera parameters:
See the Galera Limitations page for a complete list of requirements and limitations.
Server Configuration Limits
- general log and slow query log have to be file type and cannot be CVS or any other storage engine
- query cache has to be disabled (only for versions prior to 5.5.40-galera and 10.0.14-galera)
Bootstrapping a New Cluster
To bootstrap a new cluster you need to start the first mysqld server with the option --wsrep-new-cluster on the command line like so:
$ mysqld --wsrep-new-cluster
This implies to the server that there is no existing cluster to connect to and it will create a new history UUID.
Restarting the server with the same setting will cause it to create new history UUID again, it won't reconnect to the old cluster. See next section about how to reconnect to an existing cluster.
However, keep in mind that most users are not going to bootstrap a server by executing "mysqld --wsrep-new-cluster" manually. Instead, most users will use one of the following wrappers to bootstrap a node, depending on the server's operating system.
On operating systems that use SysV init, a node can be bootstrapped in the following way:
$ service mysql bootstrap
On operating systems that use systemd, a node can be bootstrapped in the following way:
MariaDB starting with 10.1.8
Systemd support and the galera_new_cluster script were added in MariaDB 10.1.
Adding Another Node to a Cluster
Once you have a cluster running and you want to add/reconnect another node to it, you must supply an address of one of the cluster members in the cluster address URL. E.g. if the first node of the cluster has the address 192.168.0.1, then a test to add a second node would look like this:
$ mysqld --wsrep_cluster_address=gcomm://192.168.0.1 # DNS names work as well, IP is preferred for performance
The new node only needs to connect to one of the existing members. It will automatically retrieve the cluster map and reconnect to the rest of the nodes, of course it's better to list all nodes of the cluster so that any node can join a cluster connecting to any other node, even if one or more are down.
wsrep_cluster_address parameter should be added in
my.cnf of each node, listing all the nodes of the cluster, if a node has its own IP address listed in such variable that is not a problem and it will be simply ignored(You may still see messages in the log suggesting it's being ignored).
Once all members agree on the membership, state exchange will be initiated during which the new node will be informed of the cluster state. If its state is different from that of the cluster (which is normally the case) it will request a snapshot of the state from the cluster) and install it before becoming ready for use.
Restarting the Cluster
If you shut down all nodes you effectively terminated the cluster (not the data of course, but the running cluster), hence you'll need to bootstrap the cluster again, bootstrap a Galera cluster basically means tell the first node to start not to try to connect to any other node. Not bootstrapping a Galera node(that is using a regular service start) means that a Galera node will try to connect to at least one of the nodes listed in the
wsrep_cluster_address variable, and stop if it can't connect to any of them.
In some cases Galera will refuse to bootstrap a node if there are chances it's not the most advanced node, it can deduce it if it was not the last one of the cluster to be shut down or a crash happened, in such cases a manual intervention is needed.
If you know for sure which node is the most advanced you can edit the
grastate.dat file in the datadir (http://galeracluster.com/2016/11/introducing-the-safe-to-bootstrap-feature-in-galera-cluster/) and set
safe_to_bootstrap=1 or add an option in
In the file
grastate.dat you can read the
seqno which can suggest which node is the most advanced(higher number).
State Snapshot Transfer
There are two conceptually different ways to transfer a state from one MariaDB server to another:
- Using mysqldump. This requires the receiving server to be fully initialized and ready to accept connections before the transfer. This method is, by definition, blocking, in that it blocks the donor server from modifying its own state for the duration of the transfer. It is also the slowest of all, and that might be an issue in a loaded cluster.
- Copying data files directly. This requires that the receiving server is initialized after the transfer. mariabackup, and other methods fall into this category. These methods are much faster than mysqldump, but they have certain limitations. For example, they can be used only on server startup and the receiving server must be configured very similarly to the donor (e.g. innodb_file_per_table should be the same and so on). Some of these methods (e.g. mariabackup) can be potentially made non-blocking on the donor. Such methods are supported via a scriptable interface.
MariaDB Galera Cluster comes with the following SST scripts:
Uses the MariaDB Backup utility for performing snapshot state transfers. One of the two non-locking methods. Note that if you use the mariabackup SST method, you also need to have socat installed on the server. This is needed to stream the backup from the donor to the joiner. This is a limitation inherited from the xtrabackup-v2 SST method. Available from MariaDB 10.2.10 and MariaDB 10.1.26.
This script runs only on the sending side and pipes mysqldump output to the mysql client connected to the receiving server. mysqldump needs a username/password pair set in the wsrep_sst_auth variable in order to get the dump. This method requires a READ LOCK (the donor will be read-only during the whole process).
This is the default method. This method uses the rsync utility for snapshots. rsync should be available by default on all modern linux distributions. This is the fastest and recommended method, especially for large datasets since it copies binary data. It requires a READ LOCK during the whole process, as for mysqldump.
As of MariaDB 10.3.10. MariaDB 10.2.18 and MariaDB 10.1.36, stunnel can be used to encrypt data over the wire. Be sure to have it installed, and to set the tkey and tcert options, referring to the Galera Cluster documentation. The certificate directory will have to be hashed: openssl rehash /path/to/certs/.
Percona XtraBackup does not work with MariaDB 10.1 or greater if encryption or compression is used, or when innodb_page_size is set to some value other than 16K. It also does not work with MariaDB 10.2 or greater if innodb_safe_truncate=ON is set. It also does not work with MariaDB 10.3 or greater. For the cases where Percona XtraBackup is not supported, see Mariabackup instead.
This method uses the xtrabackup open-source hot backup utility to perform snapshot state transfers. Along with mariabackup, it is a non-locking method; however, it requires some additional setup. Please refer to the xtrabackup SST documentation for more information.
xtrabackup-v2 should be used instead of
xtrabackup starting from MariaDB 5.5.33.
SSTs and Systemd
MariaDB's systemd unit file has a fairly low timeout value by default. If your SST takes longer than about 2 minutes, this low default timeout can cause systemd to assume that mysqld startup has failed, which causes it to kill the mysqld process. To work around this, you can reconfigure the MariaDB systemd unit to have an infinite timeout, such as by executing one of the following commands:
If you are using systemd 228 or older, then you can execute the following to set an infinite timeout:
sudo tee /etc/systemd/system/mariadb.service.d/timeoutstartsec.conf <<EOF [Service] TimeoutStartSec=0 EOF sudo systemctl daemon-reload
Systemd 229 added the infinity option, so if you are using systemd 229 or later, then you can execute the following to set an infinite timeout:
sudo tee /etc/systemd/system/mariadb.service.d/timeoutstartsec.conf <<EOF [Service] TimeoutStartSec=infinity EOF sudo systemctl daemon-reload
See customizing systemd for more details.
Note that systemd 236 added the EXTEND_TIMEOUT_USEC environment variable that allows services to extend the startup timeout during long-running processes. MariaDB uses that functionality during long SSTs to extend the timeout on systems with systemd versions that support it. Therefore, if you are using systemd 236 or later, then you should not need to manually override TimeoutStartSec, even if your SSTs run for longer than the configured value. See MDEV-15607 for more information.
State Transfer Failure
It is easy to see that a failure in state transfer generally renders the receiving node unusable. Therefore, should the failure be detected, it will abort. Restarting a node after a
mysqldump failure may require manual restoration of the administrative tables.
Minimal Cluster Size
In order to avoid a split-brain condition, the minimum recommended number of nodes in a cluster is 3. Blocking state transfer is yet another reason to require a minimum of 3 nodes in order to enjoy service availability in case one of the members fails and needs to be restarted. While two of the members will be engaged in state transfer, the remaining member(s) will be able to keep on serving client requests.
In some cases, if Galera Cluster's automatic SSTs repeatedly fail, then it can be helpful to perform a "manual SST". See the following pages on how to do that:
- Manual SST of Galera Cluster node with Mariabackup
- Manual SST of Galera Cluster node with Percona XtraBackup
Configuration and Monitoring
A number of parameters need to be set in order for MariaDB Galera to work:
- wsrep_provider — Path to the Galera library
- wsrep_cluster_address — see cluster connection URL
- binlog_format=ROW — see Binary Log Formats
- innodb_doublewrite=1 (the default) when using Galera provider of version >= 2.0.
- query_cache_size=0 (only for versions prior to 5.5.40-galera, 10.0.14-galera and 10.1.2)
- wsrep_on=ON — Enable wsrep replication (starting 10.1.1)
These are just optimizations made relatively safe by synchronous replication — you always recover from another node.
- Codership's MySQL/Galera Configuration Basics (includes sample configuration)
- Galera status variables can be queried with the standard
SHOW STATUS LIKE 'wsrep_%';
- The notification command can be defined to be invoked when cluster membership or node status changes. It can communicate the event to a some monitoring agent.
- Codership on Google Groups (
codership-team 'at' googlegroups (dot) com) - A great mailing list for Galera users.
- What is MariaDB Galera Cluster?
- About Galera Replication
- Galera Use Cases
- Codership: Using Galera Cluster
- ↑ The cluster will choose the most suitable state donor or, alternatively, a desired node can be specified for that role on startup