Getting Started with MariaDB Galera Cluster

MariaDB Galera Cluster is MariaDB plus the MySQL-wsrep patch from Codership.

Downloading and Installing MariaDB Galera Cluster

There are two things you need:

1. MariaDB Galera Cluster

2. The Galera wsrep provider

Download and install MariaDB Galera Cluster binaries as you would regular MariaDB. This includes installing MariaDB Galera Cluster using Yum or Apt. The primary difference is to install the MariaDB Galera Server package instead of the MariaDB Server package and to install the Galera package (the Galera wsrep provider). The Galera package is included in the MariaDB repositories to make installation easier.

If you choose the package manager route, start by configuring your package manager using the Repository Configuration Tool. Then use Yum or Apt (depending on which package manager you use) to install MariaDB Galera Server and Galera. For example, on Ubuntu, you would do the following after configuring Apt:

sudo apt-get update
sudo apt-get install mariadb-galera-server galera

Full instructions for installing MariaDB Galera Cluster with yum and apt-get are available at the following two locations:

• Installing MariaDB Galera Cluster with YUM

• Installing MariaDB Galera Cluster with apt-get

If MariaDB is already installed on the server the package manager will uninstall the appropriate packages prior to installing the MariaDB Galera Cluster packages.

One note if you decide to compile from source: When compiling MariaDB Galera Cluster from the source tarball set: -DWITH_WSREP=ON and -DWITH_INNODB_DISALLOW_WRITES=1.

Prerequisites

Swap Size Requirements

During normal operation a MariaDB Galera node does not consume much more memory than a regular MariaDB server. Additional memory is consumed for the certification index and uncommitted writesets, but normally this should not be noticeable in a typical application. There is one exception though:

1. Writeset caching during state transfer. When a node is receiving a state transfer it cannot process and apply incoming writesets because it has no state to apply them to yet. Depending on a state transfer mechanism (e.g. mysqldump) the node that sends the state transfer may not be able to apply writesets as well. Thus they need to cache those writesets for a catch-up phase. Currently the writesets are cached in memory and, if the system runs out of memory either the state transfer will fail or the cluster will block waiting for the state transfer to end.

To control memory usage for writeset caching, check the Galera parameters: gcs.recv_q_hard_limit, gcs.recv_q_soft_limit, and gcs.max_throttle.

Getting Started

Bootstrapping a new cluster

To bootstrap a new cluster you need to start the first mysqld server with an empty cluster address URL specified either in my.cnf or on the command line like so:

$ mysqld --wsrep_cluster_address=gcomm://

This implies to the server that there is no existing cluster to connect to and it will create a new history UUID.

Restarting the server with the same setting will cause it to create new history UUID again, it won't reconnect to the old cluster. See next section about how to reconnect to an existing cluster.

Adding another node to a cluster

Once you have a cluster running and you want to add/reconnect another node to it, you must supply an address of one of the cluster members in the cluster address URL. E.g. if the first node of the cluster has the address 192.168.0.1, then adding a second node would look like this:

$ mysqld --wsrep_cluster_address=gcomm://192.168.0.1  # DNS names work as well

The new node only needs to connect to one of the existing members. It will automatically retrieve the cluster map and reconnect to the rest of the nodes.

Once all members agree on the membership, state exchange will be initiated during which the new node will be informed of the cluster state. If its state is different from that of the cluster (which is normally the case) it will request a snapshot of the state from the cluster^[1]) and install it before becoming ready for use.

Shutting down the cluster

If you shut down all nodes, you effectively terminated the cluster (not the data of course, but the running cluster), hence I suppose the right way is to start the first node with gcomm:// again.

State Snapshot Transfer

There are two conceptually different ways to transfer a state from one MariaDB server to another:

1. Using mysqldump. This requires the receiving server to be fully initialized and ready to accept connections before the transfer. This method is, by definition, blocking, in that it blocks the donor server from modifying its own state for the duration of the transfer. It is also the slowest of all, and that might be an issue in a loaded cluster.

2. Copying data files directly. This requires that the receiving server is initialized after the transfer. xtrabackup, and other methods fall into this category. These methods are much faster than mysqldump, but they have certain limitations. For example, they can be used only on server startup and the receiving server must be configured very similarly to the donor (e.g. innodb_file_per_table should be the same and so on). Some of these methods (e.g. xtrabackup) can be potentially made non-blocking on the donor. Such methods are supported via a scriptable interface.

SST Scripts

MariaDB Galera Cluster comes with the following following SST scripts:

mysqldump

This is a default method. This script runs only on the sending side and pipes mysqldump output to mysql client connected to receiving server. As noted here, mysqldump needs a username/password pair set in the wsrep_sst_auth variable in order to get the dump.

State transfer failure

It is easy to see that a failure in state transfer generally renders the receiving node unusable. Therefore, should the failure be detected, it will abort. Restarting node after a mysqldump failure may require manual restoration of the administrative tables.

Minimal cluster size

In order to avoid a split-brain condition, the minimum recommended number of nodes in a cluster is 3. Blocking state transfer is yet another reason to require a minimum of 3 nodes in order to enjoy service availability in case one of the members fails and needs to be restarted. While two of the members will be engaged in state transfer, the remaining member(s) will be able to keep on serving client requests.

Configuration and Monitoring

While every effort is made to get MySQL/Galera node to work out of the box, still few parameters need to be set to get it working.

Mandatory settings:

1. wsrep_provider — path to the Galera library
2. wsrep_cluster_address — cluster connection URL
3. binlog_format=ROW
4. default_storage_engine=InnoDB
5. innodb_autoinc_lock_mode=2
6. innodb_locks_unsafe_for_binlog=1

Optional settings

These are just optimizations made relatively safe by synchronous replication — you always recover from another node.

1. innodb_flush_log_at_trx_commit=2

Configuration

• MySQL/Galera Configuration Basics (includes sample configuration)

Monitoring

1. Wsrep-related status variables in MariaDB Galera Cluster server can be queried with the standard

mysql> SHOW STATUS LIKE 'wsrep_%';

1. query. See this page for a detailed variable description.

2. The notification command can be defined to be invoked when cluster membership or node status changes. It can communicate the event to a some monitoring agent.

See also

• Codership on Google Groups (codership-team 'at' googlegroups (dot) com) - A great mailing list for Galera users.
• What is MariaDB Galera Cluster?
• About Galera Replication
• Galera Use Cases
• Codership: Using Galera Cluster

Footnotes

1. ↑ The cluster will choose the most suitable state donor or, alternatively, a desired node can be specified for that role on startup