March 18, 2015

MariaDB Galera Cluster: A Quirk of Bootstrapping

MariaDB Galera Cluster: A Quirk of BootstrappingMariaDB Galera Cluster is part of the easily deployed High Availability product MariaDB Enterprise Cluster. This blog raises a question about the behaviour of the bootstrapping method.

Old Bootstrapping Method

When Galera was first released, the usual method used to bootstrap a cluster was to set wsrep_cluster_address to 'gcomm://'. With no other node addresses specified in wsrep_cluster_address, the node would form its own cluster:

[gmontee@localhost ~]$ sudo service mysql start --wsrep_cluster_address='gcomm://'
Starting MySQL....                                         [  OK  ]
[gmontee@localhost ~]$ mysql -u root --execute="SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_ready', 'wsrep_cluster_size', 'wsrep_cluster_status', 'wsrep_connected');"
+----------------------+---------+
| Variable_name        | Value   |
+----------------------+---------+
| wsrep_cluster_size   | 1       |
| wsrep_cluster_status | Primary |
| wsrep_connected      | ON      |
| wsrep_ready          | ON      |
+----------------------+---------+

Newer Bootstrapping Method

A while ago, the old bootstrapping method was replaced with the bootstrap option to the mysql.server init script:

[gmontee@localhost ~]$ sudo service mysql bootstrap
Bootstrapping the cluster.. Starting MySQL....             [  OK  ]
[gmontee@localhost ~]$ mysql -u root --execute="SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_ready', 'wsrep_cluster_size', 'wsrep_cluster_status', 'wsrep_connected');"
+----------------------+---------+
| Variable_name        | Value   |
+----------------------+---------+
| wsrep_cluster_size   | 1       |
| wsrep_cluster_status | Primary |
| wsrep_connected      | ON      |
| wsrep_ready          | ON      |
+----------------------+---------+

This newer bootstrapping method was enabled by the --wsrep-new-cluster command-line option. The bootstrap option of the init script passes this option to mysqld under the hood. Explicitly starting the server with this option is equivalent to using the bootstrap option:

[gmontee@localhost ~]$ sudo service mysql start --wsrep-new-cluster
Starting MySQL....                                         [  OK  ]
[gmontee@localhost ~]$ mysql -u root --execute="SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_ready', 'wsrep_cluster_size', 'wsrep_cluster_status', 'wsrep_connected');"
+----------------------+---------+
| Variable_name        | Value   |
+----------------------+---------+
| wsrep_cluster_size   | 1       |
| wsrep_cluster_status | Primary |
| wsrep_connected      | ON      |
| wsrep_ready          | ON      |
+----------------------+---------+

The newer bootstrapping method is more user-friendly. The name of the option matches its intended use (bootstrapping a new cluster), which is more intuitive than overriding wsrep_cluster_address.

A quirk of the new method

Because the new node is forming a new cluster on its own, you may think that you don't have to specify a value of wsrep_cluster_address at all when you use the newer bootstrapping method. Why should the node need to know what other addresses are in the cluster? It will be the only one!

However, this is not true!

[gmontee@localhost ~]$ sudo service mysql start --wsrep-new-cluster --wsrep_cluster_address=''
Starting MySQL....                                         [  OK  ]
[gmontee@localhost ~]$ mysql -u root --execute="SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_ready', 'wsrep_cluster_size', 'wsrep_cluster_status', 'wsrep_connected');"
+----------------------+--------------+
| Variable_name        | Value        |
+----------------------+--------------+
| wsrep_cluster_size   | 0            |
| wsrep_cluster_status | Disconnected |
| wsrep_connected      | OFF          |
| wsrep_ready          | OFF          |
+----------------------+--------------+

Not setting wsrep_cluster_address or setting it to the empty string has a special meaning: it waits for wsrep_cluster_address to be set dynamically. Only after it's set does WSREP get initialized.

[gmontee@localhost ~]$ mysql -u root --execute="SET GLOBAL wsrep_cluster_address='gcomm://';"
[gmontee@localhost ~]$ mysql -u root --execute="SHOW GLOBAL STATUS WHERE Variable_name IN ('wsrep_ready', 'wsrep_cluster_size', 'wsrep_cluster_status', 'wsrep_connected');"
+----------------------+---------+
| Variable_name        | Value   |
+----------------------+---------+
| wsrep_cluster_size   | 1       |
| wsrep_cluster_status | Primary |
| wsrep_connected      | ON      |
| wsrep_ready          | ON      |
+----------------------+---------+

Final Thoughts

Considering that wsrep_cluster_address is listed as a mandatory option, this behavior shouldn't be too surprising. However, it is a bit unintuitive.

Has anyone else been confused by this behavior in the past?

About Geoff Montee

Geoff Montee is a Support Engineer with MariaDB. He has previous experience as a Database Administrator/Software Engineer with the U.S. Government, and as a System Administrator and Software Developer at Florida State University.

Read all posts by Geoff Montee